Low-illumination image enhancement method based on double-branch decoupling network and multi-mode perception

Abstract

The invention discloses a low-illumination image enhancement method based on a double-branch decoupling network and multi-mode perception, and belongs to the technical field of computer vision and image processing. According to the method, a dual-branch decoupling architecture with a channel self-attention recovery network as a trunk is constructed, the dual-branch decoupling architecture comprises a high-brightness attention branch and a low-brightness attention branch, and differentiation recovery is carried out on a bright part region and a dark part region in an image; in order to realize accurate decoupling, a brightness loss attention algorithm is designed, and each branch is guided to concentrate on feature extraction of a specific illumination region by using an adaptive mask strategy; meanwhile, in combination with CLIP multi-mode loss, the generated image is guided to approach to a high-quality visual style by means of text semantic information; and finally, performing deep fusion and up-sampling on the double-branch features through an RCAB ultra-resolution module, and outputting a high-resolution enhanced image. According to the method, the technical problems of illumination optimization conflict caused by global processing in low-illumination image enhancement and overhigh computing power consumption of high-resolution image processing are solved.

Description

Low-light image enhancement method based on dual-branch decoupling network and multimodal sensing Technical Field

[0001] This invention relates to the fields of computer vision and image processing technology, and in particular to a low-light image enhancement method based on a dual-branch decoupling network and multimodal perception. Background Technology

[0002] Low-light images often suffer from underexposure or localized overexposure, severe noise, limited dynamic range, and difficulty in detail recovery. Traditional enhancement methods, such as gamma correction and histogram equalization, have limited effectiveness and fail to meet practical needs. In contrast, end-to-end single-stage methods based on deep learning have made significant progress in the field of low-light image enhancement. To address the problem of low signal-to-noise ratio (SNR), some studies have proposed a SNR-aware framework, combining a Transformer backbone network with convolutional networks to process long and short distance features, thereby improving the overall image quality. The Structure and Tone Guided Auxiliary Learning Network (SHAL-Net) decomposes the image into luminance and reflectance, and performs luminance adjustment and denoising on these two components separately. The Robust and Fast Low-Light Image Enhancement Network (RFLLIE) employs a lightweight head and heavyweight tail architecture, combined with a decomposition-guided coarse-to-fine restoration strategy, to improve image contrast and suppress noise. Furthermore, image enhancement methods based on Retinex theory have been widely applied. For example, Retinexformer significantly improves the visual effect of images by estimating illumination information and using it to compensate for low-light images. The latest low-light super-resolution method, IDB, also uses a two-branch approach, focusing on the ease of pixel recovery and separately recovering "difficult" and "easy" pixels. Although the above methods each have their advantages, they still face two major bottlenecks: First, the coupling conflict of illumination tasks. Existing methods mostly adopt a globally shared parameter mechanism, applying the same set of weights to the entire image and ignoring the non-uniformity of illumination distribution. This makes it difficult for the network to balance the two contradictory goals of "high-gain brightening of dark areas" and "low-gain suppression of bright areas," easily leading to local overexposure or insufficient enhancement. Second, the computational limitations of high-resolution processing. Existing networks mostly perform complex end-to-end operations at the original resolution. As the image resolution increases (e.g., 4K), the computational load and memory usage increase dramatically, making it difficult to balance real-time performance and detail recovery, and unable to effectively process high-resolution low-light images. Summary of the Invention

[0003] The purpose of this invention is to provide a low-light image enhancement method based on a dual-branch decoupled network and multimodal perception, which aims to solve the technical problems of illumination optimization conflicts caused by global processing and excessive computing power consumption in high-resolution image processing in existing low-light image enhancement technologies.

[0004] To achieve the above objectives, this invention provides a low-light image enhancement method based on a dual-branch decoupling network and multimodal perception, comprising the following steps: S1, downsampling the input low-light image to obtain a low-resolution low-light image; S2, inputting the low-resolution low-light image into a dual-branch decoupling module, using two structurally identical but parameter-independent sub-networks as high-brightness and low-brightness branches respectively, extracting high-brightness and low-brightness features and generating enhanced feature maps and prediction maps respectively; the sub-networks are based on a U-shaped channel-level self-attention recovery network as the backbone, integrating a Gamma pre-enhancement module and a channel self-attention module; S3, employing brightness loss... An attention-delay algorithm generates an adaptive brightness-aware mask, which is then used to spatially gate and fuse the predicted image output by the dual-branch network to obtain a low-resolution intermediate enhanced image. S4: A CLIP multimodal semantic constraint mechanism is introduced to apply semantic loss supervision to the low-resolution intermediate enhanced image, optimizing its color and style. S5: The semantically optimized low-resolution enhancement features are input into the RCAB super-resolution module to reconstruct a high-resolution enhanced image. S6: An end-to-end joint training strategy is adopted, using a total loss function composed of multi-scale pixel-level loss and multimodal semantic loss to supervise network training, achieving efficient enhancement of low-light images.

[0005] Preferably, in S2, the U-shaped channel-level self-attention recovery network adopts an encoder-decoder structure. The encoder consists of three stages, each containing a convolutional layer and a channel self-attention module. It gradually reduces the spatial resolution of the feature map through a downsampling operation with a stride of 2, while doubling the number of channels. The decoder symmetrically upsamples through deconvolutional layers, introducing lateral skip connections to concatenate the feature maps of the corresponding encoder layers with the decoder feature maps in the channel dimension to achieve the fusion of low-level texture information and high-level semantic information. The lowest resolution layer of the U-shaped channel-level self-attention recovery network stacks two channel self-attention modules to capture long-distance global dependencies. Preferably, in S2, the implementation process of the Gamma pre-enhancement module is as follows: the input features are initially extracted through a 3×3 depthwise separable convolution and a Group Norm layer; compressed into a global feature vector through a 1×1 convolution and global average pooling; and an adaptive Gamma coefficient is generated using the Sigmoid activation function and applied to the original image. Preferably, in S2, the execution content of the channel self-attention module includes the following steps: S21, setting the input features as... The query matrix Q, key matrix K, and value matrix V are generated through 1x1 convolution and reshape operations. S22. Calculate the dot product of K and Q after transpose to generate a channel covariance matrix of size C×C. The calculation formula is as follows: , where d is the scaling factor; S23, after adding the result of S22 to the value matrix V, input it into the gated feedforward network, and then expand the channels through pointwise convolution, depthwise convolution and GELU activation function to enhance the network's nonlinear expression ability of local detail features; then after pointwise convolution dimensionality reduction and residual connection, the enhanced features are output.

[0006] Preferably, in S3, the brightness loss attention algorithm includes constructing an adaptive brightness-aware mask and mask-based image fusion, specifically as follows: S31, Constructing an adaptive brightness-aware mask: S311, Brightness component extraction and normalization: Assume the input of the dual-branch network is a low-resolution image. The luminance component Y of the image is extracted using a weighted method and then normalized. ;in for Pixel coordinates, R is The red channel, G is The green channel, B is The blue channel; S312, Global Illumination Threshold Estimation: Calculate the average brightness of the entire image. As a dynamic segmentation threshold: Where H and W are respectively Height and width; S313, Local smoothing filtering: A block smoothing strategy is introduced for local smoothing filtering, specifically, the brightness map Y is divided into non-overlapping blocks. Local neighborhood patches, denoted as Calculate each Average brightness within: ;in Indicates a local area The average brightness within; using Alternative The smoothed brightness distribution map is obtained from all pixel values ​​within the area. S314, Mask Binarization Generation: Based on Global Thresholding Smooth brightness map Perform binarization to generate the final brightness attention mask. : S32. Mask-based image fusion: S321. Using mask M to process the highlight branch feature map output by the dual-branch network. Prediction chart and low-brightness branch feature map Prediction chart Spatial gated fusion is performed to generate low-resolution intermediate enhancement results. : ;in S322 represents element-wise multiplication; intermediate results... Compared with the downsampled real reference image Supervise and construct low-resolution reconstruction loss. : ;in, S323. Describe the L1 norm; S323. Calculate the loss function according to the chain rule. The partial derivative of the bi-branch output, i.e., the error gradient: ; ;in, The function represents a sign function; it returns 1 for a positive input, -1 for a negative input, and 0 for a zero input. Preferably, in S4, the implementation process of the CLIP multimodal semantic constraint mechanism is as follows: S41, construct several sets of discriminative positive and negative text pairs through prompt word engineering, and set positive prompt words for the positive text. Negative prompts in negative text S42. During the training iteration, the low-resolution image is enhanced. The text pairs are mapped to the same feature space using CLIP's image encoder and text encoder, respectively, and their cosine similarity is calculated. S43. Construct a combined loss function that includes absolute distance constraints and relative probability constraints, i.e., multimodal semantic loss. : ;in, Let represent the cosine similarity between A and B; By minimizing the similarity difference, the image is brought closer to the positive text, while the image is pushed further away from the negative text. The similarity is transformed into a probability distribution based on the Softmax form of the penalty term. The higher the probability that an image is misidentified as a low-quality sample, the greater the penalty loss. Preferably, in S5, the RCAB super-resolution module performs the following: S51, recovering the highlight features from the output of the dual-branch network. Low brightness recovery features The features are stitched together along the channel dimension to form a fused feature body containing complete illumination information; S52, the stitched features undergo preliminary feature fusion through a 3×3 convolutional layer, and the Tanh activation function is introduced to constrain the feature distribution. Within the interval; S53, deep feature extraction is performed through several concatenated residual channel attention blocks, introducing global long skip connections to solve the gradient vanishing problem in deep networks and pass low-frequency structural information, directly adding shallow features to deep output features; S54, upsampling is performed using a PixelShuffle layer to rearrange multi-channel feature maps into spatial pixel blocks, using channel information to generate high-resolution details and avoiding checkerboard artifacts; S55, the upsampled features are passed through a reconstruction convolutional layer to output the final high-resolution enhanced image. Preferably, in S6, the total loss function Defined as the weighted sum of the three sub-losses: ;in, , , These are the weighting coefficients; Calculate intermediate results for low-resolution reconstruction loss. Compared with the downsampled real reference image The L1 distance between them; For multimodal semantic loss; The high-resolution reconstruction loss is used to calculate the final generated high-resolution enhanced image. Compared to the original high-resolution real image The L1 pixel error between them is calculated using the following formula: The present invention also provides a low-light enhancement network based on dual-branch decoupling and multimodal perception, including a dual-branch decoupling module, a brightness loss attention module, a multimodal optimization module, and an RCAB super-resolution module. The dual-branch decoupling module is used to extract features of different illumination regions, the brightness loss attention module is used to generate an adaptive mask and achieve feature fusion, the multimodal optimization module is used to apply semantic constraints, and the RCAB super-resolution module is used for high-resolution image reconstruction.

[0007] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of a low-light image enhancement method.

[0008] Therefore, the low-light image enhancement method based on the dual-branch decoupling network and multimodal perception adopted in this invention has the following beneficial effects: (1) To address the optimization conflict caused by uneven illumination distribution: a dual-branch decoupling architecture is constructed, and high and low brightness attention branches are used to process different illumination areas respectively. On this basis, an adaptive mask is generated by the brightness loss attention algorithm, and the physical fusion and decoupling of the dual-branch outputs are realized at the pixel level, thus resolving the contradiction between brightening dark areas and suppressing bright areas.

[0009] (2) To address the issue of semantic and color deviation during the enhancement process: a CLIP multimodal semantic constraint mechanism in low-resolution space is introduced. CLIP loss is applied to the low-resolution enhanced image after the brightness loss attention module. Semantic supervision is introduced into the core nodes of the enhancement stage to ensure that the intermediate restored image conforms to the high-quality text description in terms of illumination and color style. At the same time, it avoids direct adversarial interference to high-resolution textures, thereby effectively improving the visual perception quality of the image while ensuring pixel-level indicators (PSNR / SSIM).

[0010] (3) To address the difficulty of high-resolution image processing: by completing illumination restoration and semantic style correction in low-resolution space, the computational load is greatly reduced; then, the RCAB super-resolution module is used to upsample the semantically and illumination-optimized features to the original resolution, thus achieving high-efficiency and high-fidelity high-resolution image reconstruction.

[0011] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0012] Figure 1 is a schematic diagram of the overall architecture of an embodiment of the present invention; Figure 2 is a structural diagram of the channel-level self-attention recovery network of an embodiment of the present invention; Figure 3 is a structural diagram of the Gamma pre-enhancement module of an embodiment of the present invention; Figure 4 is a structural diagram of the channel self-attention module of an embodiment of the present invention; Figure 5 is a visualization of the intermediate results of the brightness loss attention algorithm of an embodiment of the present invention; Figure 6 is a visualization of the dual-branch decoupling feature heatmap of an embodiment of the present invention; Figure 7 is a schematic diagram of the multimodal optimization module of an embodiment of the present invention; Figure 8 is a structural diagram of the RCAB super-resolution module of an embodiment of the present invention; Figure 9 is a comparison diagram of the LOL-v1 enhancement results of an embodiment of the present invention; Figure 10 is a comparison diagram of the LOL-v2 enhancement results of an embodiment of the present invention; Figure 11 is a comparison diagram of the ACDC-syn enhancement results of an embodiment of the present invention. Detailed Implementation

[0013] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0014] Unless otherwise defined, the technical or scientific terms used in this invention shall have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly. Examples

[0015] This invention provides a low-light enhancement network based on dual-branch decoupling and multimodal perception. The network architecture is shown in Figure 1, and the overall structure adopts a cascaded strategy of "low-resolution enhancement - high-resolution reconstruction". It includes a dual-branch decoupling module, a luminance loss attention module, a multimodal optimization module, and an RCAB super-resolution module. The dual-branch decoupling module is used to extract features from different illumination regions, the luminance loss attention module is used to generate adaptive masks and achieve feature fusion, the multimodal optimization module is used to apply semantic constraints, and the RCAB super-resolution module is used for high-resolution image reconstruction.

[0016] A low-light image enhancement method based on a dual-branch decoupling network and multimodal perception was constructed using the aforementioned network. The general workflow is as follows: the downsampled low-resolution low-light image is input into the dual-branch decoupling module. Two isomorphic but parameter-distributed channel self-attention backbone networks are used to extract highlight and low-light features respectively, generating enhanced images. Subsequently, an adaptive mask is generated through a brightness loss attention module, guiding the two branches to focus on regions with different brightness levels. Simultaneously, CLIP multimodal semantic constraints are introduced during the low-resolution enhancement process, utilizing text priors to optimize the image's color and style. Finally, the stitched low-resolution enhanced features are reconstructed into a high-resolution image via the RCAB super-resolution module, achieving end-to-end super-resolution enhancement. The specific process is as follows: S1, the input low-light image is downsampled to obtain a low-resolution low-light image.

[0017] S2. The low-resolution low-light image is input into the dual-branch decoupling module to achieve illumination decoupling and feature extraction. Two sub-networks with the same structure but independent parameters are used as high-brightness branches and low-brightness branches, respectively, to extract high-brightness and low-brightness features and generate enhanced feature maps and prediction maps, respectively. The sub-networks are based on a U-shaped channel-level self-attention recovery network and integrate a Gamma pre-enhancement module and a channel self-attention module to achieve accurate extraction of multi-scale illumination features and global context modeling.

[0018] The architecture of the U-shaped channel-level self-attention recovery network is shown in Figure 2, employing an encoder-decoder structure. The encoder consists of three stages, each containing a convolutional layer and a channel self-attention module. It progressively reduces the spatial resolution of the feature map through a downsampling operation with a stride of 2, while doubling the number of channels; this process aims to expand the receptive field and extract deeper semantic features. The decoder symmetrically upsamples through deconvolutional layers, progressively restoring the spatial size of the feature map. To compensate for the spatial details lost during downsampling, lateral skip connections are introduced to concatenate the feature maps of corresponding layers in the encoder and decoder along the channel dimension, achieving the fusion of low-level texture information and high-level semantic information. The lowest resolution layer of the U-shaped channel-level self-attention recovery network stacks two channel self-attention modules to capture long-range global dependencies.

[0019] To reduce the burden on the network for image enhancement, a Gamma pre-enhancement module is introduced before the formal feature extraction. The module structure is shown in Figure 3. This module aims to simulate the Gamma correction process in traditional image processing, but its parameters are fully adaptively learned. The implementation process of the Gamma pre-enhancement module is as follows: input features are initially extracted through a 3×3 depthwise separable convolution and a GroupNorm layer; then compressed into a global feature vector through a 1×1 convolution and global average pooling; finally, an adaptive Gamma coefficient is generated using the Sigmoid activation function and applied to the original image. This process allows the network to dynamically adjust the nonlinear response of the features according to the brightness distribution of the input image, providing a more balanced input distribution for subsequent feature extraction.

[0020] While traditional visual Transformers overcome the limitations of traditional convolutional neural networks in modeling long-range dependencies, their computational complexity for calculating self-attention in the spatial dimension increases quadratically with image resolution. To address this issue, a channel self-attention module was designed to calculate correlations along the channel dimension, as shown in Figure 4. The execution of the channel self-attention module includes the following steps: S21, setting the input feature as... The query matrix Q, key matrix K, and value matrix V are generated through 1x1 convolution and reshape operations. S22. Calculate the dot product of the transposed K and Q to generate a channel covariance matrix of size C×C. The computational complexity of this process is related to the number of pixels. The relationship is linear, and the calculation formula is: , where d is the scaling factor. Through the above S21 and S22, the generated channel attention map effectively captures the global dependencies between different channel features, enabling the network to adaptively filter key features, which is crucial for distinguishing noise from texture.

[0021] S23. The result of S22 is added to the value matrix V and then input into the gated feedforward network. The network is expanded through pointwise convolution, depthwise convolution and GELU activation function to enhance the nonlinear expression ability of the network for local detailed features. Then, it is reduced by pointwise convolution and residual connection to output enhanced features.

[0022] The channel self-attention module combines the global perception capability of the attention mechanism with the local refinement capability of the gated feedforward network, effectively improving the feature extraction efficiency and quality of the network.

[0023] S3. In a dual-branch decoupling network, establishing branch optimization objectives to enable high-brightness and low-brightness branches to spontaneously adapt to their respective illumination regions is key to achieving efficient enhancement. Traditional feature fusion methods, such as channel splicing or simple addition, often rely on the network's own fitting ability to allocate weights. This implicit learning method is prone to getting stuck in local optima when dealing with extreme illumination differences, leading to incomplete decoupling. This method proposes a luminance loss attention algorithm based on physical priors. This algorithm is designed as an explicit supervision strategy during the training phase. By constructing an adaptive illumination-aware mask, strong physical constraints are introduced in the image fusion and loss calculation stages, thereby guiding the network to achieve accurate decoupling of complex illumination scenes at the parameter learning level. The luminance loss attention algorithm includes constructing an adaptive luminance-aware mask and mask-based image fusion. The specific process is as follows: S31. To transform the physical illumination characteristics of the image into a computable supervision signal, the luminance loss attention algorithm first generates a binary mask that can accurately indicate the distribution of bright and dark areas. The process includes the following steps, aimed at extracting robust illumination structure information from noisy low-light images: S311, Luminance component extraction and normalization: Assume the input of the dual-branch network is a low-resolution image. The luminance component Y of the image is extracted using a weighted method and then normalized to eliminate the interference of color information on illumination assessment. ;in for Pixel coordinates, R is The red channel, G is The green channel, B is The blue channel; according to the weighting proposed in the ITU-R BT.601 standard, RGB is converted into a luminance map. This formula converts a color image into grayscale luminance values, that is, from the RGB color space to a grayscale image. The weighting coefficients represent the sensitivity of the human eye to different colors: the human eye is most sensitive to green, with a weight of 0.587; second most sensitive to red, with a weight of 0.299; and least sensitive to blue, with a weight of 0.114.

[0024] S312. Global Illumination Threshold Estimation: To adaptively distinguish between relatively bright and dark areas in different scenes, the average brightness of the entire image is calculated. As a dynamic segmentation threshold: Where H and W are respectively The height and width; this dynamic threshold ensures that the algorithm is well adapted to images with different exposure levels (such as overall dark or overall bright).

[0025] S313. Local Smoothing Filtering: Low-light images are often accompanied by severe shot noise. Directly thresholding the pixel-level brightness Y results in a broken, discrete, and speckled mask, disrupting the semantic continuity of the illuminated area. A block-based smoothing strategy is introduced for local smoothing filtering, specifically dividing the brightness map Y into non-overlapping blocks. Local neighborhood patches, denoted as Calculate each Average brightness within: ;in Indicates a local area The average brightness within; using Alternative The smoothed brightness distribution map is obtained from all pixel values ​​within the area. In this embodiment, k=4 is chosen based on experience. Selecting k=4 is optimal for simultaneously suppressing noise variance and maintaining edge sharpness. During the verification process, when k<4, the number of local neighboring pixel samples is insufficient, making it difficult to eliminate the disruption of mask semantic continuity caused by discrete spots; when k>4, an excessively large smoothing window can lead to visual blurring in the illumination transition area, resulting in a loss of local light and shadow depth. Choosing k=4 ensures mask spatial continuity while preserving the structural features of the image to the greatest extent. Local smoothing filtering effectively suppresses noise and accurately reflects the illumination level of each patch.

[0026] S314. Mask Binarization Generation: Based on Global Thresholding Smooth brightness map Perform binarization to generate the final brightness attention mask. : To visually verify the effectiveness of the mask generation algorithm described above, this embodiment visualizes the binarized mask of the intermediate process, as shown in Figure 5. The white area accurately covers the bright light strip under the eaves, the illuminated wall surface, and the window light source, with clear edge contours. Furthermore, due to the introduction of the Patch smoothing mechanism, the mask is coherent internally, without any broken noise or voids. The black area accurately includes the pitch-black night sky background and the shadow areas of the building's backlight.

[0027] The visualization results show that the proposed mask generation algorithm can robustly map the illumination distribution in complex scenes into a binary structure, providing an accurate spatial reference for the subsequent dual-branch network to handle the "suppression" and "brightening" tasks respectively.

[0028] S32. Mask-based image fusion: S321. Using mask M to process the highlight branch feature map output by the dual-branch network. Prediction chart and low-brightness branch feature map Prediction chart Spatial gated fusion is performed to generate low-resolution intermediate enhancement results. : ;in S322 represents element-wise multiplication; intermediate results... Compared with the downsampled real reference image Supervise and construct low-resolution reconstruction loss. : ;in, S323. Calculate the loss function according to the chain rule. (The L1 norm is represented by the sum of absolute errors.) The partial derivative of the bi-branch output, i.e., the error gradient: ; ;in, The sign function returns 1 for a positive input, -1 for a negative input, and 0 for a zero input. This gradient gating mechanism mathematically enforces the division of labor within the two-branch network. The trained backbone network parameters internalize the ability to respond to specific lighting features, achieving robust decoupling enhancement without additional computational overhead.

[0029] The gradient gating mechanism described above aims to force the dual-branch network to learn the ability to perceive illumination distribution during training. To verify whether this decoupling strategy is truly effective, i.e., whether the network still has the ability to distinguish between bright and dark regions and perform differential processing after the explicit mask input is removed during the inference phase, this embodiment performs a visualization analysis of the feature map before the last convolutional layer of the trained dual-branch network.

[0030] As shown in Figure 6, three scenes with typical high dynamic range lighting characteristics were selected, and feature heatmaps of the low-brightness and high-brightness branches were drawn using Class Activation Mapping (CAM) technology. The closer the color in the heatmap is to red (the higher the value), the higher the attention that branch pays to the corresponding region in the image. Observing the three sets of samples in the figure, it can be found that the thermal responses of the two branches exhibit significant spatial complementarity.

[0031] The low-brightness branch precisely focuses on the low-light areas of the image. For example, in Figure 6(a), this branch focuses on the shadows of the building's eaves and the backlit side of the red pillars; in Figure 6(c), it focuses on the dark walls in the lower left corner and the plaque area in the center. This demonstrates that the low-brightness branch has successfully learned the task of "brightening dark areas," automatically locating and enhancing details lost due to underexposure.

[0032] The high-brightness branch focuses primarily on the bright areas of the image. In Figure 6(b), this includes the sky and brightly lit walls directly exposed to sunlight; while in Figure 6(a), less attention is paid to the dark areas, with only the bright sky above and reflective areas receiving high attention. This indicates that the high-brightness branch focuses on "brightness fine-tuning," aiming to prevent over-enhancing of relatively normally exposed areas.

[0033] The heatmaps described above were generated during maskless inference, demonstrating that after training with the proposed brightness loss attention algorithm, the parameters of the dual-branch network have successfully internalized the perceptual logic of illumination intensity. The network no longer relies on external mask inputs but can spontaneously allocate processing tasks to the corresponding branches based on the texture and brightness statistical features of the input image, thus achieving robust decoupling at the physical level.

[0034] S4. Although pixel-level reconstruction losses based on L1 or MSE can recover the structural information of an image, single pixel supervision often tends to produce results with flat colors and insufficient contrast, making it difficult to restore the natural lighting texture that conforms to human visual aesthetics. In order to introduce semantic-level perceptual constraints in the low-light enhancement task, the CLIP multimodal optimization module and multimodal semantic constraint mechanism are introduced for auxiliary supervision during the training phase. They only apply to the low-resolution enhanced intermediate images output by the dual-branch network, aiming to use the low-dimensional feature space to correct global color and exposure style, while avoiding excessive semantic constraints from causing adversarial interference with the pixel-level detail recovery target in the high-resolution reconstruction stage, and serving as an auxiliary means to promote the decoupling of the dual branches. The multimodal optimization module is shown in Figure 7. The implementation process of the CLIP multimodal semantic constraint mechanism is as follows: S41. Construct several sets of discriminative positive and negative text pairs through prompt word engineering, and set positive prompt words for positive text. Negative prompts in negative text Examples of positive text: "a vibrant, well-exposed photo"; examples of negative text: "a dark, underexposed photo". S42. During training iterations, low-resolution images are enhanced. The text pairs are mapped to the same feature space using CLIP's image encoder and text encoder, respectively, and their cosine similarity is calculated. S43. Construct a combined loss function that includes absolute distance constraints and relative probability constraints, namely, multimodal semantic loss. : ;in, Let represent the cosine similarity between A and B; By minimizing the similarity difference, the image is brought closer to the positive text, while the image is pushed further away from the negative text. The similarity is transformed into a probability distribution using a Softmax-based probabilistic penalty term. This is achieved through contrastive learning, where the higher the probability of an image being misidentified as a low-quality sample, the greater the penalty loss. This combined constraint forces the dual-branch network to establish correct color tone and exposure levels at low resolution, laying a high-quality semantic foundation for subsequent high-resolution detail reconstruction.

[0035] S5. To ensure the final output image has high-resolution fine texture, the semantically optimized low-resolution enhancement features are input into the RCAB super-resolution module to reconstruct the high-resolution enhanced image. The structure of the RCAB super-resolution module is shown in Figure 8. This module participates in both the training and inference stages of the network, aiming to efficiently map the low-resolution illumination features output by the dual-branch decoupled network back to the high-resolution image space. The RCAB super-resolution module performs the following: S51. Highlight recovery features output by the dual-branch network. Low brightness recovery features The features are stitched together along the channel dimension to form a fused feature body containing complete illumination information; S52, the stitched features undergo preliminary feature fusion through a 3×3 convolutional layer, and the Tanh activation function is introduced to constrain the feature distribution. Within the interval; this serves to normalize features and stabilize the training gradient of subsequent deep networks.

[0036] S53. Deep feature extraction is performed using several cascaded residual channel attention blocks (RCABs). Each RCAB unit integrates a residual learning path and a channel attention mechanism, adaptively adjusting the feature response weights based on the correlation between channels. This allows the network to focus on key feature channels that contribute to the reconstruction of high-frequency information while ignoring redundant information. Global long skip connections are introduced to address the gradient vanishing problem in deep networks and to pass low-frequency structural information. Shallow features are directly added to deep output features, achieving efficient learning of residual features.

[0037] S54. After completing deep feature fusion, to restore the image to its original high-resolution size, a PixelShuffle layer is used for upsampling. This rearranges the multi-channel feature maps into spatial pixel blocks, utilizing channel information to generate high-resolution details and avoiding checkerboard artifacts. S55. The upsampled features are then passed through a reconstruction convolutional layer to output the final high-resolution enhanced image. After S5, the RCAB super-resolution module successfully decouples the tasks of "lighting / color restoration" and "detail / texture reconstruction". That is, the dual-branch network completes the decoupling and restoration of physical lighting in low-resolution space. The RCAB super-resolution module focuses on super-resolution reconstruction of high-fidelity textures, realizing large-size low-light image enhancement with low computational cost.

[0038] S6. An end-to-end joint training strategy is adopted, using a total loss function composed of multi-scale pixel-level loss and multi-modal semantic loss to supervise network training, achieving efficient enhancement of low-light images. Total loss function... Defined as the weighted sum of the three sub-losses: ;in, , , These are weighting coefficients; in this embodiment, they are taken as 1, 0.1, and 1 respectively. and As a baseline weight, the reference empirical value is set to 1 to ensure that the enhanced image is more consistent with the real reference image and will not produce serious artifacts. Significantly smaller than and At the same time, the model can effectively take into account both low-level visual indicators and high-level semantic features. The values ​​of 1, 0.1, and 1 used in this embodiment are steady-state optimal solutions obtained from training on the dataset, which can enable the network to achieve synergy in convergence speed and generation quality. Calculate intermediate results for low-resolution reconstruction loss. Compared with the downsampled real reference image The L1 distance between them; this loss is achieved through the gradient gating mechanism of the brightness mask, which forces the dual-branch network to decouple and restore physical illumination in a low-dimensional space. To achieve multimodal semantic loss, a pre-trained CLIP model is used to constrain the color and style of low-resolution enhanced images, making them approximate high-quality text descriptions in the feature space, thereby introducing prior information from human visual perception. The high-resolution reconstruction loss is used to supervise the output of the RCAB super-resolution module and calculate the final generated high-resolution enhanced image. Compared to the original high-resolution real image The L1 pixel error between them is calculated using the following formula: ; It focuses on constraining the high-frequency texture and edge details of the image to ensure that the final output has high-fidelity structural information while maintaining decoupled lighting.

[0039] Through these three losses, the low-light enhancement network based on dual-branch decoupling and multimodal awareness can achieve the best balance between "physical correctness", "semantic naturalness" and "detail clarity". Finally, the backpropagation algorithm is used to update all parameters of the dual-branch backbone network and the RCAB super-resolution module simultaneously, achieving robust end-to-end enhancement. The training process of the network is shown in the table below:

[0040] To test the effectiveness and robustness of this method, this embodiment is validated on three public datasets with different characteristics.

[0041] (1) Dataset Introduction and Experimental Setup This embodiment selects three representative low-light image datasets, covering real scenes, synthetic scenes, and high-resolution low-light road scenes, to evaluate the performance of this method. ① LOL-v1 Dataset (Real Scene): This dataset contains 500 pairs of real extremely low-light / normal-light images with a resolution of 600x400. 485 pairs are used for training, and 15 pairs are used for testing. Since this dataset consists of real images with low resolution, the main focus is on evaluating the model's ability to enhance and restore colors in extremely low light conditions.

[0042] ②LOL-v2 Dataset (Synthetic Scenes): This dataset contains 1000 image pairs, with 900 pairs used for training and 100 pairs used for testing, at a resolution of 384x384. This dataset is primarily used to evaluate the model's ability to adjust lighting in synthetic low-light environments.

[0043] ③ ACDC-syn dataset (high-resolution scene): This dataset contains 400 pairs of training images and 500 pairs of test images, with a resolution of 1920x1080. Due to the high resolution of this dataset, it is downsampled by four times and used as input for various model methods to verify the ability of this method to process high-resolution low-light images.

[0044] For datasets with different resolutions, this embodiment employs an adaptive experimental strategy to verify the generality of the architecture: ① End-to-end enhancement mode (for LOL-v1 / v2): Given the relatively small resolution of these two datasets, no additional downsampling operation is required. Therefore, the network configuration is adjusted to directly perform end-to-end feature mapping and image generation, focusing on verifying the enhancement effect of the dual-branch decoupling network and CLIP semantic constraints.

[0045] ② Combined Enhancement and Super-Resolution Mode (for ACDC-syn): Given the high resolution of this dataset, to reduce computational load while verifying the function of the RCAB module, the original image is downsampled by 4 times as input. This embodiment not only needs to restore illumination and color but also needs to reconstruct the image back to its original high resolution, fully leveraging the combined advantages of "low-light enhancement + super-resolution".

[0046] (2) Quantitative Analysis: To objectively verify the advancement of the proposed method, this embodiment conducted rigorous quantitative tests on three mainstream low-light image datasets (LOL-v1, LOL-v2, ACDC-syn). The experiment selected nine SOTA (State-of-the-Art) algorithms published in top conferences and journals in recent years (2022-2025) for comparison, including SNR, IAT, Restormer, Retinexformer, SHAL-Net, RFLLIE, IDB, and Uretinex++.

[0047] The evaluation metrics used are the commonly used PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity). Specific quantitative comparison data are shown in Table 1. Table 1: Comparison of metrics across different low-light image datasets.

[0048] LOL-v1: On the real-world extremely low-light dataset LOL-v1, this method demonstrates exceptional structure recovery capabilities. While Retinexformer and RFFLIE slightly outperform each other on the PSNR metric, this method ranks first with a score of 0.872 on the SSIM metric, which better reflects human visual perception, significantly outperforming Retinexformer (0.845) and RFFLIE (0.848). PSNR is sensitive to absolute pixel error, while SSIM is sensitive to edges and textures. This result indicates that this method performs better in removing real-world noise and preserving object edges, resulting in a clearer visual appearance. This is attributed to the effective noise suppression achieved through the attention mechanism in the high and low brightness branches of this method.

[0049] LOL-v2: On the more complex lighting environment of the LOL-v2 dataset, our method achieves a PSNR of 26.14 dB and an SSIM of 0.960. Compared to the FLW algorithm in 2023 (PSNR 26.09 dB) and the RFLLIE algorithm in 2024 (SSIM 0.952), our method maintains a leading edge. This demonstrates that the CLIP multimodal semantic constraint proposed in our method plays a crucial role in complex lighting adjustments, enabling the generated image to maintain high brightness while not introducing artifacts, and instead possessing extremely high structural integrity (SSIM close to 0.96).

[0050] ACDC-syn: This method demonstrates a significant performance advantage on the ACDC-syn dataset. It achieves a PSNR of 28.68 dB and an SSIM of 0.882. Compared to the latest IDB (TITS'25) dataset in 2025 (PSNR 27.59 dB), this method represents a 1.09 dB improvement. ACDC-syn typically involves high-resolution or synthetic scenes. Other comparative algorithms, such as Retinexformer and Uretinex++, mostly focus on brightness adjustment and lack the ability to reconstruct high-frequency details. However, this method, by introducing an RCAB super-resolution reconstruction module and a dual-branch decoupled architecture, can simultaneously achieve "illumination enhancement" and "detail super-resolution," thus significantly surpassing existing technologies in pixel accuracy and structure restoration.

[0051] (3) Qualitative analysis In order to intuitively verify the technical effect of this method in image enhancement and detail reconstruction, this embodiment visually compares it with the existing technology in a real low-light scene (LOL-v1), a synthetic low-light scene (LOL-v2), and a night driving scene (ACDC-syn) that are degraded by four times downsampling.

[0052] Figures 9, 10, and 11 all use a layout that compares the full image with local details: columns 1, 3, and 5 show the enhanced full image, while columns 2, 4, and 6, which are immediately to the right of them, correspond to magnified views of the areas within the red boxes in the figures, clearly revealing the differences in texture restoration between the various methods.

[0053] In the extreme 4x downsampling scenario (Figure 9), the input image suffers severe information loss, posing a significant challenge to the algorithm's robustness. Observing the overall image in column 1, the images generated by contrast methods (such as Restormer, FLW, and URetinex++) are not only generally blurry but also accompanied by obvious blocky noise and artifacts, severely affecting the image's structural information. The full image generated by our method still maintains a clear geometric structure and a clean background, without obvious noise amplification. A magnified view further confirms this advantage: as shown in column 2, faced with severe pixel loss, the number "11" recovered by methods such as Restormer and Retinexformer almost completely degenerates into a blurry color block, lacking readability. Even IDB, with its super-resolution capabilities, exhibits certain artifacts and distortions in its results. Our method successfully reconstructs the stroke structure of the number "11," with a smooth background and clear white dots. Similarly, in the electronic screen scene of "09:33" in column 4, our method is the only model that can relatively clearly restore every digit detail, fully demonstrating its reconstruction potential under extremely low resolution conditions.

[0054] Unlike the relatively simple lighting distribution of the LOL-v1 dataset, the LOL-v2-syn dataset contains more complex and varied lighting environments, and the image size is smaller (384×384). This requires the model to have extremely strong lighting decoupling and local contrast enhancement capabilities, rather than simply relying on global brightness enhancement. As shown in Figure 10, in all three scenes, the output of Uretinex+ appears completely white due to severe lighting correction errors, and the image content is completely destroyed. This also verifies the vulnerability of the traditional global Retinex model when dealing with highly dynamic synthetic data. Excluding the failed model, further comparing the performance of other algorithms in color restoration and texture reconstruction, the advantages of this method are more prominent: in the forest details in the second column, color accuracy is the biggest challenge. Observing the enhancement results of IDB, SNR, RFLLIE, and Retinexformer, not only are the leaf textures blurred, but there is also a significant tone shift, and the overall picture presents an unnatural withered yellow. In contrast, this method not only clearly restores the texture of branches and leaves, but also accurately replicates the emerald green hue in the reference image, demonstrating extremely high color fidelity. For traffic sign areas containing geometric shapes (column 4), the image generated by this method has the sharpest edges, with vivid red and blue color matching and no overflow. In contrast, most comparison algorithms suffer from edge blurring or dull colors. Regarding the restoration of children's clothing colors (column 6), most comparison algorithms exhibit color distortion here; even the relatively better-performing SNR algorithm restores a noticeably pale and whitish color in the restored pants. This method accurately restores the deep, rich color texture of the clothing, thanks to the effectiveness of the multimodal optimization module in guiding semantic-level color restoration.

[0055] Addressing the challenge of recognizing minute targets in nighttime driving scenarios, as shown in Figure 11, observing the magnified images of all the comparative methods (including SNR, IAT, Restormer, FLW, Retinexformer, etc.) reveals a common flaw—severe mosaic effect and jaggedness. In the blue straight-ahead and right-turn sign in column 2, the aforementioned methods recover a blurry block of pixels inside the sign, with the arrow indicating the direction unclear. This is unacceptable in applications with extremely high safety requirements, such as autonomous driving. In the red no-stopping signs in columns 4 and 6, methods such as Retinexformer and URetinex++ can only recover red blocks, with severely distorted circular geometric contours and numerous artifacts at the edges. In contrast, our method demonstrates strong detail restoration capabilities. In the blue sign scene (column 2), our method not only clearly restores the circular outer contour of the sign but also accurately reconstructs the internal white arrow guidance with sharp edges and no blurring caused by interpolation. In the red logo scenario (columns 4 and 6), even with extremely limited input information, this method still accurately restored the logo's bright red background and blue background while maintaining a certain structure.

[0056] Qualitative analysis results show that this method not only effectively solves the problems of uneven exposure and color distortion in low light environments, but also breaks through the resolution bottleneck of traditional enhancement algorithms on distant small targets, and has significant technical advantages in visual perception quality.

[0057] Therefore, this invention employs the aforementioned low-light image enhancement method based on a dual-branch decoupling network and multimodal perception, constructing a dual-branch decoupling architecture with a channel self-attention recovery network as its backbone. This architecture includes a high-brightness attention branch and a low-brightness attention branch, which perform differentiated restoration of bright and dark regions in the image, respectively. To achieve accurate decoupling, a brightness loss attention algorithm is designed, utilizing an adaptive masking strategy to guide each branch to focus on feature extraction in specific illumination areas. Simultaneously, combined with CLIP multimodal loss, textual semantic information is used to guide the generated image towards a high-quality visual style. Finally, the RCAB super-resolution module performs deep fusion and upsampling of the dual-branch features, outputting a clear, high-resolution enhanced image.

[0058] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A low-light image enhancement method based on a dual-branch decoupling network and multimodal sensing, characterized in that, Includes the following steps: S1. Downsample the input low-light image to obtain a low-resolution low-light image; S2. Input the low-resolution low-light image into a dual-branch decoupling module, using two structurally identical but parameter-independent sub-networks as high-brightness and low-brightness branches respectively, extracting high-brightness and low-brightness features and generating enhanced feature maps and prediction maps respectively; the sub-networks use a U-shaped channel-level self-attention recovery network as the backbone, integrating a Gamma pre-enhancement module and a channel self-attention module; S3. An adaptive brightness-aware mask is generated using a brightness loss attention algorithm. The mask is then used to perform spatial gating fusion on the prediction map output by the dual-branch network to obtain a low-resolution intermediate enhanced image. S4. Introduce the CLIP multimodal semantic constraint mechanism to apply semantic loss supervision to the low-resolution intermediate enhanced image and optimize the color and style of the image; S5. Input the semantically optimized low-resolution enhanced features into the RCAB super-resolution module to reconstruct the high-resolution enhanced image; S6. Adopt an end-to-end joint training strategy, supervise the network training through the total loss function composed of multi-scale pixel-level loss and multimodal semantic loss, and achieve efficient enhancement of low-light images.

2. The low-light image enhancement method based on a dual-branch decoupling network and multimodal perception according to claim 1, characterized in that, In S2, the U-shaped channel-level self-attention recovery network adopts an encoder-decoder structure. The encoder consists of three stages, each containing a convolutional layer and a channel self-attention module. It gradually reduces the spatial resolution of the feature map through a downsampling operation with a stride of 2, while doubling the number of channels. The decoder symmetrically upsamples through deconvolutional layers and introduces lateral skip connections to concatenate the feature maps of the corresponding layers of the encoder and the feature maps of the decoder in the channel dimension to achieve the fusion of low-level texture information and high-level semantic information. The lowest resolution layer of the U-shaped channel-level self-attention recovery network stacks two channel self-attention modules to capture long-distance global dependencies.

3. The low-light image enhancement method based on a dual-branch decoupling network and multimodal perception according to claim 2, characterized in that, In S2, the implementation process of the Gamma pre-enhancement module is as follows: the input features are initially extracted through a 3×3 depthwise separable convolution and a Group Norm layer; compressed into a global feature vector through a 1×1 convolution and global average pooling; and an adaptive Gamma coefficient is generated using the Sigmoid activation function and applied to the original image.

4. The low-light image enhancement method based on a dual-branch decoupling network and multimodal perception according to claim 3, characterized in that, In S2, the execution of the channel self-attention module includes the following steps: S21, setting the input feature as... The query matrix Q, key matrix K, and value matrix V are generated through 1x1 convolution and reshape operations. S22. Calculate the dot product of K and Q after transpose to generate a channel covariance matrix of size C×C. The calculation formula is as follows: , where d is the scaling factor; S23, after adding the result of S22 to the value matrix V, input it into the gated feedforward network, and then expand the channels through pointwise convolution, depthwise convolution and GELU activation function to enhance the network's nonlinear expression ability of local detail features; then after pointwise convolution dimensionality reduction and residual connection, the enhanced features are output.

5. The low-light image enhancement method based on a dual-branch decoupling network and multimodal perception according to claim 4, characterized in that, In S3, the brightness loss attention algorithm includes constructing an adaptive brightness-aware mask and mask-based image fusion. The specific process is as follows: S31, Constructing an adaptive brightness-aware mask: S311, Brightness component extraction and normalization: Assume the input of the dual-branch network is a low-resolution image. The luminance component Y of the image is extracted using a weighted method and then normalized. ;in for Pixel coordinates, R is The red channel, G is The green channel, B is The blue channel; S312, Global Illumination Threshold Estimation: Calculate the average brightness of the entire image. As a dynamic segmentation threshold: Where H and W are respectively Height and width; S313, Local smoothing filtering: A block smoothing strategy is introduced for local smoothing filtering, specifically, the brightness map Y is divided into non-overlapping blocks. Local neighborhood patches, denoted as Calculate each Average brightness within: ;in Indicates a local area The average brightness within; using Alternative The smoothed brightness distribution map is obtained from all pixel values ​​within the area. S314, Mask Binarization Generation: Based on Global Thresholding Smooth brightness map Perform binarization to generate the final brightness attention mask. ： S32. Mask-based image fusion: S321. Using mask M to process the highlight branch feature map output by the dual-branch network. Prediction chart and low-brightness branch feature map Prediction chart Spatial gating fusion is performed to generate low-resolution intermediate enhancement results. ： ;in S322 represents element-wise multiplication; intermediate results... Compared with the downsampled real reference image Supervise and construct low-resolution reconstruction loss. ： ;in, S323. Describe the L1 norm; S323. Calculate the loss function according to the chain rule. The partial derivative of the bi-branch output, i.e., the error gradient: ； ;in, This represents a sign function; it returns 1 for a positive input, -1 for a negative input, and 0 for a zero input.

6. The low-light image enhancement method based on a dual-branch decoupling network and multimodal perception according to claim 5, characterized in that, In S4, the implementation process of the CLIP multimodal semantic constraint mechanism is as follows: S41, construct several sets of discriminative positive and negative text pairs through prompt word engineering, and set positive prompt words for positive text. Negative prompts in negative text S42. During the training iteration, the low-resolution image is enhanced. The text pairs are mapped to the same feature space using CLIP's image encoder and text encoder, respectively, and their cosine similarity is calculated. S43. Construct a combined loss function that includes absolute distance constraints and relative probability constraints, i.e., multimodal semantic loss. ： ;in, Let represent the cosine similarity between A and B; By minimizing the similarity difference, the image is brought closer to the positive text, while the image is pushed further away from the negative text. The probability penalty term is based on the Softmax form. It uses the idea of ​​contrastive learning to transform similarity into a probability distribution. The higher the probability that an image is misidentified as a low-quality sample, the greater the penalty loss.

7. The low-light image enhancement method based on a dual-branch decoupling network and multimodal perception according to claim 6, characterized in that, In S5, the RCAB super-resolution module performs the following: S51, recovers the highlight features output by the dual-branch network. Low brightness recovery features The features are stitched together along the channel dimension to form a fused feature body containing complete illumination information; S52, the stitched features undergo preliminary feature fusion through a 3×3 convolutional layer, and the Tanh activation function is introduced to constrain the feature distribution. Within the interval; S53, deep feature extraction is performed through several cascaded residual channel attention blocks, global long skip connections are introduced to solve the gradient vanishing problem in deep networks and to pass low-frequency structural information, and shallow features are directly added to deep output features; S54. Upsampling is performed using a PixelShuffle layer to rearrange the multi-channel feature maps into spatial pixel blocks, generating high-resolution details using channel information and avoiding checkerboard artifacts; S55. The upsampled features are then passed through a reconstruction convolutional layer to output the final high-resolution enhanced image. 。 8. The low-light image enhancement method based on a dual-branch decoupling network and multimodal perception according to claim 7, characterized in that, In S6, the total loss function Defined as the weighted sum of the three sub-losses: ;in, 、 、 These are the weighting coefficients; Calculate intermediate results for low-resolution reconstruction loss. Compared with the downsampled real reference image The L1 distance between them; For multimodal semantic loss; The high-resolution reconstruction loss is used to calculate the final generated high-resolution enhanced image. Compared to the original high-resolution real image The L1 pixel error between them is calculated using the following formula: 。 9. A low-light enhancement network based on dual-branch decoupling and multimodal sensing, characterized in that, It includes a dual-branch decoupling module, a brightness loss attention module, a multimodal optimization module, and an RCAB super-resolution module. The dual-branch decoupling module is used to extract features from different illumination regions. The brightness loss attention module is used to generate an adaptive mask and achieve feature fusion. The multimodal optimization module is used to apply semantic constraints. The RCAB super-resolution module is used for high-resolution image reconstruction.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the steps of the low-light image enhancement method according to any one of claims 1-8.

Images