TLH-Net-based low illumination and fog superposition enhancement method

Abstract

The invention discloses a TLH-Net-based image enhancement method in a low-illumination and fog superposition scene. The method comprises the following steps: 1) data set construction: constructing a low-illumination superposition fog image data set; 2) low-illumination defogging network design: designing a three-branch low-illumination defogging network; decomposing the input image into a fog image, a reflection image and an illumination image; 3) designing a fog image estimation network: designing a fog layer decomposition network; 4) a reflectogram estimation network: designing a reflectogram estimation network, decomposing the reflectogram based on the Retinex theory and performing denoising; according to the method, the Retinex theory, the atmospheric scattering model and semantic guidance are fused, the problem of target information fuzziness caused by low illumination and fog superposition is effectively solved, and the adaptability and generalization ability of the model in different scenes are improved.

Description

A Low-Light and Fog Overlay Enhancement Method Based on TLH-Net Technical Field

[0001] This invention relates to dehazing enhancement and low-light image enhancement, specifically a low-light and fog superposition enhancement method that combines bi-branch, multi-head attention, cross-attention, layer decomposition, atmospheric scattering theory, and Retinex theory. Background Technology

[0002] In the fields of computer vision and intelligent surveillance, the combined effect of low-light environments and fog / haze is a significant challenge to image quality degradation. Under low-light conditions, the light signal intensity captured by the sensor is insufficient, leading to increased noise, loss of detail, and color distortion in the image. Furthermore, the scattering effect of fog further reduces scene visibility, causing decreased contrast, uneven brightness distribution, and blurred target edges. The superposition of these two degradation factors is not a simple linear combination, but rather exacerbates imaging difficulties through complex physical interactions (such as the coupling of atmospheric scattering models and light attenuation), severely impacting the robustness of algorithms, especially in practical applications such as autonomous driving, nighttime surveillance, and remote sensing.

[0003] Imaging quality in low-light environments is limited by illumination, leading to a significant decrease in signal-to-noise ratio (SNR), and Poisson noise and quantization noise are prone to appearing in dark areas of the image. Simultaneously, the scattering effect of fog introduces atmospheric light components, causing scene radiation to attenuate and shift during propagation. When low light and fog work together, fog scattering further weakens the already weak reflected light from targets, while the non-uniform distribution of ambient light (such as artificial light sources in urban nighttime) can lead to local overexposure or color distortion. This coupled degradation makes traditional image enhancement methods based on a single degradation assumption (such as histogram equalization or dehazing algorithms) difficult to apply, and may even introduce artifacts due to incorrect prior assumptions. In autonomous driving systems, low-light scenarios combined with fog (such as foggy nighttime conditions) severely affect the collaborative perception capabilities of LiDAR and cameras, resulting in a decrease in obstacle detection rates. In the field of security monitoring, pedestrian re-identification (Re-ID) and target tracking tasks in such environments face the problem of feature extraction failure. In remote sensing image analysis, the superposition of cloud cover and insufficient lighting can reduce the accuracy of land surface classification. Especially in scenarios with high timeliness requirements, such as disaster emergency response, the degradation of image quality may directly affect decision-making efficiency.

[0004] Existing image datasets typically address low-light or haze issues individually, lacking comprehensive datasets that simultaneously incorporate both degradation factors. This deficiency renders existing image processing methods ineffective in handling complex low-light and foggy real-world scenes, resulting in image quality unsuitable for training robust deep learning models. To address this problem, this invention proposes using Retinex theory for physical modeling to synthesize a comprehensive dataset, Foggy-Cityscapes-Dark, that includes both low-light and haze conditions. This dataset effectively compensates for the shortcomings of existing data, providing a more realistic data foundation for developing more advanced low-light and foggy image enhancement processing.

[0005] Existing low-light enhancement methods often focus solely on increasing image brightness and contrast, neglecting the preservation of image details and noise suppression, resulting in low-quality enhanced images. Many enhancement methods lack flexibility and adaptability when processing low-light images in different scenes, making it difficult to achieve consistent results in diverse environments. Existing low-light enhancement methods generally suffer from two major shortcomings: detail-noise imbalance and lack of scene adaptation. Regarding the detail-noise imbalance problem, most methods focus on global brightness stretching and contrast adjustment (such as histogram equalization and gamma correction), but neglect the synergistic optimization of high-frequency detail preservation and noise amplification suppression. Such methods, based on manual design and complex priors, are prone to problems such as loss of key texture information, excessive amplification of noise in dark areas, and local overexposure. Regarding the lack of scene adaptation, existing methods often employ fixed-parameter models (such as Retinex's fixed-scale Gaussian kernel) or static mapping functions, lacking the ability to dynamically perceive the degradation characteristics of complex scenes. Such methods are prone to failure in handling non-uniform lighting scenes (such as the coexistence of neon lights and shadows in nighttime street scenes), drastic performance degradation in extremely low-light (<0.1 lux) and mixed noise (Gaussian-Poisson noise coupling) scenes, and weak cross-scene generalization. These shortcomings directly limit the usability of the enhancement results for downstream tasks. This invention develops a NE-DCE dual-branch network. The noise extraction branch uses attention mechanisms and adaptive dilated convolutions to extract local and global features and uses random parameters for Gamma Correction to simulate the coupling of noise and illumination during image enhancement, thereby improving robustness. The illumination enhancement branch, based on the Zero-DCE network, uses a brightness enhancement curve to fit the enhancement process.

[0006] Traditional dehazing methods suffer from poor generalization ability. Artificially designed and complex methods based on dark channel priors (DCP), atmospheric scattering models, etc., are prone to inaccurate transmittance estimation in dense fog areas, misjudgment of atmospheric light caused by bright objects (white vehicles, sky), and color shifts in real-world complex scenes. Furthermore, traditional methods rarely utilize attention mechanisms and lack the ability to perceive spatial adaptive degradation. To address these issues, this invention designs the SCH Dehazing King Network, which consists of two parts. The first part, a layer decomposition network based on an atmospheric scattering model, extracts the fog image. The dehazed image is then further decomposed into illumination and reflection maps by a second part based on Retinex theory, thereby improving image quality. Summary of the Invention

[0007] This invention addresses the problem of poor image enhancement effects under low light and fog conditions by constructing a low light and fog enhancement method based on TLH-Net. First, a low light fog image dataset Foggy-Cityscapes-Dark is synthesized. Then, a dedicated low light enhancement network NE-DCE and a defogging module SCH are designed to conduct a training process for low light fog enhancement.

[0008] 1. A low-light and fog overlay enhancement method based on TLH-Net, TLH-Net is shown in Figure 1, characterized by the following steps:

[0009] (1) Construct the low-light and foggy image dataset foggy-Cityscapes-Dark, including the following steps:

[0010] (1.1) Based on the publicly available Cityscapes dataset, which mainly contains 5000 high-quality pixel-level annotated images of driving scenes in urban environments from 50 different city street scenes, including 2975 training set images, 500 validation set images, and 1525 test set images, with a total of 19 categories. Foggy-Cityscapes is a synthetic dataset generated by Cityscapes, designed to simulate and study the performance of autonomous vehicles in foggy conditions. It has three levels of fog (0.005, 0.01, 0.02), from light fog to dense fog, corresponding to visibility ranges of 600, 300, and 150 meters, respectively.

[0011] (1.2) Use the EC-Zero-DCE network to reduce the brightness of each image in the Foggy-Cityscapes dataset to generate Foggy-Cityscapes-Dark.

[0012] (1.3) Preprocess the Foggy-Cityscapes-Dark image dataset by scaling all images to 640x640x3 pixels to facilitate subsequent processing.

[0013] (2) Design the fog map estimation network HE-Net (Haze Estimator Net) as shown in Figure 2, including the following steps:

[0014] (2.1) The image preprocessed in (1.3) is fed into the fog map estimation network HE-Net.

[0015] (2.2) The image output by (1.3) is processed through a 3×3 convolutional layer to extract local fog concentration features. Then, ReLU is used to enhance the feature expression ability by capturing long-distance fog distribution dependence through Multihead-Attention. The feature is deepened by looping 8 times. Finally, a transmittance map is generated by passing through a 1×1 convolutional layer and a Sigmoid layer.

[0016] (2.3) The image output by (1.3) is processed by adaptive average pooling to output a 1×1×C vector, then processed by a 3×3 convolutional layer to learn the relationship between channels and fed into ReLU to enhance the expressive power, then processed by a 1×1 convolutional layer to output (RGB) 3 channels, and finally processed by a Sigmoid layer to generate an atmospheric light map.

[0017] (2.4) Use the atmospheric scattering model to generate a fog map from the output transmittance map and atmospheric light estimation map.

[0018] (3) Design the dual-branch reflection estimation network DRE-Net (as shown in Figure 3), which includes the following steps:

[0019] (3.1) Feed the image output by (1.3) into the dual-branch reflectance estimation network DRE-Net.

[0020] (3.2) Perform Pair-Down Sampling on the image preprocessed in (1.3). This process divides the image into N 2x2 pixel blocks, randomly collects two adjacent pixels in each pixel block, and then outputs image 1 and image 2.

[0021] (3.3) Extract local spatial features from image 1 output by (3.2) through a 3x3 convolution. Feed the atmospheric light estimation map output by (2.3) into Cross-Attention to allow the network to adaptively learn the atmospheric light and illumination weights of different regions, then pass through a RuLU, and repeat 8 times. Finally, output the reflectance map through a 1x1 convolutional layer and a Sigmoid layer.

[0022] (3.4) Image 2 output from (3.2) is subjected to gamma correction. This process randomly selects a parameter between 1.2 and 1.5 to exponentially increase the image brightness to simulate noise under different lighting conditions. A 3×3 convolutional layer is then applied to extract primary spatial features, followed by a Leaky ReLU activation function to introduce nonlinearity and alleviate the gradient vanishing problem. Another 3×3 convolutional layer + Leaky ReLU is used to further deepen feature extraction. The residuals are then connected to a 7×7 adaptive dilated convolution to expand the receptive field and capture broader contextual information. Simultaneously, the dilation rate is adjusted to adapt to features at different scales, avoiding resolution loss. Finally, a sigmoid function is applied to output a noise map.

[0023] (3.5) A reflection map obtained by fusing the reflection map output by (3.3) and the noise map output by (3.4).

[0024] (3.6) The reflection map output from (3.5) is fed into the detail enhancement network as shown in Figure 4. Local detail information is captured through a 3x3 convolutional layer, followed by a 32-layer residual network to capture deeper detail information, and then another 3x3 convolution. Finally, skip connections are used to prevent overfitting.

[0025] (4) Design the Illumination Estimation Network (LUE-Net) as shown in Figure 5, which includes the following steps:

[0026] (4.1) The image output from (1.3) is fed into the dual-branch dehazing network reflection map estimation network DRE-Net.

[0027] (4.2) Local illumination features are extracted through a 3x3 convolutional layer, global illumination relationships (such as shadow-highlight correspondence) are captured through Multihead-Attention, long-distance illumination dependencies are modeled, and then a RuLU is applied, and the process is repeated 8 times. Finally, the illumination map is output through a 1x1 convolutional layer and a Sigmoid function.

[0028] (4.3) The illumination map output from (4.2) is processed through four consecutive FT modules (as shown in Figure 7). Each FT module independently processes the features, gradually deepening the feature representation. The outputs of the third and fourth FT modules are concatenated in the second dimension and input into the fifth FT module. The outputs of the second and fifth FT modules are concatenated in the second dimension and input into the sixth FT module. The outputs of the first and sixth FT modules are concatenated in the second dimension and input into the FTT module (as shown in Figure 7). The FT module first passes through a 3x3 convolutional layer, then through a Self-Attention layer to obtain broader contextual information, followed by residual connections to prevent over-focusing on local information. Finally, a ReLU activation function is used to introduce nonlinearity and alleviate the gradient vanishing problem. The FTT module has a similar structure to the FT module, but the activation function is replaced with Tanh, which maps the output to [-1,1].

[0029] (5) Design the defogging network as shown in Figure 8, including the following steps:

[0030] (5.1) The atmospheric light estimation map and atmospheric projectivity map output by (2.4), the detailed enhanced reflectance map output by (3.6), and the illumination enhanced illumination map output by (4.3) are spliced ​​and input into the defogging network.

[0031] (5.2) First, a channel self-attention is used to learn the dependencies between channels. Then, a spatial self-attention is used to capture the spatial distribution pattern of fog and the spatial relationship of illumination differences in different regions. Then, a U-net network is used to extract, fuse and restore the image illumination and dehazing features. Finally, a 1x1 convolution and Tanh activation function are used to map the output to [-1,1]. Attached Figure Description

[0032] Figure 1: Overall flowchart of the present invention

[0033] Figure 2: HE fog estimation network structure diagram

[0034] Figure 3: Network structure diagram for DRE two-branch reflectogram estimation

[0035] Figure 4: Network structure diagram for enhanced detail

[0036] Figure 5: Network structure diagram for LUE illumination estimation

[0037] Figure 6: DCE Illumination Enhancement Network Structure Diagram

[0038] Figure 7: Structure diagram of FT feature extraction module 1 and FTT feature extraction module 2

[0039] Figure 8: Defogging network structure diagram

[0040] Figure 9: Flowchart of Semantic Information-Guided Low-Light Defogging Network Detailed Implementation

[0041] The TLH-Net network designed in this invention is shown in Figure 1. The TLH-Net network consists of two parts: the DN-DCE low-light enhancement network and the SCH defogging network, as shown in Figures 2 and 4, respectively.

[0042] Step 1: Construct the low-light, foggy image dataset Foggy-Cityscapes-Dark and perform image preprocessing.

[0043] (1.1) Cityscapes is a public dataset containing street scene images from 50 cities, with a total of 5000 high-quality pixel-level labeled images (2975 for training, 500 for validation, and 1525 for testing), covering 19 categories. Based on this, the Foggy-Cityscapes dataset simulates different foggy driving environments by synthesizing fog effects, providing three visibility levels (slight: 0.005 / 600 meters, moderate: 0.01 / 300 meters, dense fog: 0.02 / 150 meters) to study the performance of autonomous driving under foggy conditions.

[0044] (1.2) Use the EC-Zero-DCE network to reduce the brightness of each image in the Foggy-Cityscapes dataset to generate Foggy-Cityscapes-Dark.

[0045] (1.3) All images in the Foggy-Cityscapes-Dark dataset are scaled to a fixed resolution of 640×640×3 using bilinear interpolation to eliminate the size inconsistency of the original images caused by differences in urban scenes and improve the computational efficiency of subsequent modules.

[0046] Step 2: HE-Net Fog Map Estimation Network

[0047] (2.1) The output of (1.3) is input into the transmittance estimation network, as shown in Figure 2. The input first passes through a 3×3 convolutional layer, which is used to extract local fog concentration features, effectively extracting spatial information of the fog and haze region. After the initial convolutional feature extraction, the Multihead-Attention mechanism is used to capture long-distance fog and haze distribution dependencies. The multihead attention mechanism computes the attention distribution in parallel through multiple "heads", enabling it to learn the relationships between different regions in the image in different subspaces. Through this mechanism, the network can focus on the global dependencies between different image regions, especially the relationships between distant fog and haze regions, which is crucial for transmittance estimation. Then, the ReLU activation function is used to enhance the model's feature learning ability by introducing nonlinear transformation, enabling the model to better adapt to complex fog and haze distributions. The entire network will perform 8 iterations of feature extraction, gradually deepening the features, thereby capturing more complex fog and haze patterns and details in the image in each iteration. Finally, a 1×1 convolutional layer is used for channel compression to reduce the dimension of the feature map, so as to better output the subsequent decision layer. Next, the network generates a transmittance map using the Sigmoid activation function. The Sigmoid function maps the output value to between 0 and 1, representing the transmittance of each pixel in the image, where 0 represents complete occlusion (heaviest haze) and 1 represents complete transparency (no haze).

[0048] (2.2) The output of (1.3) is input into the atmospheric light estimation network, as shown in Figure 2. The input is processed by adaptive average pooling. The purpose of adaptive average pooling is to reduce the spatial dimension of the image and output a fixed-size output. Here, we pass the input image through adaptive average pooling to output a vector of size 1×1×C, where C is the number of channels. The pooling operation extracts the global features of the image by calculating the average value of each channel, ensuring that important global information is not lost, while reducing the spatial dimension, making subsequent calculations more efficient. Next, it goes through a 3×3 convolutional layer, the purpose of which is to learn more complex relationships between channels. After convolution, the ReLU activation function is used to enhance the expressive power of the features. Then, it goes through a 1×1 convolutional layer, the function of which is to reduce the dimension of the previously output feature map (usually a high-dimensional feature map) to the RGB 3-channel output, and can effectively weight each channel by learning, so that the three output channels can better represent the atmospheric light information in the image. Finally, the output goes through the Sigmoid activation function to generate an atmospheric light estimation map. Sigmoi maps the output after convolution to a range of 0 to 1, where 0 represents complete black (lowest illumination) and 1 represents complete white (highest illumination).

[0049] (2.3) Combine the outputs of (2.1) and (2.2) Transmittance diagram and The atmospheric light estimation map is a fog map calculated based on the atmospheric scattering model. The formula is as follows.

[0050]

[0051] Step 3: DRE-Net Two-Branch Reflectogram Estimation Network

[0052] (3.1) Perform Pair-Down Sampling on the image from (1.3). This process mainly involves performing structured downsampling. In each 2×2 block, two spatially adjacent pixels (e.g., horizontally, vertically, or diagonally adjacent) are randomly selected and extracted into two new images (Image 1: the first set of sampling points for all blocks, Image 2: the second set of sampling points for all blocks). This generates a pair of images with identical noise distribution and reduces the resolution to 320×320×3, thus reducing computational cost.

[0053] (3.2) The image 1 output from (3.1) is fed into the REE (Reflection Estimation Net) network. The input image is processed through a 3×3 convolutional layer to extract local spatial features. The atmospheric light estimation map output from (2.2) is then fed into a Cross-Attention module, enabling the network to adaptively learn the relationship between atmospheric light and illumination weights across different regions. After passing through the Cross-Attention module, the feature map output by the network is processed by a ReLU activation function to further enhance the expressive power of the feature map. After passing through the same module structure 7 times, the number of output channels is adjusted through a 1×1 convolutional layer, and finally, the output reflectance map is restricted to the range between 0 and 1 by a Sigmoid activation function.

[0054] (3.3) The image output from (3.1) is input into the NE (Noise Estimation) network, as shown in Figure 3. After gamma correction, the gamma value is randomly sampled in the range of [1.2, 1.5] to simulate the sensor noise distribution under different illumination intensities. Local texture features are extracted through a 3×3 convolutional layer, and the number of channels is expanded to 64. Then, Leaky ReLU (negative slope = 0.2) is used to introduce nonlinearity and alleviate gradient vanishing. Then, a second set of 3×3 convolution + Leaky ReLU is used, keeping the number of channels at 64 to enhance the feature extraction capability. Finally, a 7×7 adaptive dilated convolution is used, with the dilation rate dynamically adjusted (1 / 2 / 4) to cover different receptive fields (7×7 to 19×19) to capture the global noise pattern and avoid resolution loss caused by downsampling, thereby outputting a noise map.

[0055] (3.4) The output reflection diagram of (3.2) Output noise diagrams of (3.3) To integrate.

[0056]

[0057] (3.4) The reflection map of the output of (3.3) The input image is fed into a 3×3 convolutional layer for initial feature extraction. Then, a 32-layer residual network mines finer details from the image and enhances previously subtle but crucial information for image quality and detail through multi-layer nonlinear transformations and feature learning. Another 3×3 convolutional layer further adjusts and transforms the features extracted by the 32-layer residual network, strengthening feature channels related to image details and suppressing redundant features that might interfere with detail enhancement, making the features more suitable for the output detailed reflectance map. Finally, a 3×3 convolutional layer performs a final convolutional transformation on the processed features, mapping them to the required output dimension.

[0058] Step 4: LUE-Net Reflection Graph Estimation Network

[0059] (4.1) Input the output of (1.3) into LUE-Net. The input image is passed through a 3×3 convolutional layer to extract local illumination features. This convolutional layer uses a sliding window mechanism to extract features in local regions of the image, which helps to capture subtle illumination differences and provides a foundation for subsequent global feature modeling. Multihead-Attention is used to capture the global illumination relationship in the image, and then the ReLU activation function is used to enhance its non-linear expressive power. After passing through the same module structure 7 times, the number of output channels is adjusted through a 1×1 convolutional layer. Finally, after passing through the Sigmoid activation function, the illumination map output by the model will be normalized to the range between 0 and 1, where 0 represents complete black (lowest illumination) and 1 represents complete white (highest illumination).

[0060] (4.2) The output of (4.1) is input into the DCE-Net (Deep Curve Estimation) illumination enhancement network. The input image is processed through seven FT networks and one FTT network, as shown in Figure 2. The outputs of the third and fourth FT modules are concatenated in the second dimension, and the concatenated features are then input into the fifth FT module. Next, the outputs of the second and fifth FT modules are concatenated in a similar manner and input into the sixth FT module. Finally, the outputs of the first and sixth FT modules are concatenated in the second dimension and input into the final FTT module, which then outputs the illumination-enhanced image. Each FT module consists of a 3x3 convolutional layer, a Self-Attention module, and a ReLU activation function. A 3x3 convolutional layer extracts low-level features from the input image. Then, Self-Attention calculates the correlation between features to capture global contextual information, allowing the network to focus more on important regions of the image. A residual connection is then established between the input of the 3x3 convolutional layer and the output of Self-Attention to prevent overfitting to local information or gradient vanishing during training. Finally, a ReLU activation function is used for non-linear transformation. The FTT module has a similar structure to the FT module, but differs in its activation function. The FTT module uses the Tanh activation function, mapping the output to the range of -1 to 1. The Tanh function effectively avoids excessively large or small output values, ensuring a smoother and more stable network output. The final output... .

[0061] (4.3) The illumination enhancement map in (4.2) is fused with the illumination estimation curve. The formula for the illumination estimation curve is as follows:

[0062]

[0063]

[0064] The feature map output by the FT network, I(x) (2.2) is the image output by the low-light image.

[0065] (4.4) Output fog map of (2.3) Output reflection diagram of (3.4) The output illumination diagram of (4.3) The fusion is based on Retinex theory. The Retinex theory formula is as follows:

[0066]

[0067]

[0068] (4.5) Enhance the output of (4.4) A channel-based self-attention layer is fed into the image to analyze the correlation between different channels. By learning the dependencies between channels, the weights of key feature channels are increased, and the influence of unimportant channels is suppressed. A spatial self-attention layer then focuses on information about the spatial location of the image. Considering the spatial relationship between each pixel and other pixels, spatial self-attention can capture the spatial distribution pattern of fog and the spatial relationship of illumination differences in different areas, especially for images with uneven lighting or fog. Next, a U-Net is used. In the encoding path, features are continuously extracted and compressed through convolution and downsampling to reduce the spatial size of the feature map, increase the number of channels, and capture more abstract features. In the decoding path, the spatial size of the feature map is gradually restored through upsampling and convolution. Simultaneously, features from the same or similar levels in the encoding path are combined, and image details and spatial information are restored through skip connections. Finally, a 1x1 convolution and Tanh activation function are used to map the output to [-1, 1].

[0069] Step 5: Semantic guidance loss and SC-DCE network loss

[0070] (5.1) The final enhanced graph output by (4.4) Input the pre-trained semantic segmentation model deeplabv3 to generate a semantic graph. Then, use the resulting semantic graph... and the semantic graph that comes with Cityscapes The semantic consistency loss is constructed, and the semantic information guides the network enhancement structure, as shown in Figure 9. The formula for the semantic consistency loss is as follows:

[0071]

[0072] If it is true, These are predicted values.

[0073]

[0074] For the s-th category, For semantic category space, The total number of categories in the semantic category space. For brightness.

[0075] (5.1) The loss of the SC-DCE network is given by the following formula:

[0076]

[0077]

[0078] This invention is set to 5. This invention is set to 2, This invention is set to 4. This invention is set to 10. This invention is set to 5. This invention is set to 5. The invention is set to 10.

[0079] The formula for loss due to similarity in illumination distribution is as follows:

[0080]

[0081] Histogram KL divergence term, Calculate the histogram distribution of the luminance components. This loss ensures that the luminance distribution of the enhanced image conforms to the statistical characteristics of the natural scene.

[0082] The attention-guided feature loss is formulated as follows:

[0083]

[0084] Extract using pre-trained VGG-19 and Multi-layer features are used to force structural consistency with the real image in the attention region.

[0085] The formula for the two-domain gradient consistency loss is as follows:

[0086]

[0087] For horizontal / vertical gradient operators, The attenuation coefficient is set to 10 in this invention.

[0088] The formula for mean squared error loss is as follows:

[0089]

[0090] The HD network supervision loss is given by the following formula:

[0091]

[0092] The transmittance map monitoring loss is calculated using the following formula:

[0093]

[0094] The gradient operator is used in the horizontal / vertical direction, and the role of the transmittance map supervision loss is to smooth the edges of the transmittance map.

[0095] The atmospheric map monitoring loss is calculated using the following formula:

[0096]

[0097] The purpose of atmospheric map monitoring loss is to prevent drastic fluctuations in atmospheric light.

[0098] The loss is decomposed into illumination and reflection maps, and the formula is as follows:

[0099]

[0100] The formula for the illumination map loss is as follows:

[0101]

[0102] The effect of the lighting map loss is to smooth the lighting.

[0103] The formula for the reflection loss is as follows:

[0104]

[0105] The structure-aware loss function aims to improve the quality of the generated image by comparing high-level features of the image. The total variational loss is used to preserve edge information.

[0106] The formula for mean squared error loss is as follows:

[0107]

[0108] (6) To comprehensively verify the performance of the proposed augmentation network in low-light and fog environments, this section designs a comprehensive test scheme involving multiple datasets and multiple metrics. The test process is divided into reference evaluation (for datasets containing real values) and no-reference evaluation (for datasets lacking real values). The algorithm's performance is quantified from multiple dimensions, including pixel-level accuracy, structural similarity, and human perception quality, and compared with mainstream methods. The process includes the following steps:

[0109] (6.1) Dataset Selection: To verify the enhancement capabilities of this invention in low-light and foggy environments, we selected the constructed Foggy-Cityscape-Dark, GTA5, NHC, and NHR datasets for performance analysis. Foggy-Cityscape-Dark is a low-light foggy day dataset synthesized based on Cityscapes, containing paired foggy / non-fog-low-light / normal-light images, providing ground truth values ​​for quantitative analysis. GTA5 is a virtual scene dataset rendered by a game engine, which can generate degraded images with different fog concentrations and lighting conditions. Its high-fidelity ground truth values ​​are suitable for model generalization testing. NHR (Nighttime Hazy Rural) is a low-light hazy rural road dataset. The scene structure is simple, but the fog layer distribution is uneven, challenging the model's enhancement capabilities in weakly textured areas.

[0110] (6.2) Selection of evaluation metrics: For the Foggy-Cityscape-Dark and GTA5 datasets with ground truth values, PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity) are used as reference metrics, and LPIPS (Learned Perceptual Image Patch Similarity) is used as a non-reference metric. For the NHC and NHR datasets without ground truth values, LPIPS, BRISQUE, and NIMA are used as non-reference losses. PSNR can quantify the pixel-level error between the augmented image and the ground truth value, in dB. A higher value indicates better reconstruction accuracy. SSIM evaluates the similarity of images in brightness, contrast, and structure, ranging from [0,1], and is sensitive to edge and texture preservation. LPIPS is a perceptual similarity metric based on a pre-trained VGG network. A lower value indicates smaller differences in human visual perception and can capture color distortion and detail distortion. BRISQUE is a quality score based on the statistical features of natural scenes. A lower value indicates less image distortion (such as noise and blur). NIMA combines aesthetics and technical quality in its deep learning score, ranging from [1, 10], with higher scores indicating greater alignment with human aesthetic preferences.

[0111] (6.3) Testing and Evaluation: First, all test images were uniformly resized to 640×640 resolution, and bilinear interpolation was used to preserve the geometric structure. Only normalization was performed before inputting the images into the network. The test output images were evaluated by quantitative indicators and human subjective evaluation, and evaluation was also performed for extreme environments with low-light and fog images.

Claims

1. A low-light and fog overlay enhancement method based on TLH-Net, TLH-Net as shown in Figure 1, characterized in that... Includes the following steps: (1) Construct the low-light and foggy image dataset foggy-Cityscapes-Dark, including the following steps: (1.1) Based on the publicly available Cityscapes dataset, which mainly contains 5000 high-quality pixel-level annotated images of driving scenes in urban environments from 50 different city street scenes, including 2975 training set images, 500 validation set images, and 1525 test set images, with a total of 19 categories); Foggy-Cityscapes is a synthetic dataset generated by Cityscapes, which aims to simulate and study the performance of autonomous vehicles in foggy conditions. It has three levels of fog (0.005, 0.01, 0.02), from light fog to dense fog, corresponding to visibility ranges of 600, 300, and 150 meters, respectively; (1.2) Using the EC-Zero-DCE network, the brightness of each image in the Foggy-Cityscapes dataset is reduced to generate Foggy-Cityscapes-Dark; (1.3) The Foggy-Cityscapes-Dark image dataset is preprocessed, and all images are uniformly scaled to 640x640x3 pixels for subsequent processing; (2) The fog map estimation network HE-Net (Haze Estimator Net) is designed as shown in Figure 2, including the following steps: (2.1) The image preprocessed in (1.3) is sent into the fog map estimation network HE-Net; (2.2) The image output in (1.3) is processed through a 3×3 convolutional layer to extract local fog concentration features, and then ReLU is used to enhance the feature expression ability by capturing long-distance fog distribution dependence through Multihead-Attention, and the features are deepened by 8 cycles. Finally, a 1×1 convolutional layer and a Sigmoid layer are used to generate a transmissivity map; (2.3) The image output by (1.3) is processed by adaptive average pooling to output a 1×1×C vector, then processed by a 3×3 convolutional layer to learn the relationship between channels and fed into ReLU to enhance the expressive power, then processed by a 1×1 convolutional layer to output (RGB) 3 channels, and finally processed by a Sigmoid layer to generate an atmospheric light map. (2.4) Use the atmospheric scattering model to generate a fog map from the output transmittance map and atmospheric light estimation map; (3) Design the dual-branch reflectance estimation network DRE-Net (Dual Reflection Estimation Net) as shown in Figure 3, including the following steps: (3.1) Input the image output from (1.3) into the dual-branch reflectance estimation network DRE-Net; (3.2) Perform Pair-Down Sampling on the image preprocessed in (1.3). This process divides the image into N 2x2 pixel blocks, randomly collects two adjacent pixels in each pixel block, and then outputs image 1 and image 2; (3.3) Extract local spatial features from image 1 output from (3.2) through a 3x3 convolution; input the atmospheric light estimation map output from (2.3) into Cross-Attention so that the network can adaptively learn the atmospheric light and illumination weights of different regions, and then pass through a RuLU and loop 8 times; finally, output the reflectance map through a 1x1 convolutional layer and Sigmoid. (3.4) Perform gamma correction on the output image 2 of (3.2). This process randomly selects a parameter between 1.2 and 1.5 to exponentially increase the brightness of the image to simulate noise under different lighting conditions. Then a 3×3 convolutional layer is passed to extract primary spatial features, followed by a Leaky ReLU activation function to introduce nonlinearity and alleviate the gradient vanishing problem; a 3×3 convolutional layer + Leaky ReLU is used again to further deepen feature extraction; then a 7×7 adaptive dilated convolution is connected to the residual to expand the receptive field to capture a wider range of contextual information, while adjusting the dilation rate to adapt to features of different scales to avoid resolution loss, and finally a noise map is output through Sigmoid; (3.5) The reflection map output by (3.3) and the noise map output by (3.4) are fused to form a reflection map; (3.6) The reflection map output by (3.5) is fed into the detail enhancement network as shown in Figure 4; a 3x3 convolutional layer is used to capture local detail information, and then a 32-layer residual network is used to capture deep detail information, followed by a 3x3 convolution; finally, a skip connection is used to prevent information overfitting; (4) The illumination map estimation network LUE-Net (Illumination Estimation Net) is designed as shown in Figure 5, including the following steps: (4.1) The image output from (1.3) is fed into the dual-branch dehazing network reflectance estimation network DRE-Net; (4.2) Local illumination features are extracted through a 3x3 convolutional layer, global illumination relationships (such as shadow-highlight correspondence) are captured through Multihead-Attention, long-distance illumination dependence is modeled, and then a RuLU is passed through, and the loop is repeated 8 times; finally, the illumination map is output through a 1x1 convolutional layer and Sigmoid; (4.3) The illumination map output from (4.2) is processed through 4 consecutive FT modules (as shown in Figure 7). Each FT module independently processes the features to gradually deepen the feature representation; the outputs of the third and fourth FT modules are concatenated in the second dimension and input into the fifth FT module, the outputs of the second and fifth FT modules are concatenated in the second dimension and input into the sixth FT module, and the outputs of the first and sixth FT modules are concatenated in the second dimension and input into the sixth FT module. In the second dimension, the first concatenation input is fed into the FTT (as shown in Figure 7) module; the FT module first passes through a 3x3 convolutional layer, then through a Self-Attention to obtain broader contextual information, then through residual connections to prevent over-focusing on local information, and finally through a ReLU activation function to introduce nonlinearity and alleviate the gradient vanishing problem; the FTT module is similar in structure to the FT module, except that the activation function is replaced with Tanh and the output is mapped to [-1,1]; (5) Design the dehazing network as shown in Figure 8, including the following steps: (5.1) The atmospheric light estimation map and atmospheric projectivity map output by (2.4), the reflection map after detail enhancement output by (3.6), and the illumination map after illumination enhancement output by (4.3) are concatenated and input into the dehazing network; (5.2) First, a channel self-attention network learns the dependencies between channels. Then, a spatial self-attention network captures the spatial distribution pattern of fog and the spatial relationships of illumination differences in different regions. Next, a U-net network extracts, fuses, and restores image illumination and dehazing features. Finally, a 1x1 convolution and Tanh activation function map the output to [-1, 1].

Images