Target detection method based on DFLLOD-Net under low-illumination superposition fog weather

Through the DFLLOD-Net network, combined with multiple enhancement modules and attention detection modules in low illumination and foggy weather conditions, the problem of poor target detection in the existing technology is solved, and efficient image processing and object detection are achieved, improving detection accuracy and robustness.

CN118918035APending Publication Date: 2024-11-08GUILIN UNIV OF ELECTRONIC TECH
View PDF 0 Cites 6 Cited by

Patent Information

Application Number
CN202410935035.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-12
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

The prior art has poor target detection effect under low illumination and foggy weather conditions, image processing methods fail to make full use of multi-scale information and attention mechanisms, resulting in loss of detail and noise enhancement, and image enhancement and object detection are regarded as independent stages. , affects overall performance.

Method used

DFLLOD-Net network was designed, combining low-illumination enhancement module, defog enhancement module, denoising enhancement module and conditional adaptive module, and dynamically adjust the module weight through the ExDark-fog dataset and atmospheric scattering model, and use the attention target detection module. Achieve end-to-end optimization, coordinate image processing and detection models.

Benefits of technology

It significantly improves the accuracy and robustness of object detection in low illumination and foggy weather conditions, enhances the model's generalization ability to complex environments, and improves image quality and detection performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918035A_ABST
    Figure CN118918035A_ABST
Patent Text Reader

Abstract

The invention discloses a DFLLOD-Net-based target detection method under low-illumination superposition foggy weather. The method comprises the following steps: (1) constructing a low-illumination foggy image data set ExDark-fog; (2) a low-illumination enhancement module is designed, and the brightness and the contrast ratio of the image under the low-illumination condition are improved; (3) a defogging enhancement module is designed, so that the influence of fog on the image definition is effectively removed; (4) a denoising enhancement module is designed, image noise is reduced, and image quality is enhanced; (5) a condition adaptive module is designed, and the weight can be adaptively adjusted based on the conditions of fog density, illumination intensity and the like in the image so as to optimize and coordinate the functional performance of the three different enhancement processing modules; and (6) inputting the enhanced image into a detection module. The DFLLOD-Net network realizes full-process automation and optimization from image preprocessing to target detection through the highly specialized modules and end-to-end training strategies, greatly improves the performance and robustness of the model under complex visual conditions, and provides an innovative solution for target detection in low-illumination superposition foggy weather.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of target detection technology, and is a target detection method under low illumination and foggy weather based on DFLLOD-Net. The method constructs the ExDark-fog dataset, and uses a conditional adaptive module to dynamically adjust the weights of a low illumination enhancement module, a defogging enhancement module, a denoising enhancement module, and an attention target detection module under low illumination and foggy weather conditions, thereby achieving efficient target recognition under low illumination and foggy weather conditions, and significantly improving detection accuracy and robustness. Background Art

[0002] Most of the existing low-light or foggy image datasets exist separately, lacking a comprehensive dataset that includes both low-light and haze environments. When processing low-light or foggy images, existing image processing methods often ignore the complex environmental factors in the image, resulting in low quality of the generated images, which cannot be effectively used to train deep learning models. The present invention combines the depth estimation network and the atmospheric scattering model to synthesize the low-light foggy image dataset ExDark-fog, which overcomes the limitations of the dataset for a single environmental condition and provides a more realistic data basis for subsequent image processing and target detection.

[0003] Although traditional defogging and low-light enhancement methods have improved image quality to a certain extent, they usually rely on physical models or simple image processing techniques and fail to make full use of the multi-scale information and attention mechanism of the image. This leads to the existing models often having problems such as detail loss and noise enhancement in complex scenes, especially under low-light and foggy conditions. In addition, existing methods often regard image enhancement and target detection as two independent stages. This separate processing flow may cause the image enhancement step to be unable to fully adapt to the needs of target detection, affecting the overall performance. In order to overcome these challenges, the present invention proposes a DFLLOD-Net network, a deep learning model that integrates multiple innovative modules, specifically for target detection tasks under low-light foggy weather conditions. The design of the DFLLOD-Net network includes: a low-light enhancement module LEM, which improves the brightness and contrast of the image under low-light conditions while retaining details. A defogging enhancement module FREM, which effectively removes the adverse effects of fog on image clarity and restores the clarity of the image. A denoising enhancement module DEM, which reduces image noise, enhances image quality, and improves brightness and contrast. The conditional adaptive module CAM dynamically adjusts the feature extraction process according to the image's lighting and fog density conditions, generates conditional encodings through deep feature analysis and fully connected layers, and intelligently activates subsequent processing modules. The attention object detection module AODM combines SPPFELAN and the attention enhanced convolution block AECB. The DFLLOD-Net network implements end-to-end optimization from image preprocessing to object detection, and achieves more coordinated performance improvements by jointly optimizing the parameters of the image processing and detection models.

[0004] Through these innovative designs, the DFLLOD-Net network not only improves the accuracy of target detection under low illumination and foggy conditions, but also enhances the model's generalization ability for complex environments, providing an effective solution for image processing and target detection under low illumination and severe weather conditions. Summary of the invention

[0005] The present invention provides a target detection method under low illumination and foggy weather based on DFLLOD-Net, which is specially designed for low illumination and foggy conditions and is constructed based on the DFLLOD-Net network. The method provides an effective solution to the limitations of traditional technologies in complex visual environments. By constructing the ExDark-fog dataset, combining depth estimation with the atmospheric scattering model, the real low illumination and foggy conditions are simulated. The method innovatively designs the DFLLOD-Net network to integrate three enhancement modules, namely low illumination enhancement LEM, defogging enhancement FREM, and denoising enhancement DEM, as well as a conditional adaptive module CAM. These modules work together and dynamically adjust to adapt to different lighting and fog density conditions. In particular, the attention target detection module AODM in the network combines SPPFELAN and the attention enhancement convolution block AECB to further improve the detection accuracy. Through an end-to-end training strategy, the DFLLOD-Net network significantly improves the target detection performance under low illumination and foggy weather conditions.

[0006] The present invention comprises the following main steps:

[0007] (1) Constructing the low-light foggy image dataset ExDark-fog includes the following steps:

[0008] (1.1)) Based on the public ExDark dataset, it contains 7363 images covering 12 different target categories. These images are taken under various low-light conditions.

[0009] (1.2) Use the Monodepth2 network to estimate the depth of each image in the ExDark dataset and generate a depth map; combine the depth map and fog concentration coefficient to generate transmission;

[0010] (1.3)) Apply the atmospheric scattering model to synthesize foggy images;

[0011] (1.4) After synthesis, all images of the ExDark-fog image dataset are uniformly resized to 640x640x3 pixels. The ExDark-fog dataset is divided into training set, test set and validation set, with the ratios of 3 / 4, 1 / 8 and 1 / 8 respectively.

[0012] (2) Design of Low-Light Enhancement Module (LEM), such as Figure 2 As shown, the following steps are included:

[0013] (2.1) First, the low-light foggy images in the ExDark-fog dataset and their self-attention features are input into the encoding module, as follows: Figure 2As shown in Figure 2. This module consists of three low-light adaptive encoding blocks (LAEBs), as shown in Figure 2. Figure 3 As shown in Figure 2. Each LAEB uses a series of convolutional layers, including 1×1, 3×3, and 5×5 convolution kernels, followed by GeLU activation functions, and 1×1 convolution kernels for channel mixing. In addition, each LAEB also contains a downsampling module DownSample, which uses a 3×3 convolution kernel with a stride of 2 and a ReLU activation function to reduce the spatial size of the feature map.

[0014] (2.2) After being processed by the Encoding Module, the feature data is then sent to the Latent Space Sampling Module (LSSM), as Figure 2 As shown in Figure 2. In this module, the statistical calculation operator is first applied to analyze the features, from which the mean μ and variance σ are calculated. Using these statistics, the sampling feature sample z is generated.

[0015] (2.3) After the latent space sampling module LSSM successfully obtains feature samples, these samples are combined with the features extracted by the self-attention mechanism. The combined features are then sent to the decoding module

[0016] (Decoding Module), such as Figure 2 The decoding module consists of three low-light adaptive decoding blocks (LADB), as shown in Figure 3 As shown in the figure, each LADB is equipped with a 3×3 convolution kernel for feature refinement, a GeLU activation function to introduce nonlinearity, and a 3×3 deconvolution kernel in conjunction with an upsampling layer to gradually restore the high resolution of the image.

[0017] (2.4) The output of the Decoding Module is added to the input to obtain the enhanced feature map.

[0018] (3) Design of Fog Removal Enhancement Module (FREM), such as Figure 4 As shown, the following steps are included:

[0019] (3.1) The ExDark-fog low-light foggy image is sent to the dehazing enhancement module FREM. First, the module consists of two convolutional layers. Each convolutional layer uses a 3×3 convolution kernel with a step size of 1 to extract the preliminary features of the image.

[0020] (3.2) The features initially extracted in (3.1) are sent to the multi-scale feature extraction module (Multi-scale Features) MSF, such as Figure 4 As shown in Figure 2. The module consists of three consecutive multi-scale feature extraction units (MSFU), such as Figure 5 As shown in the figure, each unit contains four parallel branches, which extract features of different scales through convolution kernels of different sizes, and enhance the model's ability to capture multi-scale information of images through Concat and ReLU activation functions.

[0021] (3.3) After the MSF module in (3.2) completes the multi-scale feature extraction, the CSA (Channel Spatial Attention) attention module is introduced, as shown in Figure 6 As shown in Figure 2. The CSA module is designed as two parallel branches: the channel attention (CA) branch and the spatial attention (SA) branch. The CA branch uses a global pooling operation followed by a fully connected layer to generate a channel attention map that reflects the importance of different channels. The SA branch captures global context information through 1×1 convolution, and further generates a spatial attention map through global average pooling and global maximum pooling.

[0022] (3.4) After the CSA attention module is processed, the network structure continues with two stages of skip connections, such as Figure 4 As shown in the figure. First, the 1×1 convolution kernel is used to reduce or increase the dimension of the feature channel to ensure that the output feature maps generated by different levels are compatible in dimension. Then, the channel matching is further achieved through the 3×3 convolution kernel, and these feature maps are added and fused to integrate information between different levels and enhance the overall feature representation capability.

[0023] (4) Design a denoising enhancement module (DEM), such as Figure 7 As shown, the following steps are included:

[0024] (4.1) The ExDark-fog low-light foggy weather image is sent to the denoising enhancement module DEM, which first passes through a CBR and then is divided into two branches for denoising enhancement: the first branch first introduces a 1×1 fully connected layer and a Sigmoid activation function to generate an attention map. Then, the number of channels is adjusted through a 1×1 convolution layer to fuse with the feature map of the second branch.

[0025] (4.2) The second branch first adjusts the number of channels through a 1×1 convolutional layer to accommodate subsequent processing requirements. Then, the spatial dimension of the feature map is reduced through a 3×3 convolutional layer and downsampling with a stride of 2, and the denoising process begins at the same time. Next, the feature map passes through two 3×3 convolutional layers with a stride of 1 and a padding of 1, and the output of the downsampling layer is added to the output of the convolution block channel by channel and pixel by pixel. This structure is repeated three times to deepen the feature extraction and denoising process. After the last cycle, upsampling is performed through a 3×3 deconvolution layer with a stride of 2 to restore the spatial size of the feature map while ensuring that the denoising effect is consistent at all scales.

[0026] (4.3) The features extracted by the two branches are concat-merged in the channel dimension and further integrated through a 3×3 convolutional layer and a subsequent 1×1 convolutional layer, ensuring that the final output feature map enriches the image information and enhances its expressiveness while denoising.

[0027] (5) Design of Conditional Adaptive Module (CAM), such as Figure 1 As shown, the following steps are included:

[0028] (5.1) First, we receive the ExDark-fog low-light foggy weather condition image input. The image first passes through a preliminary feature extraction layer, using a 3×3 convolution kernel and a stride of 1 for convolution.

[0029] (5.2) The features in (5.1) are fed into a deep feature analysis, which consists of three convolutional layers Conv1, Conv2, and Conv3, with 3×3, 5×5, and 7×7 convolution kernels, respectively, and strides of 1, 2, and 2. Each convolutional layer is followed by batch normalization and LeakyReLU activation function to deeply analyze the image features.

[0030] (5.3) The output of the deep feature analysis in (5.2) is reduced in dimension through a 3×3 average pooling layer, and feature compression is performed through a 1×1 convolution.

[0031] (5.4) The reduced features are then fed into a fully connected layer to generate conditional encodings that quantize the image conditions such as illumination and fog density. After full connection and softmax function, the weight vectors W1, W2, and W3 are generated to ensure that the sum of the weights is 1.

[0032] (5.5) The weight vector is multiplied and added with the output of the LEM, FREM and DEM modules, and then adjusted to 3 channels after a 1×1 convolution.

[0033] (6) Design of Attention-based Object Detection Module (AODM), such as Figure 1 As shown, the following steps are included:

[0034] (6.1) The feature map output by (5.5) is sent to the AODM target detection module. The module first passes through four CBS modules for feature extraction, and then passes through the C2f module. The preliminary feature obtained is represented as C1; C1 is further processed by the downsampling SCDown and C2f modules, and the result is represented as C2; C2 is further processed by the downsampling SCDown, C2f and SPPFELAN modules, and the result is represented as C3; The SPPFELAN module structure is as follows Figure 8 As shown, it is used to enhance the feature extraction capability.

[0035] (6.2) C3 in (6.1) is upsampled and concatenated with C2, and then passed through the C2f module, the result is represented as C4; C4 is further upsampled and fused with C1, the result is represented as C5, which realizes the effective integration of features of different resolutions; then, C5 is processed by CBS and concatenated with C4, and the concatenated result is passed through the C2f module, the result is represented as C6; finally, C6 is downsampled SCDown and concatenated with C3, the result is represented as C7; this process not only realizes the fusion of cross-scale features, but also optimizes the hierarchical structure of features.

[0036] (6.3) C5, C6, and C7 in (6.2) are respectively fed into the Attention-Enhanced Convolution Block (AECB) module, as Fig. 9 As shown in Figure 2. This module combines convolution operations and attention mechanisms to capture local features and global context information of images.

[0037] (6.4) The features processed by the AECB module are then sent to the detection head for target detection.

[0038] (7) Testing the entire DFLLOD-Net target detection network under low illumination and foggy weather conditions includes the following steps:

[0039] (7.1)) Send the ExDark-fog dataset to the low illumination enhancement module LEM, the defogging enhancement module FREM and the denoising enhancement module DEM of DFLLOD-Net for enhancement processing;

[0040] (7.2) Using the conditional adaptive module CAM to dynamically adjust the weight of the module (7.1) according to the image illumination and fog density conditions to optimize the image enhancement effect;

[0041] (7.3) The enhanced image obtained in (7.2) is sent to the attention object detection module AODM for feature extraction and object detection;

[0042] (7.4) Visualize the detected results.

[0043] This technical solution solves the problem of poor detection effect in target detection due to low imaging contrast, blurred details, uneven illumination and other factors in low illumination and foggy weather. First, this technology constructs the ExDark-fog dataset to synthesize images of low illumination and foggy environments, thereby enhancing the model's generalization ability for complex environments; secondly, the designed conditional adaptive module CAM is used to dynamically adjust the weights of the three modules, namely the low illumination enhancement module LEM, the defogging enhancement module FREM and the denoising enhancement module DEM, to effectively extract features; the clarity and brightness contrast of the image in low illumination and foggy environments are respectively improved, thereby improving the image quality. Finally, the enhanced image is sent to the attention target detection module AODM for detection, achieving efficient target detection in low illumination and foggy environments, and significantly improving the accuracy and robustness of detection through end-to-end training. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 Schematic diagram of DFLLOD-Net network structure in the embodiment;

[0045] Figure 2 This is a schematic diagram of the LEM module structure in the embodiment;

[0046] Figure 3 Schematic diagram of the structure of LAEB and LADB in the embodiment;

[0047] Figure 4 Schematic diagram of the FREM module structure in the embodiment;

[0048] Figure 5 This is a schematic diagram of the MSFU module structure in the embodiment;

[0049] Figure 6 Schematic diagram of the structure of the CSA attention module in the embodiment;

[0050] Figure 7 This is a schematic diagram of the DEM module structure in the embodiment;

[0051] Figure 8 It is a schematic diagram of the structure of the SPPFELAN module in the embodiment;

[0052] Fig. 9 This is a schematic diagram of the AECB module structure in the embodiment; DETAILED DESCRIPTION

[0053] Step 1: Build the low-light foggy image dataset ExDark-fog

[0054] 1-1 Based on the public ExDark dataset, it contains 7,363 images covering 12 different target categories. These images are taken under various low-light conditions.

[0055] 1-2 Use the Monodepth2 network to estimate the depth of each image in the ExDark dataset and generate a depth map D(X); combine the depth map and the fog concentration coefficient β to generate the transmission map t(x), the formula is

[0056] t(x)=e -βD(x) (1)

[0057] 1-3 Apply the atmospheric scattering model to synthesize the foggy image J(x), the formula is

[0058] J(x)=I(x)t(x)+A(1-t(x)) (2)

[0059] Where A is set to the pixel value with the highest brightness in the image;

[0060] 1-4 After synthesis, all images of the ExDark-fog image dataset are uniformly adjusted to 640x640x3 pixels. The ExDark-fog dataset is divided into training set, test set and validation set, with ratios of 3 / 4, 1 / 8 and 1 / 8 respectively.

[0061] Step 2: LEM Low Light Enhancement Module

[0062] The LEM module adapts to low-light images through an encoding-decoding structure, uses LAEB for feature extraction and downsampling, generates feature samples through statistically guided sampling, and then gradually restores the resolution and enhances the details through LADB, and finally fuses them to obtain an enhanced feature map.

[0063] 2-1 First, the low-light foggy images in the ExDark-fog dataset and their self-attention features are input into the encoding module, such as Figure 2 As shown in Figure 2. The module consists of three low illumination adaptive coding blocks LAEB, such as Figure 3As shown in Figure 2. Each LAEB uses a series of convolutional layers, including 1×1, 3×3, and 5×5 convolution kernels, followed by GeLU activation functions, and 1×1 convolution kernels for channel mixing. In addition, each LAEB also contains a downsampling module DownSample, which uses a 3×3 convolution kernel with a stride of 2 and a ReLU activation function to reduce the spatial size of the feature map.

[0064] 2-2 After being processed by the Encoding Module, the feature data is then sent to the latent space sampling module LSSM, as shown in Figure 2 As shown in Figure 2. In this module, the statistical calculation operator is first applied to analyze the features, from which the mean μ and variance σ are calculated. Using these statistics, the sampling feature sample z is generated.

[0065] z=μ+σ·s(0,1) (3)

[0066] Among them, μ, σ are the mean and standard deviation, and s(0, 1) is a Gaussian distribution sampling operator with a mean of 0 and a variance of 1.

[0067] 2-3 After the latent space sampling module LSSM successfully obtains feature samples, these samples are combined with the features extracted by the self-attention mechanism. The combined features are then sent to the decoding module Decoding Module, as shown in Figure 2 The decoding module consists of three low illumination adaptive decoding blocks LADB, as shown in Figure 3 As shown, each LADB includes a 3×3 convolution kernel, a GeLU activation function, a 3×3 deconvolution kernel, and an upsampling layer to gradually restore the image resolution and enhance the details.

[0068] 2-4 Add the output of the Decoding Module to the input to generate a low-light enhancement feature map.

[0069] Step 3: FREM Dehaze Enhancement Module

[0070] First, the low-light foggy image is subjected to preliminary feature extraction to extract the local features of the image; second, the extracted features are sent to the multi-scale feature extraction module and enhanced using three multi-scale feature extraction units; then, attention is used to extract features and the enhanced features are fused. Figure 4 shown.

[0071] 3-1 First, the original input low-light foggy image is received for preliminary feature extraction. It consists of two convolutional layers, each with a 3×3 convolution kernel, a step size of 1, and same padding. The first convolutional layer outputs a feature map of size 640×640 and a channel number of C1=64. The second convolutional layer further extracts features, and the output feature map size and channel number remain unchanged. The local features of the image are initially extracted to provide a basic feature map for subsequent multi-scale feature extraction.

[0072] 3-2 The features extracted in 3-1 are sent to the multi-scale feature extraction MSF. This module consists of three consecutive multi-scale feature extraction units MSFU, such as Figure 5 As shown, each unit contains four parallel branches. The first branch: first, the dimension is increased through a convolution with a convolution kernel size of 1×1, and then a convolution with a convolution kernel size of 3×3 and a step size of 1 is used to extract features, and finally a convolution with a convolution kernel size of 1×1 is used to adjust the number of channels. The second branch: first, the dimension is increased through a convolution with a convolution kernel size of 1×1, and then a convolution with a convolution kernel size of 5×5 and a step size of 1 is used to extract features, and finally a convolution with a convolution kernel size of 1×1 is used to adjust the number of channels. The third branch: first, the dimension is increased through a convolution with a convolution kernel size of 1×1, and then a convolution with a convolution kernel size of 7×7 and a step size of 1 is used to extract features, and finally a convolution with a convolution kernel size of 1×1 is used to adjust the number of channels. The fourth branch: The number of channels is adjusted through a convolution with a size of 1×1. The feature maps obtained by the four branches are spliced, and the ReLU activation function is applied to obtain the output feature map F of the MSF module. msf By using convolution kernels of different sizes to capture features of different scales, the model's ability to capture multi-scale information of images is enhanced, which helps improve the dehazing effect.

[0073] 3-3 After the MSF module in (3-2) completes the multi-scale feature extraction, the CSA (Channel and Spatial Attention) attention module is introduced, as shown in Figure 6 As shown in Figure 2, this module consists of two parts: channel attention (CA) and spatial attention (SA), which aims to enhance the expressiveness of feature maps and highlight the key information in the image. Channel attention (CA) first uses global pooling to extract the multi-scale features. msf The dimension is reduced to 1×1×C2, and then three fully connected layers and two activation functions (ReLU) are used to generate the channel attention map M C . Global pooling allows the model to capture the statistics of the entire feature map, while the fully connected layer and activation function help learn the complex relationship between channels. The last layer of full connection is normalized by the Sigmoid function. The generated channel attention map MC It reflects the importance of different channels and enhances the response of the feature map. C With the original feature map F msf Multiply them together and then add the result to the original feature map F msf Add together to get the enhanced feature map F C .

[0074] F C =F msf +(M C ⊙F msf ) (4)

[0075] This operation combines the weights of the channel attention map with the information of the original feature map, adding weight adjustment for inter-channel correlation.

[0076] Spatial Attention (SA) first extracts global context information in the feature map through a 1×1 convolution, keeping the spatial dimension of the feature map unchanged. Then, the context information is subjected to global average pooling (GAP) and global maximum pooling (GMP), respectively, to capture the spatial distribution and saliency information of the feature map. The two feature maps are combined through the Concat operation and the spatial attention map M is generated through a 3×3, Sigmoid convolution layer. S The generated spatial attention map M S The important areas in the image are emphasized and the expression of local features is enhanced. S Feature map F enhanced with channel attention C Multiply them together to get the final enhanced feature map F CS .

[0077] F CS =M S ⊙F C (5)

[0078] The final enhanced feature map F CS The attention mechanism combining the channel and spatial dimensions provides richer and more discriminative feature representations for subsequent skip connections and object detection tasks.

[0079] 3-4 After the CSA attention module is processed, the network structure continues with two stages of skip connections, such as Figure 4 As shown in Figure 1, features at different levels are integrated and feature representation is enhanced. The output F1 of the first 3×3 convolutional layer of the preliminary feature extraction is reduced or increased in channel dimension through a 1×1 convolution kernel to match the output F of the CSA attention module. cs Then, the output of this 1×1 convolution and the output of the CSA attention module are matched through the 3×3 convolution kernel and added to obtain F skip1The first skip connection enhances the combination of local features in the feature map F1 and the global features output by the CSA attention module, improving the expressiveness of the features. The output F2 of the second 3×3 convolutional layer of the preliminary feature extraction is then channel-adjusted through a 1×1 convolution kernel to match the output F of the first skip connection. skip1 The output of this 1×1 convolution is then matched with the output of the first skip connection through a 3×3 convolution, and then added. The second skip connection integrates the multi-level features from preliminary feature extraction to CSA attention to the first skip connection, providing the network with rich contextual information and detail information. The skip connection enables the network to capture both local details and global contextual information of the image through cross-layer feature fusion, thereby improving the performance of the dehazing enhancement module.

[0080] Step 4: DEM denoising enhancement module

[0081] First, the features are extracted preliminarily, and the extracted features are sent to the two branches of the denoising module for separate processing: the first branch uses a fully connected layer and a Sigmoid activation function to generate an attention map to identify and enhance important features while suppressing noise; the second branch focuses on reducing the spatial dimension of the image and enhancing deep features through convolution operations while removing noise; secondly, the features processed by the two branches are concat-merged and further integrated through a convolution layer to enhance the expressiveness of the features; the DEM denoising enhancement module structure is as follows: Figure 7 shown.

[0082] 4-1 The input image first passes through a CBR layer for preliminary feature extraction and nonlinear transformation, where the CBR layer consists of a 3×3 convolution kernel, batch normalization, and ReLU activation function.

[0083] 4-2 sends the features extracted in 4-1 to the denoising enhancement module and divides it into two branches. The first branch first performs a 1×1 convolution and then introduces a fully connected layer and a Sigmoid activation function to generate an attention map, identify and enhance important features, and suppress noise. Then, the number of channels is adjusted through a 1×1 convolution layer to merge with the feature map of the second branch.

[0084] The second branch first adjusts the number of channels through a 1×1 convolutional layer to accommodate subsequent processing needs. Then, the spatial dimension of the feature map is reduced through a 3×3 convolutional layer and downsampling with a stride of 2. Next, the feature map passes through two 3×3 convolutional layers with a stride of 1 and a padding of 1. The output of the downsampling layer is added to the output of the convolutional block channel by channel and pixel by pixel, and this structure is repeated three times to deepen the feature extraction and denoising process. After the last cycle, upsampling is performed through a 3×3 deconvolution layer with a stride of 2 to restore the spatial size of the feature map while ensuring that the denoising effect is consistent at all scales.

[0085] 4-3 concats the features extracted by the two branches of 4-2 in the channel dimension, and further integrates the features through a 3×3 convolution layer and a subsequent 1×1 convolution layer, ensuring that the final output feature map enriches the image information and enhances the expression ability while denoising.

[0086] Step 5: CAM Condition Adaptation Module

[0087] First, it receives the input image of low-light foggy weather conditions and passes it through the preliminary feature extraction layer and deep feature analysis. This module generates conditional encoding through the fully connected layer, quantizes the conditions such as image illumination and fog density, and adaptively adjusts the weights of the three enhancement modules.

[0088] 5-1 First, receive the ExDark-fog low-light foggy weather condition image input. The image first passes through a preliminary feature extraction layer, using a 3×3 convolution kernel and a stride of 1 for convolution, and the size of the output feature map is the same as the input image, with 64 channels.

[0089] 5-2 The 5-1 feature is input into the deep feature analysis, which consists of three convolutional layers Conv1, Conv v 2. Con v The 3-layer cascade consists of 3×3, 5×5, and 7×7 convolution kernels with strides of 1, 2, and 2. Each convolution layer is followed by batch normalization and LeakyReLU activation function to deeply analyze image features.

[0090] 5-3 reduces the dimension of the output of the deep feature analysis in 5-2 through a 3×3 average pooling layer with a stride of 2, halves the size, and performs feature compression through a 1×1 convolution.

[0091] The reduced features are then fed into a fully connected layer to generate conditional encodings that quantify image conditions such as illumination and fog density. After full connection and softmax function, weight vectors w1, w2, and w3 are generated to ensure that the sum of the weights is 1.

[0092] The 5-5 weight vector is multiplied and added with the output feature maps F1, F2, and F3 of the LEM, FREM, and DEM modules, as shown in formula (8), and then adjusted to 3 channels after a 1×1 convolution.

[0093] Output=w1F1+w2F2+W3F3 (6)

[0094] Step 6: AODM object detection module

[0095] The enhanced feature map is sent to the AODM module to first extract features through CBS, and then undergo multiple downsampling and upsampling operations, combined with C2f and SPPFELAN to enhance feature representation. The feature map is spliced ​​and channel fused to enhance multi-scale feature integration, and finally input into AECB and detection head to achieve accurate detection, such as Figure 1 shown.

[0096] 6-1 The feature map output by 5-5 is sent to the AODM target detection module, which first passes through four CBS modules for feature extraction, and then passes through the C2f module. The preliminary feature obtained is represented as C1 ; C1 is further processed by the downsampling SCDown and C2f modules, and the result is represented as C2; C2 is further processed by the downsampling SCDown, C2f and SPPFELAN modules, and the result is represented as C3; the SPPFELAN module structure is as follows Figure 8 As shown, it is used to enhance the feature extraction capability of the backbone.

[0097] 6-2 upsamples C3 in 6-1 and concatenates it with C2, and then passes it through the C2f module, and the result is represented by C4; C4 continues to be upsampled and fused with C1, and the result is represented by C5, realizing the effective integration of features of different resolutions; then, C5 is processed by CBS and concatenated with C4, and the concatenated result is passed through the C2f module again, and the result is represented by C6; finally, C6 is concatenated with C3 after downsampling SCDown, and the result is represented by C7; this process not only realizes the fusion of cross-scale features, but also optimizes the hierarchical structure of features.

[0098] 6-3 Send C5, C6, and C7 in 6-2 into the attention mechanism enhanced convolution module (Attention-Enhanced Convolution Block) AECB module respectively, such as Fig. 9As shown in the figure. AECB consists of two parts: the local branch and the global branch. The local branch first uses 1×1 convolution to adjust the channel dimension to compress and expand the number of channels of the feature map. Then, a channel shuffle operation is performed to group the input tensor along the channel dimension, and a depth-wise separable convolution is used in each group to rearrange and mix the channels. After the shuffle, the results of each group are concatenated on the channel dimension to form a new output tensor. Finally, a convolution layer is used to further extract features.

[0099] The global branch generates three tensors of query (Q), key (K), and value (C) through 1×1 convolution and 3×3 depthwise convolution, and their shapes are H×W×C. The query Q is reshaped into Q∈R HXWXC , the key K is reshaped into K∈R HXWXC The attention map A∈R is calculated through the interaction of Q and K c×c , which reduces the computational burden of computing conventional large-scale attention maps. The output F of the global branch att Defined as:

[0100] F att =W 1x1 (Attention(Q, K, V))+Y (7)

[0101] The attention function is defined as:

[0102]

[0103] Where α is a learnable scaling parameter that controls the size of the K and Q matrix multiplications before applying the softmax function. The final output F of the AECB module is out It is obtained by adding the output of the global branch and the local branch:

[0104] F out =F conv +F att (9)

[0105] This module combines convolution operations and attention mechanisms to capture local features and global contextual information of images.

[0106] 6-4 The features processed by the AECB module are then sent to the detection head for target detection. In order to achieve accurate training of target detection tasks under low illumination and fog superposition conditions, a comprehensive loss function is used, which combines classification loss, positioning loss and confidence loss to optimize network performance. The loss function is defined as follows:

[0107] Classification loss L class Calculate the difference between the predicted class probability and the true class label, using the cross entropy loss formula:

[0108]

[0109] in, represents the category set, y c is the one-hot encoding of the true label, and p c is the probability that the model predicts class c.

[0110] Positioning loss L loc Measure the difference between the predicted bounding box and the true bounding box, usually using mean absolute error or squared error:

[0111]

[0112] here, are the predicted bounding box parameters, b i are the corresponding ground-truth bounding box parameters.

[0113] Confidence loss L conf Evaluate the model's confidence in whether the predicted box contains the target:

[0114]

[0115] in, is the confidence of the prediction, I obj is a binary variable indicating the presence of the target.

[0116] Finally, the total loss function L total For weighted sum:

[0117] L total =λ class L class +λ loc L loc +λ conf I conf (13)

[0118] λ class , loc and λ conf is a weight parameter used to balance classification, localization and confidence losses.

[0119] Step 7: Testing the DFLLOD-Net Network

[0120] 7-1 Send the ExDark-fog dataset to the low illumination enhancement module LEM, defogging enhancement module FREM and denoising enhancement module DEM of DFLLOD-Net for enhancement processing;

[0121] 7-2 uses the conditional adaptive module CAM to dynamically adjust the weight of the 7-1 module according to the image's lighting and fog density conditions to optimize the image enhancement effect;

[0122] 7-3 Send the enhanced image obtained in 7-2 to the attention target detection module AODM for feature extraction and target detection;

[0123] 7-4 Visualize the detected results.

Claims

1. A target detection method in low-light superimposed foggy weather based on DFLLOD-Net, which is characterized by the following method steps: (1) Constructing the low-light foggy image dataset ExDark-fog includes the following steps: (1.1)) Based on the public ExDark dataset, it contains 7363 images covering 12 different target categories. These images are taken under various low-light conditions. (1.2) Use the Monodepth2 network to estimate the depth of each image in the ExDark dataset and generate a depth map; combine the depth map and fog concentration coefficient to generate transmission; (1.3)) Apply the atmospheric scattering model to synthesize foggy images; (1.4) After synthesis, all images of the ExDark-fog image dataset are uniformly resized to 640x640x3 pixels. The ExDark-fog dataset is divided into training set, test set and validation set, with the ratios of 3 / 4, 1 / 8 and 1 / 8 respectively. (2) The design of the Low-Light Enhancement Module (LEM), as shown in FIG2 , includes the following steps: (2.1) First, the low-light foggy images in the ExDark-fog dataset and their self-attention features are input into the encoding module, as shown in Figure 2. The module consists of three low-light adaptive encoding blocks (LAEBs), as shown in Figure 3. Each LAEB uses a series of convolutional layers, including 1×1, 3×3, and 5×5 convolution kernels, followed by GeLU activation functions, and 1×1 convolution kernels for channel mixing. In addition, each LAEB also contains a downsampling module DownSample, which uses a 3×3 convolution kernel with a stride of 2 and a ReLU activation function to reduce the spatial size of the feature map. (2.2) After being processed by the Encoding Module, the feature data is then sent to the Latent Space Sampling Module (LSSM), as shown in Figure 2. In this module, the statistical calculation operator is first applied to analyze the features, from which the mean μ and variance σ are calculated. Using these statistics, the sampled feature samples z are generated. (2.3) After the latent space sampling module LSSM successfully obtains feature samples, these samples are combined with the features extracted by the self-attention mechanism. The combined features are then sent to the decoding module (Decoding Module), as shown in Figure 2. The decoding module consists of three low-light adaptive decoding blocks (LADB), as shown in Figure 3. Each LADB is equipped with a 3×3 convolution kernel for feature refinement, a GeLU activation function to introduce nonlinearity, and a 3×3 deconvolution kernel with an upsampling layer to gradually restore the high resolution of the image. (2.4) The output of the Decoding Module is added to the input to obtain the enhanced feature map. (3) The design of the Fog Removal Enhancement Module (FREM), as shown in FIG4 , includes the following steps: (3.1) The ExDark-fog low-light foggy image is sent to the dehazing enhancement module FREM. First, the module consists of two convolutional layers. Each convolutional layer uses a 3×3 convolution kernel with a step size of l to extract the preliminary features of the image. (3.2) The features initially extracted in (3.1) are sent to the multi-scale feature extraction module (Multi-scale Features) MSF, as shown in Figure 4. This module consists of three consecutive multi-scale feature extraction units (Multi-scale Feature Unit) MSFU, as shown in Figure 5. Each unit contains four parallel branches, which extract features of different scales through convolution kernels of different sizes, and enhance the model's ability to capture multi-scale information of images through Concat and ReLU activation functions. (3.3) After the MSF module in (3.2) completes the multi-scale feature extraction, the CSA (Channel and Spatial Attention) attention module is introduced, as shown in Figure 6. The CSA module is designed as two parallel branches: the channel attention (CA) branch and the spatial attention (SA) branch. The CA branch uses a global pooling operation followed by a fully connected layer to generate a channel attention map that reflects the importance of different channels. The SA branch captures global context information through 1×1 convolution, and further generates a spatial attention map through global average pooling and global maximum pooling. (3.4) After the CSA attention module is processed, the network structure continues with two stages of jump connections, as shown in Figure 4. First, the feature channel is reduced or increased in dimension using a 1×1 convolution kernel to ensure that the output feature maps generated by different levels are compatible in dimension. Subsequently, channel matching is further achieved through a 3×3 convolution kernel, and these feature maps are added and fused, thereby integrating information between different levels and enhancing the overall feature representation capability. (4) Design a denoising enhancement module (DEM), as shown in FIG7 , including the following steps: (4.1) The ExDark-fog low-light foggy weather image is sent to the denoising enhancement module DEM, which first passes through a CBR and then is divided into two branches for denoising enhancement: the first branch first introduces a 1×1 fully connected layer and a Sigmoid activation function to generate an attention map. Then, the number of channels is adjusted through a 1×1 convolution layer to fuse with the feature map of the second branch. (4.2) The second branch first adjusts the number of channels through a 1×1 convolutional layer to accommodate subsequent processing requirements. Then, the spatial dimension of the feature map is reduced through a 3×3 convolutional layer and downsampling with a stride of 2, and the denoising process begins at the same time. Next, the feature map passes through two 3×3 convolutional layers with a stride of 1 and a padding of 1, and the output of the downsampling layer is added to the output of the convolution block channel by channel and pixel by pixel. This structure is repeated three times to deepen the feature extraction and denoising process. After the last cycle, upsampling is performed through a 3×3 deconvolution layer with a stride of 2 to restore the spatial size of the feature map while ensuring that the denoising effect is consistent at all scales. (4.3) The features extracted by the two branches are concat-merged in the channel dimension and further integrated through a 3×3 convolutional layer and a subsequent 1×1 convolutional layer, ensuring that the final output feature map enriches the image information and enhances its expressiveness while denoising. (5) The design of the Conditional Adaptive Module (CAM), as shown in FIG1 , includes the following steps: (5.1) First, we receive the ExDark-fog low-light foggy weather condition image input. The image first passes through a preliminary feature extraction layer, using a 3×3 convolution kernel and a stride of 1 for convolution. (5.2) The features in (5.1) are fed into a deep feature analysis, which consists of three convolutional layers Conv1, Conv2, and Conv3, with 3×3, 5×5, and 7×7 convolution kernels, respectively, and strides of 1, 2, and 2. Each convolutional layer is followed by batch normalization and LeakyReLU activation function to deeply analyze the image features. (5.3) The output of the deep feature analysis in (5.2) is reduced in dimension through a 3×3 average pooling layer, and feature compression is performed through a 1×1 convolution. (5.4) The reduced features are then fed into a fully connected layer to generate conditional encodings that quantize the image conditions such as illumination and fog density. After full connection and softmax function, the weight vectors W1, W2, and W3 are generated to ensure that the sum of the weights is 1. (5.5) The weight vector is multiplied and added with the output of the LEM, FREM and DEM modules, and then adjusted to 3 channels after a 1×1 convolution. (6) The design of the Attention-based Object Detection Module (AODM), as shown in Figure 1, includes the following steps: (6.1) The feature map output by (5.5) is sent to the AODM target detection module, which first passes through four CBS modules for feature extraction and then passes through the C2f module. The preliminary feature obtained is denoted as C1. C1 is further processed by the downsampling SCDown and C2f modules, and the result is denoted as C2. C2 is further processed by the downsampling SCDown, C2f and SPPFELAN modules, and the result is denoted as C3. The SPPFELAN module structure, as shown in Figure 8, is used to enhance the feature extraction capability. (6.2) C3 in (6.1) is upsampled and concatenated with C2, and then passed through the C2f module, the result is represented as C4; C4 is further upsampled and fused with C1, the result is represented as C5, which realizes the effective integration of features of different resolutions; then, C5 is processed by CBS and concatenated with C4, and the concatenated result is passed through the C2f module, the result is represented as C6; finally, C6 is downsampled SCDown and concatenated with C3, the result is represented as C7; this process not only realizes the fusion of cross-scale features, but also optimizes the hierarchical structure of features. (6.3) C5, C6, and C7 in (6.2) are respectively fed into the Attention-Enhanced Convolution Block AECB module, as shown in Figure 9. This module is a combination of convolution operation and attention mechanism to capture local features and global context information of the image. (6.4) The features processed by the AECB module are then sent to the detection head for target detection. (7) Testing the entire DFLLOD-Net target detection network under low illumination and foggy weather conditions includes the following steps: (7.1)) Send the ExDark-fog dataset to the low illumination enhancement module LEM, the defogging enhancement module FREM and the denoising enhancement module DEM of DFLLOD-Net for enhancement processing; (7.2) Using the conditional adaptive module CAM to dynamically adjust the weight of the module (7.1) according to the image illumination and fog density conditions to optimize the image enhancement effect; (7.3) The enhanced image obtained in (7.2) is sent to the attention object detection module AODM for feature extraction and object detection; (7.4) Visualize the detected results.

Citation Information

Cited By

  • Complex environment image semantic segmentation method based on fine-grained enhancement network

    CN119723086A

  • Complex environment image semantic segmentation method based on fine-grained enhancement network

    CN119723086B

  • Road condition monitoring method and system under complex light based on lightweight enhanced network

    CN119888647A

  • Method and System for Road Condition Monitoring under Complex Lighting Conditions Based on a Lightweight Enhancement Network

    CN119888647B

  • Underwater image enhancement method and system based on combination of multiple convolution models and attention mechanism

    CN120278912A