A method and system for detecting haze visibility under low light conditions at night
By deeply integrating nighttime video images with meteorological factors, and employing a light effect suppression network and a self-attention module, the accuracy problem of visibility detection under low-light conditions at night has been solved, enabling reliable detection and early warning of nighttime visibility, and improving technical support for traffic safety and environmental monitoring.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NANJING UNIV OF POSTS & TELECOMM
- Filing Date
- 2026-03-19
- Publication Date
- 2026-06-23
AI Technical Summary
Existing visibility detection technologies struggle to effectively identify target objects in low-light conditions at night, leading to increased traffic safety risks. Furthermore, existing equipment is expensive and has limited coverage.
By deeply fusing nighttime video image information with real-time relevant meteorological factors, an optical effect suppression network is used to decompose the image. Combined with a dual-path self-attention module and cross-scale feature fusion, visual features of image visibility are extracted and dynamically fused with high-dimensional meteorological factor features. Visibility prediction is then performed using the DMFNet network.
It achieves accurate and reliable visibility detection under low-light conditions at night, improves the model's ability to perceive local luminescence details and global haze concentration, provides a reliable end-to-end early warning system, and supports nighttime traffic safety and environmental monitoring.
Smart Images
Figure CN121884068B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of visibility detection technology, specifically relating to a method and system for detecting visibility in fog and haze under low-light conditions at night. Background Technology
[0002] Atmospheric visibility is a key meteorological parameter characterizing atmospheric transparency, significantly impacting public safety, transportation, environmental monitoring, and industrial production. In road traffic, low visibility (such as fog, haze, smoke, rain, and snow) is a leading cause of serious chain-reaction traffic accidents. In maritime navigation, visibility directly affects ship safety and port scheduling efficiency. In urban management, visibility data is crucial for air pollution early warning and public health protection. Therefore, developing high-precision, wide-coverage, and low-cost automated visibility monitoring technology has urgent social demand and market value. Visibility can be further divided into daytime visibility and nighttime visibility. Generally, visibility refers to the maximum horizontal distance at which a person with normal vision can see and distinguish a black, moderately sized object against a cloudless sky background during the day. Nighttime visibility, on the other hand, refers to the maximum horizontal distance at which a medium-intensity luminous object can be seen and identified.
[0003] Current mainstream visibility monitoring methods have significant shortcomings, making it difficult to meet the needs of all-weather, large-scale deployment. Common visibility detection methods include scattering visibility meters and transmission visibility meters. These devices are based on the principle of atmospheric extinction physics, require precisely calibrated optical components, have low reliability in complex environments, and are expensive with limited detection coverage. With the development of computer vision and deep learning technologies, daytime visibility estimation methods based on visible light images are becoming increasingly widely used. The principle is to use deep learning networks to extract features such as contrast attenuation, color distortion, or dark channels of objects (such as mountains, buildings, and signs) under natural lighting conditions to estimate visibility. However, at night, due to the disappearance of natural light, reliance on artificial light sources (vehicle lights, streetlights) with limited intensity and uneven distribution leads to low overall image brightness and poor signal-to-noise ratio. The edges, textures, and color information of target objects are severely degraded or disappeared in low light, and traditional image feature extraction (such as gradient and contrast) fails. At the same time, glare and halo effects cause details in the vicinity of key targets to be obscured, making it difficult to identify distant targets.
[0004] Nighttime hours are a high-risk period for traffic accidents and environmental hazards, with low visibility exacerbating the risks. However, the unique characteristics of the nighttime environment—extremely low ambient light, dominance of artificial light sources, complex light pollution, and concealed features—lead to the ineffectiveness of existing detection technologies. Therefore, there is an urgent need for a visibility detection technology specifically designed for nighttime scenarios that can overcome the limitations of low-light conditions. Summary of the Invention
[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method and system for detecting fog and haze visibility under low light conditions at night. By deeply fusing nighttime video image information with real-time relevant meteorological factors, accurate and reliable nighttime visibility detection is achieved.
[0006] This invention provides the following technical solution:
[0007] Firstly, a method for detecting visibility in haze under low-light conditions at night is provided, including:
[0008] Acquire video data of low-light nighttime scenes and meteorological factor data related to haze, and select corresponding nighttime image data based on the timestamp of the meteorological factor data;
[0009] Nighttime image data is decomposed into a light effect layer and a background layer, and a light effect suppression network is used to locate and remove light effect regions to obtain a refined background image.
[0010] Multi-scale feature extraction is performed on the refined background image. A dual-path self-attention module is used to adaptively focus on key regions. The image visibility visual features are obtained through cross-scale feature fusion, cross-scale feature interaction, and multi-head self-attention calculation.
[0011] Feature extraction is performed on meteorological factor data to obtain high-dimensional meteorological factor features;
[0012] The image visibility visual features are dynamically fused with high-dimensional meteorological factor features. The resulting cross-modal deep fusion features are then input into the visibility prediction regression head to calculate the visibility prediction value.
[0013] Furthermore, the meteorological factor data related to haze includes horizontal wind speed, vertical wind speed, relative humidity, temperature, and air pressure.
[0014] Furthermore, the process of decomposing the nighttime image data into a light effect layer and a background layer includes:
[0015] Assume the image model is:
[0016] ;
[0017] Where I represents the nighttime image, H represents the reflectivity layer, L represents the shadow layer, and G represents the light effect layer. For element-wise multiplication;
[0018] The decomposition target is the background layer without the light effect layer G. :
[0019] = ;
[0020] Separation of the light effect layer and the background layer is achieved by minimizing the overlap between them in the gradient space, as shown in the following formula:
[0021] ;
[0022] in, Represents the gradient exclusion function. and These represent the optical effect layer G and the background layer, respectively, using bilinear interpolation downsampling. , and They are respectively and Gradient mapping at the nth scale, and As the normalization factor, The function is used to perform saturation mapping on gradient magnitude. For element-wise multiplication, This represents the Frobenius norm, used to globally constrain the common response of the light effect layer and the background layer in the gradient domain.
[0023] Furthermore, the method of using a light effect suppression network to locate and remove the light effect region includes:
[0024] The optical effect domain feature tensor and the non-optical effect reference domain feature tensor are input into the optical effect suppression network. The encoder at the front end of the generator performs feature extraction and optical effect layer guidance to obtain the basic feature map and intermediate layer feature map.
[0025] A classifier is used to classify the intermediate layer feature maps. First, the response weights of each feature channel for the light effect category are learned through global average pooling. Then, the activation weight map of the light effect interference region is generated in the spatial dimension by using the class activation map (CAM). Finally, the basic feature map output by the encoder is multiplied element-wise with the CAM weights to obtain the attention feature map.
[0026] Employing an attention loss function based on domain adversarial mechanism The classifier is driven to accurately capture the light effect region and provide feedback to guide the generator. Its formula is defined as follows:
[0027] ;
[0028] In the formula, E represents the probability distribution of the auxiliary classifier's output. For mathematical expectation, For the characteristic tensor of the optical effect domain, For the feature tensor of the reference domain without optical effect;
[0029] The image is reconstructed using the feature matrix weighted by the attention feature map through the decoder, eliminating the luminous effect components and filling in the background details to obtain a refined background image with the luminous effect layer removed. .
[0030] Furthermore, the multi-scale feature extraction of the refined background image, employing a dual-path self-attention module for adaptive focusing on key regions, includes:
[0031] The refined background image is input into the lightweight EfficientNet-B3 basic feature extractor. Downsampling is performed at the beginning of each stage of EfficientNet-B3 to select representative feature maps at different scales.
[0032] Feature maps of different scales are input into a dual-path self-attention module to adaptively focus on regions that significantly affect visibility;
[0033] The dual-path self-attention module consists of two parallel branches: spatial attention and channel attention. and channel attention The formula is as follows:
[0034] ;
[0035] ;
[0036] Where Conv represents convolution, Concat represents connection, AvgPool represents average pooling, MLP represents a multilayer perceptron network, F represents the input feature map, and MaxPool represents max pooling. These are learnable parameters.
[0037] Furthermore, the image visibility visual features are obtained through cross-scale feature fusion, cross-scale feature interaction, and multi-head self-attention computation, including:
[0038] After attention weighting of features at each scale, the number of channels and size of all scale feature maps are aligned by a 1*1 convolution operation, and then an adaptive gating mechanism is introduced to perform cross-scale feature fusion.
[0039] Multi-branch convolution operations are performed on the feature map after cross-scale feature fusion to obtain features of different dimensions, and the features of different dimensions are then serialized.
[0040] Multi-head self-attention calculation is performed on the serialized vector, and all attention heads are concatenated and linear transformation is performed to obtain the image visibility visual features.
[0041] Furthermore, the feature extraction of the meteorological factor data includes:
[0042] Meteorological factor data are used to construct an initial 5-dimensional meteorological feature vector. The initial 5-dimensional meteorological feature vector is then subjected to Z-score standardization to obtain the standardized meteorological feature vector.
[0043] The standardized meteorological feature vectors are fed into a cascaded feature coding network. They pass through the first fully connected layer and the second fully connected layer in sequence, and the discrete meteorological data located in the 5-dimensional low-order physical space are initially mapped and projected to the high-dimensional semantic feature space.
[0044] The high-dimensional features that have undergone initial dimensionality enhancement are input into the attention layer, and attention weights are adaptively allocated by calculating the correlation between elements within the features.
[0045] The features, after being weighted and enhanced by the self-attention layer, are fed into the third fully connected layer for linear mapping and dimensional adjustment to obtain high-dimensional meteorological factor features.
[0046] Furthermore, the dynamic fusion of image visibility visual features and high-dimensional meteorological factor features, followed by inputting the resulting cross-modal deep fusion features into the visibility prediction regression head to calculate the visibility prediction value, includes:
[0047] High-dimensional meteorological factor features are transformed through three independent linear transformation layers. 𝑚𝑒𝑡 Mapping to a query vector matrix, the image visibility visual feature is represented by 𝐹 𝑖𝑚𝑔 The mapping is represented by a key vector matrix and a value vector matrix, and the mapping formula is as follows:
[0048] ;
[0049] ;
[0050] ;
[0051] in, Characteristics of high-dimensional meteorological factors The query vector matrix obtained by mapping, Visual features for image visibility The key vector matrix obtained by mapping, Visual features for image visibility The value vector matrix obtained by mapping , , Let be a learnable weight matrix, and d be the feature dimension. and Here, represents the dimensions of the key vector and value vector in the attention mechanism, respectively, and R is the real number field.
[0052] by As a guide, dynamic query Cross-modal attention features are calculated by selecting the visual feature region that best matches the current weather conditions. :
[0053] ;
[0054] in, Represents the normalization function;
[0055] Cross-modal attention features Characteristics of high-dimensional meteorological factors Residual connections are performed, followed by layer normalization, to obtain cross-modal deep fusion features. :
[0056] ;
[0057] in, For layer normalization function;
[0058] The cross-modal deep fusion features are input into the visibility prediction regression head to calculate the visibility prediction value. :
[0059] ;
[0060] in, This represents a multilayer perceptron network used for... Perform nonlinear feature extraction; and These are the weight matrix and bias term of the output layer, respectively.
[0061] Furthermore, the method also includes:
[0062] Based on the visibility prediction value and the preset graded response mechanism, the visibility decision for haze is obtained;
[0063] The decision on visibility in haze is transmitted to the computing node, which then sends the decision log to the cloud platform for storage and sends real-time early warning information to various management platforms at the airport through the cloud platform.
[0064] Secondly, a system for detecting fog and haze visibility under low-light conditions at night is provided, including:
[0065] The data acquisition module is used to acquire video data and meteorological factor data related to haze in low-light nighttime scenes, and select the corresponding nighttime image data based on the timestamp of the meteorological factor data.
[0066] The image preprocessing module is used to decompose nighttime image data into a light effect layer and a background layer, and uses a light effect suppression network to locate and remove light effect regions to obtain a refined background image.
[0067] The image visibility visual feature extraction module is used to extract multi-scale features from the refined background image. It adopts a dual-path self-attention module to adaptively focus on key luminous targets, and obtains the image visibility visual features through cross-scale feature fusion, cross-scale feature interaction and multi-head self-attention calculation.
[0068] The meteorological factor feature extraction module is used to extract features from meteorological factor data to obtain high-dimensional meteorological factor features;
[0069] The feature fusion module is used to dynamically fuse image visibility visual features with high-dimensional meteorological factor features;
[0070] The visibility prediction module is used to input the obtained cross-modal deep fusion features into the visibility prediction regression head to calculate the visibility prediction value.
[0071] Compared with the prior art, the beneficial effects of the present invention are:
[0072] (1) This invention addresses the feature degradation and glare interference caused by low illumination and strong artificial light sources at night. It decomposes nighttime images into light effect layers and background layers through a physical imaging model, and uses a light effect suppression network composed of generator and classifier combined with class activation map to achieve accurate positioning and removal of light effect areas, thereby restoring the true background visual features at night to the greatest extent.
[0073] (2) This invention extracts features from the refined background image at multiple scales, uses a dual-path self-attention module to adaptively focus on key areas, and establishes a cross-scale global dependency relationship in the feature space through cross-scale feature fusion, cross-scale feature interaction and multi-head self-attention calculation, which greatly enhances the model’s ability to deeply perceive local luminous details and global haze concentration.
[0074] (3) This invention innovatively introduces five meteorological factors that are strongly correlated with haze: horizontal wind speed, vertical wind speed, relative humidity, temperature, and air pressure. By extracting features from meteorological factor data, high-dimensional meteorological factor features are obtained. Then, the image visibility visual features are dynamically fused with the high-dimensional meteorological factor features. The obtained cross-modal deep fusion features are then input into the visibility prediction regression head to calculate the visibility prediction value. Unlike the pure visual solution of traditional technology, the cross-modal fusion proposed in this invention significantly improves the robustness of detection and can be combined with an end-to-end early warning system to achieve data spatiotemporal alignment and automated hierarchical alarm, providing a reliable technical closed loop for the nighttime operation of key places such as airports.
[0075] (4) This invention provides an innovative deep learning network architecture DMFNet (DarkSightMeteorological Fusion Network), which achieves accurate and reliable nighttime visibility detection by deeply fusing nighttime video image information with real-time relevant meteorological factors. This method provides key technical support for nighttime traffic safety and environmental monitoring by making full use of the ability of deep learning to process complex visual information and unstructured data, as well as the physical constraints provided by meteorological data. Attached Figure Description
[0076] Figure 1 This is a flowchart illustrating the method for detecting fog and haze visibility under low-light conditions at night, as described in an embodiment of the present invention.
[0077] Figure 2 This is a diagram of the algorithm structure for locating and removing optical effect regions using an optical effect suppression network in an embodiment of the present invention.
[0078] Figure 3 This is a network structure diagram for extracting visual features of image visibility in an embodiment of the present invention;
[0079] Figure 4 This is a network structure diagram for extracting high-dimensional meteorological factor features in an embodiment of the present invention;
[0080] Figure 5 This is a time-series diagram of fog and haze visibility detection under low-light conditions at night, as described in an embodiment of the present invention. Detailed Implementation
[0081] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.
[0082] The term "and / or" simply describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship.
[0083] Example 1
[0084] like Figure 1 As shown, this embodiment provides a method for detecting visibility in fog and haze under low-light conditions at night, including:
[0085] Acquire video data of low-light nighttime scenes and meteorological factor data related to haze, and select corresponding nighttime image data based on the timestamp of the meteorological factor data;
[0086] Nighttime image data is decomposed into a light effect layer and a background layer, and a light effect suppression network is used to locate and remove light effect regions to obtain a refined background image.
[0087] Multi-scale feature extraction is performed on the refined background image. A dual-path self-attention module is used to adaptively focus on key regions. The image visibility visual features are obtained through cross-scale feature fusion, cross-scale feature interaction, and multi-head self-attention calculation.
[0088] Feature extraction is performed on meteorological factor data to obtain high-dimensional meteorological factor features;
[0089] The image visibility visual features are dynamically fused with high-dimensional meteorological factor features. The resulting cross-modal deep fusion features are then input into the visibility prediction regression head to calculate the visibility prediction value.
[0090] Example 2
[0091] This embodiment takes an airport scenario as an example to provide a method for detecting fog and haze visibility under low-light conditions at night. The steps are as follows:
[0092] Step 1: Acquire video data of low-light nighttime scenes and meteorological factor data related to haze. Based on the timestamp of the meteorological factor data, select the corresponding nighttime image data.
[0093] In this embodiment, cameras were deployed on the top floor of the airport control tower and the roof of the terminal building to capture video data in low-light nighttime scenarios covering the runways, taxiways, and aprons. Simultaneously, the airport's Automatic Weather Observation System (AWOS) was used to collect relevant meteorological factor data at fixed time intervals. Through investigation and research, five meteorological factors related to haze formation were selected, including horizontal wind speed (WS), vertical wind speed (WV), relative humidity (RH), temperature (TEMP), and air pressure (PRES). Furthermore, the video stream was transmitted using the RTSP protocol, and the meteorological data was transmitted using the MQTT protocol. By accessing an NTP time server and using the meteorological data timestamp as a reference, corresponding video frame image data was selected to ensure consistency in data acquisition time.
[0094] Step 2: Decompose the nighttime image data into a light effect layer and a background layer, and use a light effect suppression network to locate and remove the light effect region to obtain a refined background image.
[0095] Nighttime images suffer from low light, strong glare, and feature degradation, severely impacting visual quality and subsequent applications. Traditional enhancement methods primarily focus on increasing brightness in dark areas, easily leading to overexposure and distortion in bright areas. This step addresses feature degradation and glare interference caused by low light and strong artificial light sources at night. A physical imaging model decomposes the nighttime image into a light effect layer and a background layer. A light effect suppression network, composed of a generator, classifier, and class activation map, is used to accurately locate and remove light effect regions, maximizing the restoration of realistic nighttime background visual features. The specific method is as follows:
[0096] Step 2.1: Decompose the nighttime image into a light effect layer and a background layer using a physical imaging model.
[0097] Assume the image model is:
[0098] ;
[0099] Where I represents the nighttime image, H represents the reflectivity layer, L represents the shadow layer, and G represents the light effect layer. For element-wise multiplication;
[0100] The decomposition target is the background layer without the light effect layer G. :
[0101] = ;
[0102] To achieve the effect between the light effect layer G and the background layer To achieve complete decoupling, this invention designs a gradient exclusion loss function. Considering that the optical effect layer G exhibits smooth, gradually varying characteristics in space, and its gradient distribution shows a short-tailed effect, while the background layer... It contains rich texture details, and its gradient distribution exhibits a long-tail effect. Based on this, the present invention achieves separation between the light effect layer and the background layer by minimizing the overlap between them in the gradient space, as shown in the following formula:
[0103] ;
[0104] in, Represents the gradient exclusion function. and These represent the optical effect layer G and the background layer, respectively, using bilinear interpolation downsampling. , and They are respectively and Gradient mapping at the nth scale, and As the normalization factor, The function is used to perform saturation mapping on gradient magnitude. This is element-wise multiplication, which calculates the common response of the two layers in the gradient domain through element-wise multiplication; This represents the Frobenius norm, used to globally constrain the common response of the light effect layer and the background layer in the gradient domain. During optimization, this model ensures that the light effect layer and the background layer do not simultaneously retain high-frequency gradient information at the same location, thereby achieving physical-level image component separation.
[0105] Step 2.2: Use a light effect suppression network to locate and remove the light effect region.
[0106] To achieve precise localization and effective suppression of luminous effect regions in nighttime images, this invention constructs a luminous effect suppression network based on a combination of generative adversarial mechanisms and class activation mapping (CAM). Figure 2 As shown, the network mainly consists of a generator and a classifier.
[0107] Step 1: Input the dual-domain feature tensor.
[0108] Two feature tensors from different data domains are constructed at the input of the optical effect suppression network. One is the optical effect domain feature tensor. This tensor is derived from the initial estimate of the background image without the optical effect layer. It is formed by splicing the optical effect layer G in the channel dimension; the second is the feature tensor of the reference domain without optical effect. Based on real reference images without lighting effects All-zero matrix diagram with consistent size Composed of multiple parts. Complete zero image. The introduction of ensures strict alignment of the channel dimensions, and since its value is zero, it does not inject any prior information about optical effects into the reference domain.
[0109] The second step is encoder feature extraction and optical effect layer guidance.
[0110] The generator front end employs a VGG network architecture as the encoder (basic feature extractor). When the two input feature tensors enter the encoder, multiple convolutional operations extract feature maps containing spatial information and high-dimensional semantics from the input image. Simultaneously, to enhance the network's sensitivity to lighting effects, the predicted lighting effect layer G is adaptively scaled and modulated multiplicatively with the feature maps of the encoder's intermediate layers. This lighting effect-guided mechanism prompts the encoder to actively tilt its representational capabilities towards lighting effect regions during the feature extraction stage, ultimately yielding the basic feature map and intermediate layer feature maps.
[0111] Step 3: Classifier discrimination and attention feature map generation.
[0112] The intermediate feature maps output by the encoder are fed into an auxiliary classifier, which performs a domain classification task to determine whether the currently encoded features originate from... still In the discrimination process, the classifier first learns the response weights of each feature channel for the light effect category through global average pooling (GAP) operation; then, it uses the class activation map (CAM) to generate an activation weight map of the light effect interference region in the spatial dimension; finally, it multiplies the basic feature map output by the encoder with the CAM weights element-wise to generate the attention feature map.
[0113] Step 4: Adversarial game and loss function optimization.
[0114] Employing an attention loss function based on domain adversarial mechanism The classifier is driven to accurately capture the light effect region and provide feedback to guide the generator. Its formula is defined as follows:
[0115] ;
[0116] In the formula, E represents the probability distribution of the auxiliary classifier's output. For mathematical expectation, For the characteristic tensor of the optical effect domain, It is the feature tensor of the reference domain without optical effect.
[0117] The loss function forces the classifier to accurately locate the region where the light effect is concentrated, and uses the activation state of the region to constrain the generator, forcing the generator to effectively erase these high-weight light effect features in subsequent processing.
[0118] Step 5: Decoder reconstruction and output of the image after removing the light effect layer.
[0119] The decoder receives the feature matrix weighted by the attention feature map. Since the attention mechanism assigns extremely high suppression weights to the luminous effect regions, the decoder uses the attention-weighted feature matrix to reconstruct the image through a series of upsampling and deconvolution operations. During reconstruction, the decoder selectively eliminates luminous effect components and fills in background details, resulting in a refined background image with the luminous effect layer removed. .
[0120] Step 3: Perform multi-scale feature extraction on the refined background image, capturing visibility information at different scales from original pixel-level details to high-level semantic features. A dual-path self-attention module adaptively focuses on key regions. Through cross-scale feature fusion, cross-scale feature interaction, and multi-head self-attention computation, the model's ability to perceive near details and distant haze distribution is enhanced, resulting in visual features of image visibility. For example... Figure 3 As shown, the specific implementation steps are as follows:
[0121] Step 3.1: Multi-scale feature extraction based on lightweight backbone network.
[0122] The refined background image is input into the lightweight EfficientNet-B3 basic feature extractor. Each stage of EfficientNet-B3 consists of multiple MBConv (Mobile Inverted Bottleneck Convolution) modules stacked together. Downsampling is performed at the beginning of each stage of EfficientNet-B3 to select representative feature maps of different scales.
[0123] In this embodiment, four representative feature maps are selected, with sizes of 1 / 2, 1 / 4, 1 / 8, and 1 / 16 of the original image, respectively. These features of different scales are then input into the dual-path self-attention module.
[0124] Step 3.2: Focusing on key features based on the dual-path self-attention module.
[0125] The feature maps at the four different scales mentioned above are input into the dual-path self-attention module to guide the network to adaptively focus on regions that significantly affect visibility. The dual-path self-attention module consists of two parallel branches: spatial attention and channel attention. Spatial attention focuses on light areas (runway lights, building lights) in the image, extracting visibility information. Channel attention enhances feature channels related to visibility. and channel attention The formula is as follows:
[0126] ;
[0127] ;
[0128] Where Conv represents convolution, Concat represents connection, AvgPool represents average pooling, MLP represents a multilayer perceptron network, F represents the input feature map, and MaxPool represents max pooling. These are learnable parameters.
[0129] Step 3.3: Cross-scale feature fusion based on adaptive gating mechanism.
[0130] After attention weighting of features at each scale, the number of channels and size of all scale feature maps are aligned by a 1*1 convolution operation, and then an adaptive gating mechanism is introduced to perform cross-scale feature fusion.
[0131] Assume the input feature map is ,in R is the real number field, B is the batch size, C is the number of channels, H and W are the spatial dimensions of the features, and the gating generation function is:
[0132] ;
[0133] In the formula, The weight vector generated for each feature map, This is the normalization function, used here to ensure that the weights are normalized across the scale; This is the learnable weight matrix in the gating mechanism; The feature transformation operator is used to extract global context information from the feature map; finally, the features at important scales are enhanced by weighted summation to obtain the final feature vector.
[0134] Step 3.4: Cross-scale feature interaction and global average pooling processing.
[0135] In order to establish cross-scale dependencies in the feature space and enhance the model's global understanding of haze distribution, this step performs multi-branch convolution operations on the feature map after cross-scale feature fusion to obtain features of different dimensions, and then performs serialization processing on the features of different dimensions.
[0136] In this embodiment, the fused feature is convolved using convolution kernels of different sizes (3*3, 5*5, and 7*7) to obtain multi-scale receptive field features containing different local context ranges. Let... These are the features obtained through convolution with different kernels. Then, to extract global context information and adapt to the computation of the attention mechanism, these three dimensions of features are serialized and expanded into one-dimensional data:
[0137] ;
[0138] Where Vec is the vectorization operation and Seq is the serialized feature vector.
[0139] Step 3.5: Multi-head self-attention calculation and visibility feature output.
[0140] Multi-head self-attention is calculated on the serialized vector. The multi-head self-attention calculation is as follows:
[0141] ;
[0142] In the formula, The final output features of the multi-head self-attention module are... This represents a multi-head attention operation. The multi-head mechanism maps the input to different feature subspaces using multiple attention heads in parallel, where the i-th attention head... The calculation method is as follows:
[0143] ;
[0144] ;
[0145] Where Q, K, and V are the query, key, and value matrices, respectively, all derived from Seq; These are the learnable linear projection weight matrices corresponding to the i-th attention head; Indicates attention operation, This represents the dot product of the query matrix and the transpose of the key matrix, used to calculate the relevance score between positions within the feature sequence; The dimension of the key vector is used to prevent gradient vanishing. This is a normalization function, used here to transform the relevance score into an attention weight distribution.
[0146] Finally, all the attention heads obtained are concatenated and a linear transformation is performed to obtain the image visibility visual features.
[0147] Step 4: Extract features from meteorological factor data to obtain high-dimensional meteorological factor features.
[0148] Step 4.1: Construct an initial 5-dimensional meteorological feature vector from the meteorological factor data. Due to significant differences in the physical dimensions and numerical distribution ranges of various meteorological factors, to prevent gradient bias or dominance by specific large-value factors during backpropagation optimization, the initial 5-dimensional meteorological feature vector is Z-score standardized to obtain a standardized meteorological feature vector. This operation transforms it into standard distribution data with a mean of 0 and a variance of 1, effectively eliminating the negative impact of dimensional differences.
[0149] Step 4.2: Feed the standardized meteorological feature vectors into the cascaded feature coding network. For example... Figure 4 As shown, the standardized meteorological feature vector first passes through the first fully connected layer and the second fully connected layer in sequence. Through the nonlinear transformation relationship constructed by these two preceding fully connected layers, the discrete meteorological data located in the 5-dimensional low-order physical space is initially mapped and projected to the high-dimensional semantic feature space.
[0150] Step 4.3: Input the initially upgraded high-dimensional features into the self-attention layer. This self-attention layer aims to deeply mine and quantify the synergistic effects and intrinsic coupling relationships of various meteorological and physical factors in the formation of haze. By calculating the correlation of each element within the feature and adaptively allocating attention weights, the network can strengthen the feature responses of key factors that dominate haze changes under current meteorological conditions.
[0151] Step 4.4: The features after weighting and strengthening by the self-attention layer are fed into the third fully connected layer for linear mapping and dimensional adjustment to obtain high-dimensional meteorological factor features.
[0152] Step 5: Dynamically fuse the image visibility visual features with high-dimensional meteorological factor features, and then input the obtained cross-modal deep fusion features into the visibility prediction regression head to calculate the visibility prediction value.
[0153] This step delves into the high-order nonlinear correlation between meteorological and physical factors and nighttime low-light visual characteristics. Specifically, it uses three independent linear transformation layers to transform the high-dimensional meteorological factor features... 𝑚𝑒𝑡 Mapping to a query vector matrix, the image visibility visual feature is represented by 𝐹 𝑖𝑚𝑔 The mapping is represented by a key vector matrix and a value vector matrix, and the mapping formula is as follows:
[0154] ;
[0155] ;
[0156] ;
[0157] in, Characteristics of high-dimensional meteorological factors The query vector matrix obtained by mapping, Visual features for image visibility The key vector matrix obtained by mapping, Visual features for image visibility The value vector matrix obtained by mapping , d is the feature dimension, and R is the real number field; , , The weight matrix is a learnable matrix. and These represent the dimensions of the key vector and value vector in the attention mechanism, respectively.
[0158] by As a guide, dynamic query The visual feature region that best matches the current weather conditions is determined by calculation. and The dot product, divided by the scaling factor. To prevent gradient vanishing, then... The function is normalized to obtain the cross-modal attention weight distribution, and this weight distribution is then appended to the value vector matrix. The cross-modal attention features are calculated above. :
[0159] ;
[0160] This process enables the model to adaptively enhance or suppress specific visual feature responses based on real-time weather conditions.
[0161] To preserve the physical properties of the original meteorological features and accelerate network convergence, cross-modal attention features are incorporated. Characteristics of high-dimensional meteorological factors Residual connections are performed, followed by layer normalization, to obtain cross-modal deep fusion features. :
[0162] ;
[0163] in, This is the layer normalization function.
[0164] Cross-modal deep fusion features are input into a visibility prediction regression head, which is composed of a multilayer perceptron. The head performs nonlinear dimensionality reduction mapping through a fully connected layer to calculate the visibility prediction value. :
[0165] ;
[0166] in, This represents a multilayer perceptron network used for... Perform nonlinear feature extraction; and These are the weight matrix and bias term of the output layer, respectively.
[0167] Step 6: Warning of visibility in fog and haze under low light conditions at night.
[0168] like Figure 5 As shown, in step 1, the haze computing node acquires video data and haze-related meteorological factor data in low-light nighttime scenes from edge devices (such as cameras and automatic meteorological observation systems). The spatiotemporally aligned data obtained after processing is input into the DMFNet module. After processing in steps 2 to 5, the visibility prediction value is returned. The haze computing node transmits the visibility prediction value to the early warning module. The early warning module obtains the haze visibility decision based on the visibility prediction value and the preset hierarchical response mechanism and sends it back to the haze computing node. The haze computing node sends the decision log to the cloud platform for storage and sends real-time early warning information to various management platforms of the airport through the cloud platform to assist the airport's operational decisions.
[0169] Example 3
[0170] Based on the same inventive concept as Embodiment 1 or Embodiment 2, this embodiment provides a fog and haze visibility detection system under low light conditions at night, including a data acquisition module, a DMFNet module and an early warning module. The DMFNet includes an image preprocessing module, an image visibility visual feature extraction module, a meteorological factor feature extraction module, a feature fusion module and a visibility prediction module.
[0171] The data acquisition module is used to acquire video data and meteorological factor data related to haze in low-light nighttime scenes, and select the corresponding nighttime image data based on the timestamp of the meteorological factor data.
[0172] The image preprocessing module is used to decompose nighttime image data into a light effect layer and a background layer, and uses a light effect suppression network to locate and remove light effect regions to obtain a refined background image.
[0173] The image visibility visual feature extraction module is used to extract multi-scale features from the refined background image. It adopts a dual-path self-attention module to adaptively focus on key luminous targets, and obtains the image visibility visual features through cross-scale feature fusion, cross-scale feature interaction and multi-head self-attention calculation.
[0174] The meteorological factor feature extraction module is used to extract features from meteorological factor data to obtain high-dimensional meteorological factor features;
[0175] The feature fusion module is used to dynamically fuse image visibility visual features with high-dimensional meteorological factor features;
[0176] The visibility prediction module is used to input the obtained cross-modal deep fusion features into the visibility prediction regression head to calculate the visibility prediction value;
[0177] The early warning module is used to obtain fog and haze visibility decisions based on the visibility prediction value and the preset graded response mechanism, and transmit the fog and haze visibility decisions to the computing nodes, so that the computing nodes send the decision logs to the cloud platform for storage, and send real-time early warning information to various management platforms of the airport through the cloud platform.
[0178] The specific functions of each module described above are explained in the relevant content of the method in Embodiment 1, and will not be repeated here.
[0179] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0180] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0181] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0182] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0183] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.
Claims
1. A method for detecting visibility in haze under low-light conditions at night, characterized in that, include: Acquire video data of low-light nighttime scenes and meteorological factor data related to haze, and select corresponding nighttime image data based on the timestamp of the meteorological factor data; Nighttime image data is decomposed into a light effect layer and a background layer, and a light effect suppression network is used to locate and remove light effect regions to obtain a refined background image. Multi-scale feature extraction is performed on the refined background image. A dual-path self-attention module is used to adaptively focus on key regions. The image visibility visual features are obtained through cross-scale feature fusion, cross-scale feature interaction, and multi-head self-attention calculation. Feature extraction is performed on meteorological factor data to obtain high-dimensional meteorological factor features; The image visibility visual features are dynamically fused with high-dimensional meteorological factor features. Then, the resulting cross-modal deep fusion features are input into the visibility prediction regression head to calculate the visibility prediction value. The meteorological data related to haze include horizontal wind speed, vertical wind speed, relative humidity, temperature, and air pressure. The feature extraction of meteorological factor data includes: Meteorological factor data are used to construct an initial 5-dimensional meteorological feature vector. The initial 5-dimensional meteorological feature vector is then subjected to Z-score standardization to obtain the standardized meteorological feature vector. The standardized meteorological feature vectors are fed into a cascaded feature coding network. They pass through the first fully connected layer and the second fully connected layer in sequence, and the discrete meteorological data located in the 5-dimensional low-order physical space are initially mapped and projected to the high-dimensional semantic feature space. The high-dimensional features that have undergone initial dimensionality enhancement are input into the attention layer, and attention weights are adaptively allocated by calculating the correlation between elements within the features. The features after weighting and strengthening by the self-attention layer are fed into the third fully connected layer for linear mapping and dimension adjustment to obtain high-dimensional meteorological factor features. The process of dynamically fusing image visibility visual features with high-dimensional meteorological factor features, and then inputting the resulting cross-modal deep fusion features into a visibility prediction regression head to calculate the visibility prediction value includes: High-dimensional meteorological factor characteristics are transformed through three independent linear transformation layers. Mapping to a query vector matrix, representing the visual features of image visibility. The mapping is represented by a key vector matrix and a value vector matrix, and the mapping formula is as follows: ； ； ； in, Characteristics of high-dimensional meteorological factors The query vector matrix obtained by mapping, Visual features for image visibility The key vector matrix obtained by mapping, Visual features for image visibility The value vector matrix obtained by mapping , , Let be a learnable weight matrix, and d be the feature dimension. and These are the dimensions of the key vector and value vector in the attention mechanism, respectively. For the real number field; by As a guide, dynamic query Cross-modal attention features are calculated by selecting the visual feature region that best matches the current weather conditions. : ； in, Represents the normalization function; Cross-modal attention features Characteristics of high-dimensional meteorological factors Residual connections are performed, followed by layer normalization, to obtain cross-modal deep fusion features. : ； in, For layer normalization function; The cross-modal deep fusion features are input into the visibility prediction regression head to calculate the visibility prediction value. : ； in, This represents a multilayer perceptron network used for... Perform nonlinear feature extraction; and These are the weight matrix and bias term of the output layer, respectively.
2. The method for detecting visibility in haze under low-light conditions at night according to claim 1, characterized in that, The process of decomposing nighttime image data into a light effect layer and a background layer includes: Assume the image model is: ； Where I represents the nighttime image, H represents the reflectivity layer, L represents the shadow layer, and G represents the light effect layer. For element-wise multiplication; The decomposition target is the background layer without the light effect layer G. : = ； Separation of the light effect layer and the background layer is achieved by minimizing the overlap between them in the gradient space, as shown in the following formula: ； in, Represents the gradient exclusion function. and These represent the optical effect layer G and the background layer, respectively, using bilinear interpolation downsampling. , and They are respectively and Gradient mapping at the nth scale, and As the normalization factor, The function is used to perform saturation mapping on gradient magnitude. For element-wise multiplication, This represents the Frobenius norm, used to globally constrain the common response of the light effect layer and the background layer in the gradient domain.
3. The method for detecting visibility in haze under low-light conditions at night according to claim 1, characterized in that, The method of locating and removing the optical effect region using an optical effect suppression network includes: The optical effect domain feature tensor and the non-optical effect reference domain feature tensor are input into the optical effect suppression network. The encoder at the front end of the generator performs feature extraction and optical effect layer guidance to obtain the basic feature map and intermediate layer feature map. A classifier is used to classify the intermediate layer feature maps. First, the response weights of each feature channel for the light effect category are learned through global average pooling. Then, the activation weight map of the light effect interference region is generated in the spatial dimension by using the class activation map (CAM). Finally, the basic feature map output by the encoder is multiplied element-wise with the CAM weights to obtain the attention feature map. Employing an attention loss function based on domain adversarial mechanism The classifier is driven to accurately capture the light effect region and provide feedback to guide the generator. Its formula is defined as follows: ； In the formula, E represents the probability distribution of the auxiliary classifier's output. For mathematical expectation, For the characteristic tensor of the optical effect domain, For the feature tensor of the reference domain without optical effect; The image is reconstructed using the feature matrix weighted by the attention feature map through the decoder, eliminating the luminous effect components and filling in the background details to obtain a refined background image with the luminous effect layer removed. .
4. The method for detecting visibility in haze under low-light conditions at night according to claim 1, characterized in that, The process of extracting multi-scale features from the refined background image, employing a dual-path self-attention module to adaptively focus on key regions, includes: The refined background image is input into the lightweight EfficientNet-B3 basic feature extractor. Downsampling is performed at the beginning of each stage of EfficientNet-B3 to select representative feature maps at different scales. Feature maps of different scales are input into a dual-path self-attention module to adaptively focus on regions that significantly affect visibility; The dual-path self-attention module consists of two parallel branches: spatial attention and channel attention. and channel attention The formula is as follows: ； ； Where Conv represents convolution, Concat represents connection, AvgPool represents average pooling, MLP represents a multilayer perceptron network, F represents the input feature map, and MaxPool represents max pooling. These are learnable parameters.
5. The method for detecting visibility in haze under low-light conditions at night according to claim 1, characterized in that, The image visibility visual features are obtained through cross-scale feature fusion, cross-scale feature interaction, and multi-head self-attention computation, including: After attention weighting of features at each scale, the number of channels and size of all scale feature maps are aligned by a 1*1 convolution operation, and then an adaptive gating mechanism is introduced to perform cross-scale feature fusion. Multi-branch convolution operations are performed on the feature map after cross-scale feature fusion to obtain features of different dimensions, and the features of different dimensions are then serialized. Multi-head self-attention calculation is performed on the serialized vector, and all attention heads are concatenated and linear transformation is performed to obtain the image visibility visual features.
6. The method for detecting visibility in haze under low-light conditions at night according to claim 1, characterized in that, The method further includes: Based on the visibility prediction value and the preset graded response mechanism, the visibility decision for haze is obtained; The decision on visibility in haze is transmitted to the computing node, which then sends the decision log to the cloud platform for storage and sends real-time early warning information to various management platforms at the airport through the cloud platform.
7. A system for detecting visibility in fog and haze under low-light conditions at night, characterized in that, The system for implementing the method according to any one of claims 1 to 6 comprises: The data acquisition module is used to acquire video data and meteorological factor data related to haze in low-light nighttime scenes, and select the corresponding nighttime image data based on the timestamp of the meteorological factor data. The image preprocessing module is used to decompose nighttime image data into a light effect layer and a background layer, and uses a light effect suppression network to locate and remove light effect regions to obtain a refined background image. The image visibility visual feature extraction module is used to extract multi-scale features from the refined background image. It adopts a dual-path self-attention module to adaptively focus on key luminous targets, and obtains the image visibility visual features through cross-scale feature fusion, cross-scale feature interaction and multi-head self-attention calculation. The meteorological factor feature extraction module is used to extract features from meteorological factor data to obtain high-dimensional meteorological factor features; The feature fusion module is used to dynamically fuse image visibility visual features with high-dimensional meteorological factor features; The visibility prediction module is used to input the obtained cross-modal deep fusion features into the visibility prediction regression head to calculate the visibility prediction value.