Weak supervision defogging method based on uncertainty driving

Abstract

The invention discloses a weak supervision defogging method based on uncertainty driving, and the method employs a weak supervision defogging network of a U-Net architecture, and the weak supervision defogging network comprises a regional dynamic attention module RDAM, a lower sampling layer, a multi-scale feature fusion module MSFFM, and an upper sampling layer. The RDAM is used for processing an input image or the output of a previous layer of RDAM, constructing a dynamic convolutional layer and obtaining a regional dynamic attention feature map through the dynamic convolutional layer; the down-sampling layer is used for down-sampling the output of the RDAM and reducing the spatial resolution; the MSFFM is used for processing input features through a window multi-head self-attention mechanism so as to capture a long-distance dependency relationship between pixels and performing multi-scale feature extraction and fusion; the upper sampling layer is used for recovering the feature map output by the MSFFM to the resolution ratio of the feature map output by the MSFFM of the upper layer and performing feature splicing with the feature map output by the MSFFM of the upper layer; and enabling the up-sampled feature map to sequentially pass through a 3 * 3 convolutional layer, a ReLU activation function and a 1 * 1 convolutional layer, and finally outputting a defogged image.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and in particular relates to an image defogging method, in particular a weakly supervised defogging network based on deep learning. Background Art

[0002] Images taken under hazy weather conditions are often affected by uneven brightness, reduced contrast, color distortion, and blurred details, which greatly hinders the analysis and understanding of these images by vision-based industrial systems. Therefore, dehazing, as a necessary step for downstream tasks such as object detection, has become an important research direction in vision-based industrial systems. [1] . Current dehazing methods are mainly divided into prior-based dehazing methods and deep learning-based dehazing methods. The former relies on the atmospheric scattering model and identifies the relationship between them by analyzing a large number of foggy and fog-free images. However, these methods often fail in scenes that do not conform to the prior information, resulting in significant color distortion and vignetting effects. The latter estimates the atmospheric scattering model parameters or directly maps foggy images to fog-free images by training neural networks using synthetic datasets. Although the effect is significant, the generalization performance in real scenes is poor.

[0003] Weakly supervised dehazing methods have gradually become one of the research directions to solve this problem. [2,3,4] The model is trained on loosely paired foggy and non-fog images, but these methods are complex to train and have poor model stability. In addition, some teacher-student networks based on convolutional neural networks (CNNs) [5,6,7] Structures are used for dehazing, but due to the limitation of their local receptive field, they have difficulty capturing long-range dependencies. Recently, weakly supervised dehazing methods based on Transformers have been proposed. [8] Although they perform well in capturing global dependencies, they often ignore local details. Summary of the invention

[0004] The purpose of the present invention is to overcome the shortcomings of the prior art, solve the effectiveness and uncertainty problems of existing defogging methods in the real world, and provide a weakly supervised defogging method driven by uncertainty. By integrating a regional dynamic attention module (RDAM) and a multi-scale feature fusion module (MSFFM) based on a window multi-head self-attention mechanism (W-MSA), the present invention aims to improve the defogging effect in non-uniform defogging tasks, especially when processing real scenes, it can better preserve the local details and global consistency of the image.

[0005] The objective of the present invention is achieved through the following technical solutions:

[0006] A weakly supervised defogging method based on uncertainty driving includes a weakly supervised defogging network using a U-Net architecture, wherein the weakly supervised defogging network consists of a regional dynamic attention module RDAM, a downsampling layer, a multi-scale feature fusion module MSFFM, an upsampling layer, a 3x3 convolution layer, a Relu activation function and a 1x1 convolution layer; wherein:

[0007] The regional dynamic attention module RDAM is used to process the input image and the output of the previous layer of RDAM; and construct a dynamic convolution layer using spatial attention features; the regional dynamic attention module RDAM uses two parallel convolution paths to calculate the channel attention and pixel attention respectively, and then uses learnable parameters to weight the two attentions to obtain a channel-pixel collaborative attention feature map, and then obtains a regional dynamic attention feature map through a dynamic convolution layer;

[0008] The downsampling layer is used to downsample the output of the regional dynamic attention module RDAM in the previous layer to reduce the spatial resolution and obtain the input of the regional dynamic attention module RDAM in the next layer;

[0009] The multi-scale feature fusion module MSFFM is used to process the features input to the multi-scale feature fusion module MSFFM through the window multi-head self-attention mechanism to capture the long-distance dependencies between pixels and obtain the window multi-head self-attention features; the multi-scale feature fusion module MSFFM performs deconvolution and average pooling operations on the window multi-head self-attention feature map to achieve information extraction and fusion at several levels; the module also introduces parallel dilated convolution to enhance the fusion effect of local and global information;

[0010] The upsampling layer is used to restore the feature map output by the multi-scale feature fusion module MSFFM to the resolution of the feature map output by the multi-scale feature fusion module MSFFM of the previous layer, and perform feature splicing with the feature map output by the multi-scale feature fusion module MSFFM of the previous layer; finally, a 3x3 convolution layer, a Relu activation function and a 1x1 convolution layer are used to output the dehazed image.

[0011] Furthermore, in the regional dynamic attention module RDAM, channel attention and pixel attention are calculated by the following formulas respectively:

[0012] f c =f in ·σ(Conv(δ(Conv(AvgPool(f in ))))));

[0013] f in represents the input feature map of the regional dynamic attention module, f crepresents the channel attention feature; AvgPool(·) represents the average pooling operation; Conv(·) is the convolution layer; δ(·) is the ReLU activation function; σ(·) is the Sigmoid activation function, which is used to limit the weight to the range of (0,1);

[0014] By inputting the feature map f in Average pooling is performed, and then two convolutional layers and nonlinear activation are calculated;

[0015] f p =f in ·σ(Conv(δ(Conv(f in ))));

[0016] f p Represents the pixel attention feature, directly input feature map f in It is obtained by performing two convolutional layers and activation function calculations; then the channel attention features and pixel attention features are adaptively fused using the learnable parameter σ(θ):

[0017] f mix =σ(θ)·f c +(1-σ(θ))·f p ;

[0018] σ(θ) is a learnable parameter obtained through the Sigmoid function, with a value range of (0,1); f mix Represents the channel-pixel collaborative attention feature map, which represents the weighted sum of channel attention and pixel attention, and is used for subsequent dynamic convolution;

[0019] In order to achieve regional adaptive convolution, the regional dynamic attention module RDAM is equipped with a dynamic convolution layer based on spatial attention;

[0020] First, calculate the spatial attention feature f s :

[0021] f s =f in ·σ(Conv(concat(AvgPool(f in ),MaxPool(f in ))));

[0022] Enhance the spatial feature expression of the input feature map through spatial attention and calculate the regional weight W i :

[0023] W i =Softmax(Linear(AvgPool(f s ))),i=1,2,3,...,k;

[0024] where f s represents the spatial attention feature; AvgPool(·) represents the average pooling operation; concat(·) represents the feature concatenation operation; MaxPool(·) represents the maximum pooling operation, Linear(·) represents the fully connected layer; Softmax(·) is the Softmax activation function; k is the number of regions; then the weight W is applied to the k convolution kernels i Generate a set of region-adaptive dynamic convolution kernels conv i , the specific calculation process is:

[0025] conv i =W i conv0,i=1,2,3,...,k;

[0026] Among them, conv0 is the initial convolution kernel; the obtained dynamic convolution kernel is used to construct the dynamic convolution layer DyConv();

[0027] Finally, the channel-pixel collaborative attention feature f mix After the dynamic convolution layer DyConv(), the regional dynamic attention feature f is obtained out for:

[0028] f out =DyConv(f mix ).

[0029] Furthermore, the specific workflow in the multi-scale feature fusion module MSFFM is as follows:

[0030] The multi-scale feature fusion module MSFFM processes the input x through a window multi-head self-attention mechanism in To learn the long-distance dependencies between image pixels, we can obtain the window multi-head self-attention feature x W-MSA , and then perform deconvolution and average pooling operations to obtain 2 times up-sampled features x2 and 1 / 2 times down-sampled features x of different scales. 1 / 2 , downsample feature x by 1 / 4 1 / 4 Finally, refinement and fusion are performed to obtain the multi-scale fusion feature Y out ; The specific operations are as follows:

[0031] x W-MSA =W-MSA(x in )

[0032] x in is the feature input to the multi-scale feature fusion module MSFFM; W-MSA(·) is the window multi-head self-attention mechanism operation; then the window multi-head self-attention feature x W-MSARefine through a separable shared convolution layer and a dilated convolution layer to obtain the original scale refined feature Y;

[0033] In parallel, multi-head self-attention features of the window x W-MSA Perform deconvolution and average pooling operations:

[0034] x2=Deconv(x W-MSA )

[0035] Deconv(·) is a deconvolution operation used to convert x W-MSA The resolution is doubled to obtain a 2x upsampled feature x2; then it is refined through a convolution layer, a separable shared convolution layer, and a dilated convolution layer to obtain a 2x upsampled scale refined feature Y2;

[0036] x 1 / 2 =AvgPool2×2(x W-MSA )

[0037] AvgPool2×2(·) is a 2x2 average pooling operation used to convert x W-MSA The resolution is reduced by half to obtain a 1 / 2 times downsampled feature x 1 / 2 , and then refined through a transposed convolution layer, a separable shared convolution layer and a dilated convolution layer to obtain a 1 / 2 times downsampling scale refined feature Y 1 / 2 ;

[0038] x 1 / 4 =AvgPool4×4(x W-MSA )

[0039] AvgPool4×4(·) is a 4x4 average pooling operation, which obtains 1 / 4 times the downsampled feature x 1 / 4 , and then refined through two transposed convolutional layers, a separable shared convolutional layer and an expanded convolutional layer to obtain a 1 / 4 times down-sampled scale refined feature Y 1 / 4 ;

[0040] Then, the above multi-scale refined features are concatenated, and after a convolution layer and a Relu activation function, the multi-scale fusion feature Y is output. out , the calculation process is:

[0041] Y out =Relu(Conv(concat(Y 1 / 4 , Y 1 / 2 , Y, Y2)))

[0042] Through these operations, the multi-scale feature fusion module MSFFM combines the global and local information of the input image and enhances the feature expression capability of the dehazing task.

[0043] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the uncertainty-driven weakly supervised defogging method when executing the program.

[0044] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the uncertainty-driven weakly supervised defogging method are implemented.

[0045] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0046] 1. Efficient dehazing effect: The present invention adopts a weakly supervised dehazing network with a U-Net architecture, combined with downsampling and upsampling modules, to extract and restore multi-scale features of the image, thereby providing an efficient dehazing effect in complex scenes. The network can effectively remove haze from images while maintaining image details and clarity. Through quantitative comparison, the present invention achieved the highest PSNR (30.32) and SSIM (0.972) among weakly supervised methods on the HSTS test set, indicating that it has superior performance in image dehazing tasks. And the present invention exhibits stronger robustness when dealing with complex real scenes.

[0047] 2. Regional dynamic adaptive weight adjustment: The regional dynamic attention module (RDAM) of the present invention can dynamically adjust the weights of different channels and pixels, focusing on areas with higher haze concentrations in the image. This channel-pixel collaborative attention mechanism ensures that the network can adaptively handle the distribution of haze in complex scenes, so that key areas in the image can be better restored. By adaptively adjusting the weights of different channels and pixels, effective dehazing is performed for different areas in the image, avoiding residual haze, color shift and other phenomena in other methods, making the final dehazed image more realistic and delicate.

[0048] 3. Capturing long-distance dependencies: By introducing a multi-scale feature fusion module (MSFFM) based on a windowed multi-head self-attention mechanism (W-MSA), the present invention can capture long-distance dependencies between different image blocks and fully combine the local and global information of the image, thereby more effectively improving the dehazing effect, especially when processing larger scenes, it can better maintain overall consistency.

[0049] 4. Enhanced detail recovery capability: The multi-scale feature fusion module (MSFFM) effectively fuses local and global features through parallel dilated convolutions, ensuring that the network not only retains the global structural information of the image during image recovery, but also refines local details, improves the quality of the dehazed image, and avoids detail blur and color distortion.

[0050] 5. Good generalization performance in real scenes: The present invention adopts a weakly supervised learning method combined with an uncertainty-driven strategy, so that the network can still have good generalization ability in real scenes without a large amount of labeled data, and adapt to different haze environments and complex lighting conditions.

[0051] 6. Good visual effect: Qualitative comparison shows that the present invention can generate dehazed images with high visual quality in the RTTS test set of real scenes. Compared with other weakly supervised methods, the present invention can effectively remove haze, avoid problems such as oversaturation and local distortion, while retaining edge details and contour information in the image, making the dehazed image more natural and clear.

[0052] 7. Improve the generalization ability of the model: Compared with other methods that rely on generative adversarial networks (GANs), this invention significantly improves the generalization ability of the network through uncertainty-driven weakly supervised learning. Without using real haze images for training, it can still maintain good defogging performance in real scenes, overcome the problem of domain shift, and adapt to the non-uniform haze distribution in different scenes.

[0053] 8. Enhanced training stability: Compared with the problems of unstable training and poor convergence of GAN models such as ART-SS-DA, the network architecture design and uncertainty driving mechanism adopted in the present invention make the training process of the model more stable, with faster convergence speed, improve training efficiency and reduce the complexity of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 Schematic diagram of the uncertainty-driven weakly supervised dehazing network structure;

[0055] Figure 2 This is a schematic diagram of the structure of the regional dynamic attention module RDAM;

[0056] Figure 3 It is a structural diagram of the multi-scale feature fusion module MSFFM;

[0057] Figure 4 A comparison of the visual effects of different dehazing methods on the RTTS test set. DETAILED DESCRIPTION

[0058] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0059] This embodiment proposes an uncertainty-driven weakly supervised dehazing method based on a weakly supervised dehazing network (Uncertainty-Driven Weakly Supervised Dehazing Network). The network consists of a regional dynamic attention module (RDAM), a downsampling layer, a multi-scale feature fusion module (MSFFM), an upsampling layer, a 3x3 convolution layer, a Relu activation function, and a 1x1 convolution layer.

[0060] Regional Dynamic Attention Module (RDAM): Unlike traditional methods that only focus on channel or spatial weight distribution, RDAM dynamically adjusts the weights on different channels and pixels through the Channel-Pixel Collaborative Attention (CP-CA) mechanism. This mechanism combines channel attention and pixel attention to dynamically adjust weights and improve the network's perception of key channels and regions. Specifically, channel attention uses average pooling operations to calculate the importance of each channel, while pixel attention calculates the importance of each pixel through convolution operations. The fusion of the two is used to weight the input features, allowing the network to focus on areas with higher fog concentrations.

[0061] Multi-scale Feature Fusion Module (MSFFM): MSFFM uses W-MSA to learn the long-range dependencies between image patches, and further fuses and refines them through parallel dilated convolutions, thereby effectively supplementing global semantics with local details in the dehazing task. The W-MSA module calculates the global dependencies in the input feature map through a multi-head self-attention mechanism, and then achieves the fusion of multi-scale features through deconvolution and average pooling operations. Finally, parallel dilated convolution operations are used to enhance the fusion effect of multi-scale features.

[0062] like Figure 1 As shown in the figure, the dehazing network structure of this embodiment is similar to the U-Net[9] architecture, which is composed of a regional dynamic attention module (RDAM), a downsampling layer, a multi-scale feature fusion module (MSFFM), an upsampling layer, a 3x3 convolution layer, a Relu activation function and a 1x1 convolution layer. The specific process is as follows:

[0063] Input and multi-layer processing of regional dynamic attention module RDAM: The input synthetic haze image or real-world haze image is first processed by 3 layers of regional dynamic attention module (RDAM). The output of the previous layer of RDAM passes through the downsampling module and serves as the input of the next layer of RDAM.

[0064] Multi-layer processing of the multi-scale feature fusion module MSFFM: The output features of each layer of RDAM processing enter the corresponding layer of the multi-scale feature fusion module (MSFFM). MSFFM is responsible for multi-level fusion of features at different scales. The features input to the multi-scale feature fusion module MSFFM are processed by the W-MSA module to capture the long-distance dependencies between pixels. The multi-scale feature fusion module performs deconvolution and average pooling operations on the input features, integrating features at different scales to achieve multi-level information extraction and fusion. The multi-scale feature fusion module MSFFM also introduces parallel dilated convolution to enhance the fusion effect of local and global information.

[0065] Upsampling and output: The upsampling layer restores the output of the third-layer multi-scale feature fusion module MSFFM to the output resolution of the second-layer multi-scale feature fusion module MSFFM, and then performs feature splicing with the output of the second-layer multi-scale feature fusion module MSFFM. After a 3x3 convolution layer, a Relu activation function, and a 1x1 convolution layer, it is upsampled again and feature spliced ​​with the output of the first-layer multi-scale feature fusion module MSFFM. After a 3x3 convolution layer, a Relu activation function, and a 1x1 convolution layer, the dehazed image is output.

[0066] Specifically, about the Regional Dynamic Attention Module (RDAM):

[0067] like Figure 2 As shown in Figure 2, RDAM designs a channel-pixel collaborative attention mechanism (CP-CA) to dynamically assign weights to different channels and pixels based on fog distribution characteristics. Channel attention and pixel attention are calculated using the following formulas:

[0068] f c =f in ·σ(Conv(δ(Conv(AvgPool(f in ))))))

[0069] f in represents the input feature map of the regional dynamic attention module; f c represents the channel attention feature map; AvgPool(·) represents the average pooling operation; Conv(·) is the convolution layer; δ(·) is the ReLU activation function; σ(·) is the Sigmoid activation function, which is used to limit the weight to the range of (0,1);

[0070] By inputting the feature map f of the regional dynamic attention module in Average pooling is performed, and then two convolutional layers and nonlinear activation are calculated;

[0071] f p =f in ·σ(Conv(δ(Conv(f in ))))

[0072] f p Represents the pixel attention feature map, directly input feature map f in It is obtained by performing two convolutional layers and activation function calculations;

[0073] Then, the channel attention feature map and the pixel attention feature map are adaptively fused using the learnable parameter σ(θ):

[0074] f mix =σ(θ)·f c +(1-σ(θ))·f p

[0075] σ(θ) is a learnable parameter obtained through the Sigmoid function, and its value range is (0,1). mix Represents the channel-pixel collaborative attention feature map, which represents the weighted sum of channel attention and pixel attention, and is used for subsequent dynamic convolution.

[0076] To achieve region-adaptive convolution, RDAM designs a dynamic convolution layer based on spatial attention.

[0077] First, calculate the spatial attention feature f s :

[0078] f s =f in ·σ(Conv(concat(AvgPool(f in ),MaxPool(f in ))))

[0079] Enhance the spatial feature expression of the input feature map through spatial attention and calculate the regional weight W i :

[0080] W i =Softmax(Linear(AvgPool(f s ))),i=1,2,3,...,k

[0081] where f srepresents the spatial attention feature; AvgPool(·) represents the average pooling operation; concat(·) represents the feature concatenation operation; MaxPool(·) represents the maximum pooling operation, Linear(·) represents the fully connected layer; Softmax(·) is the Softmax activation function; k is the number of regions; then, weight W is applied to k convolution kernels. i Generate a set of region-adaptive dynamic convolution kernels conv i , the specific calculation process is:

[0082] conv i =W i conv0,i=1,2,3,...,k

[0083] Where conv0 is the initial convolution kernel. The obtained dynamic convolution kernel is used to construct the dynamic convolution layer DyConv().

[0084] Finally, the channel-pixel collaborative attention feature f mix After the dynamic convolution layer DyConv(), the regional dynamic attention feature f is obtained out for:

[0085] f out =DyConv(f mix )

[0086] Specifically, about the multi-scale feature fusion module (MSFFM) based on W-MSA:

[0087] like Figure 3 As shown, MSFFM processes the input x of the multi-scale feature fusion module through W-MSA in To learn the long-distance dependencies between image pixels, we can obtain the window multi-head self-attention feature x W-MSA , and then perform deconvolution and average pooling operations to obtain 2 times up-sampled features x2 and 1 / 2 times down-sampled features x of different scales. 1 / 2 , downsample feature x by 1 / 4 1 / 4 Finally, refinement and fusion are performed to obtain the multi-scale fusion feature Y out The specific operations are as follows:

[0088] x W-MSA =W-MSA(x in )

[0089] x in is the feature input to the multi-scale feature fusion module MSFFM; W-MSA(·) is the window multi-head self-attention mechanism operation; then the window multi-head self-attention feature x W-MSA Refine through a separable shared convolution layer and a dilated convolution layer to obtain the original scale refined feature Y;

[0090] In parallel, multi-head self-attention features of the window x W-MSA Perform deconvolution and average pooling operations:

[0091] x2=Deconv(x W-MSA )

[0092] Deconv(·) is a deconvolution operation used to convert x W-MSA The resolution is doubled to obtain a 2x upsampled feature x2; then it is refined through a convolution layer, a separable shared convolution layer, and a dilated convolution layer to obtain a 2x upsampled scale refined feature Y2;

[0093] x 1 / 2 =AvgPool2×2(x W-MSA )

[0094] AvgPool2×2(·) is a 2x2 average pooling operation used to convert x W-MSA The resolution is reduced by half to obtain a 1 / 2 times downsampled feature x 1 / 2 , and then refined through a transposed convolution layer, a separable shared convolution layer and a dilated convolution layer to obtain a 1 / 2 times downsampling scale refined feature Y 1 / 2 ;

[0095] x 1 / 4 =AvgPool4×4(x W-MSA )

[0096] AvgPool4×4(·) is a 4x4 average pooling operation, which obtains 1 / 4 times the downsampled feature x 1 / 4 , and then refined through two transposed convolutional layers, a separable shared convolutional layer and an expanded convolutional layer to obtain a 1 / 4 times downsampling scale refined feature Y 1 / 4 ;

[0097] Then, the above multi-scale refined features are concatenated, passed through a convolution layer and a Relu activation function, and the multi-scale fusion feature Y is output. out The calculation process is:

[0098] Y out =Relu(Conv(concat(Y 1 / 4 , Y 1 / 2 , Y, Y2)))

[0099] Through these operations, MSFFM effectively combines the global and local information of the input image and provides stronger feature expression capabilities for the dehazing task.

[0100] Specifically, about the setting of loss function:

[0101] This embodiment designs an unsupervised loss function based on heteroscedastic uncertainty. The loss function includes contrast loss, edge preservation loss and saturation loss, which enables the network to adaptively correct contrast, enhance edges and improve saturation according to regional uncertainty during the dehazing process. In addition, for labeled data, the mean square error loss function is used to measure the error between the predicted result and the true value.

[0102] (1) Heteroscedastic Uncertainty-Based Loss Function

[0103] Heteroscedastic uncertainty loss function assumes that in the foggy image dataset, samples {x1,x2,…,x n} are independent and follow a Gaussian distribution, and each sample has a different variance, expressed as:

[0104]

[0105] Among them, μ i and Respectively represent the i-th sample x i The mean and variance of , i = 1, 2…n. It is generally believed that the larger the variance, the higher the uncertainty of the pixels and features of the image.

[0106] x i : represents the i-th sample of the foggy image. μ i : is the mean of the i-th sample. is the variance of the i-th sample, indicating the uncertainty of pixels and features.

[0107] In order to enhance the visual quality of the dehazed image, a contrast loss L is used. contrast , edge holding loss L edge and saturation loss L sat Adaptively correct contrast, enhance edges and improve saturation. This embodiment calculates the heteroscedastic uncertainty loss L uncertainty The process is as follows:

[0108]

[0109] in:

[0110] L contrast =C J -C I

[0111]

[0112] L sat =SJ -S I

[0113]

[0114] Where logσ i represents the logarithm of the variance, J(i,j) and I(i,j) are the pixel values ​​of the dehazed image and the hazy image at position (i,j), respectively. and are the mean of the dehazed image and the hazy image, respectively. H and W are the height and width of the image, respectively. Canny(·) represents the Canny edge detection operator. γ is a learnable parameter obtained by the Sigmoid activation function, ranging from (0,1).

[0115] (2) Mean Squared Error Loss Function

[0116] The mean square error loss function is used for labeled data and calculates the average of the squared differences between the dehazed image and the haze-free image:

[0117]

[0118] J(i,j) represents the pixel value of the dehazed image at position (i,j). G(i,j) represents the pixel value of the haze-free image at position (i,j). H and W are the height and width of the image, respectively.

[0119] (3) Total Loss Function

[0120] The total loss function combines the above two loss functions in a linear weighted summation:

[0121] L total =λ uncertainty L uncertainty +λ mse L mse

[0122] λ uncertainty and λ mse are the hyperparameters of their respective loss functions. According to experience, when these coefficients are set to λ uncertainty = 0.5 and λ mse When =0.72, the dehazing result is the best.

[0123] Specifically, the dataset and training settings are:

[0124] In the Linux environment, the Pytorch framework was used to build a weakly supervised dehazing network and train it. The NVIDIA GeForce RTX 3090 GPU was used during the training process. During the training process, the Adam optimizer was used to adjust the model parameters, and the momentum decay exponent was set to β1 = 0.9 and β2 = 0.999. The input image size was set to 128 × 128 and the batch size was 16.

[0125] To facilitate training and evaluation, this example uses a public haze dataset. Specifically, from Haze4K [5] 3000 synthetic haze images were selected from the dataset as labeled training sets, and

[10] 3000 real haze images are selected from the real-world task-driven test set (RTTS) in the dataset as the unlabeled training set. For testing, this embodiment uses the synthetic objective test set (SOTS) and the mixed subjective test set (HSTS) in RESIDE. SOTS includes 500 outdoor and 500 indoor synthetic haze images, while HSTS includes 10 outdoor synthetic haze images. In addition, we also added HazeRD

[11] Synthetic haze dataset, which contains 15 clear images and 60 haze images generated under different visibility conditions (1000m, 500m, 200m and 100m). In order to evaluate the performance of the algorithm in real scenes, 500 real haze images were selected from RTTS as the test set to ensure that these images were not involved in the training process.

[0126] In order to verify the defogging effect of the present invention, this embodiment adopts subjective and objective evaluation methods and compares them with mainstream methods. These methods are divided into three categories: (1) Defogging methods based on prior knowledge: DCP

[12] (2010) and FVR

[13] (2015); (2) Supervised learning dehazing method: DehazeNet

[14] (2016), AOD-Net

[15] (2017), EPDN

[16] (2019), SwinIR

[17] (2021), Dehamer

[18] (2022) and Restormer

[19] (2023); (3) Weakly supervised dehazing method: RefineDNet [2] (2021), PSD

[20] (2021), DMT [5] (2021), ART-SS-SSIM

[21] (2022), ART-SS-DA

[21] (2022), Semi-UFormer

[22] (2023), SDA-GAN

[23] (2023) and Semi-SGDRL

[24] (2024).

[0127] 1. Quantitative Comparison

[0128] Table 1 shows that the proposed method obtains the highest PSNR and SSIM values ​​among the weakly supervised methods on the HSTS test set, which are PSNR30.32 and SSIM0.972 respectively. Although it lags slightly behind ART-SS-DA on the HazeRD test set, with a PSNR of 19.50 for ART-SS-DA and 18.88 for our method; on the SOTS test set, the PSNR of ART-SS-DA is 29.15, while our method is 29.23, it is worth noting that ART-SS-DA only enhances the generalization ability of the model by introducing real haze images, without considering the uncertainty factors in the image degradation process, which leads to poor visual quality of the dehazed image. In addition, the core network of ART-SS-DA adopts a generative adversarial network, which has problems of unstable training and poor convergence. On the other hand, supervised approaches Restormer and DeHamer achieve leading metrics on the SOTS dataset, with Restormer reaching PSNR37.81 and SSIM0.990, while DeHamer achieves PSNR35.18 and SSIM0.986. This significant performance can be attributed to the fact that the SOTS dataset is a synthetic dataset with more uniform scenes, allowing Transformer-based models to better capture and generalize the characteristics of the data.

[0129] Table 1: Quantitative comparison results of the present invention and mainstream technologies on SOTS, HSTS and HazeRD datasets.

[0130]

[0131]

[0132] 2. Qualitative Comparison

[0133] In order to verify the defogging effect of the present invention in real scenes, this embodiment compares it with the mainstream weakly supervised defogging method on the RTTS test set of real scenes. Figure 4(a)-(g) show the dehazed images generated using various weakly supervised methods. Since PSD relies too much on prior information, it has poor image restoration performance in real scenes with insufficient prior information, such as Figure 4 (b) shows oversaturated colors. Similarly, although RefineDNet alleviates the domain shift problem to some extent through weakly supervised training, it fails to fully consider the non-uniform distribution of haze in real scenes, resulting in over-dehazing problems, such as Figure 4 The ground area in (c) is too dark. In addition, DMT has designed subnetworks for estimating atmospheric light value and transmission rate separately, which leads to error accumulation and local distortion in the dehazed image, such as Figure 4 However, ART-SS-DA and SDA-GAN fail to achieve adaptive weight distribution between channels and pixels, resulting in different degrees of distortion in the dehazed image, including Figure 4 (e) and Figure 4 (f) shows the residual haze and color shift. Figure 4 (g) Demonstrates that the method of the present invention can effectively remove haze while retaining rich edge and contour details, producing excellent visual effects.

[0134] Preferably, the embodiment of the present application also provides a specific implementation of an electronic device capable of implementing all steps in the uncertainty-driven weakly supervised defogging method in the above embodiment, and the electronic device specifically includes the following contents:

[0135] Processor, memory, communications interface and bus;

[0136] Among them, the processor, memory, and communication interface communicate with each other through the bus; the communication interface is used to realize information transmission between related devices such as server-side devices, metering devices, and user-side devices.

[0137] The processor is used to call the computer program in the memory, and when the processor executes the computer program, all the steps in the uncertainty-driven weakly supervised defogging method in the above embodiment are implemented.

[0138] An embodiment of the present application also provides a computer-readable storage medium capable of implementing all the steps of the uncertainty-driven weakly supervised defogging method in the above embodiment. A computer program is stored on the computer-readable storage medium. When the computer program is executed by a processor, all the steps of the uncertainty-driven weakly supervised defogging method in the above embodiment are implemented.

[0139] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the hardware + program embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0140] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0141] Although the present application provides method operation steps such as embodiments or flow charts, more or fewer operation steps may be included based on conventional or non-creative labor. The order of steps listed in the embodiments is only one way of executing the order of many steps and does not represent the only execution order. When the actual device or client product is executed, it can be executed in the order of the method shown in the embodiments or the drawings or in parallel (for example, in a parallel processor or multi-threaded processing environment).

[0142] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0143] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0144] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0145] The present invention is not limited to the embodiments described above. The above description of the specific embodiments is intended to describe and illustrate the technical solution of the present invention. The above specific embodiments are merely illustrative and not restrictive. Without departing from the scope of the present invention and the scope of protection of the claims, a person of ordinary skill in the art can also make many forms of specific changes under the guidance of the present invention, which all fall within the scope of protection of the present invention.

[0146] References:

[0147] [1]Singh M,Laxmi V,Faruki P.Visibility enhancement and dehazing:Research contribution challenges and direction[J].Computer Science Review,2022,44:100473.

[0148] [2] Zhao S, Zhang L, Shen Y, et al. Refinednet: A weakly supervisedrefinement framework for single image dehazing[J]. IEEE Transactions on ImageProcessing, 2021, 30: 3391-3404.

[0149] [3]Dong Y, Li Y, Dong Q, et al. Semi-supervised domain alignment learning for single image dehazing[J]. IEEE Transactions on Cybernetics, 2022.

[0150] [4]Zhang S, Zhang

[0151] [5]Liu Y,Zhu L,Pei S,et al.From synthetic to real:Image dehazingcollaborating with unlabeled real data[C] / / Proceedings of the 29th ACMinternational conference on multimedia.ACM,2021:50-58.

[0152] [6]Li L,Dong Y,Ren W,et al.Semi-supervised image dehazing[J].IEEETransactions on Image Processing,2019,29:2766-2779.

[0153] [7]Yu H,Huang J,Liu Y,et al.Source-free domain adaptation for real-world image dehazing[C] / / Proceedings of the 30th ACM International Conferenceon Multimedia.ACM,2022:6645-6654.

[0154] [8]Song Y,He Z,Qian H,et al.Vision transformers for single imagedehazing[J].IEEE Transactions on Image Processing,2023,32:1927-1941.

[0155] [9]Ronneberger O,Fischer P,Brox T.U-net:Convolutional networks forbiomedical image segmentation[C] / / Medical Image Computing and Computer-Assisted Intervention–MICCAI.Springer,2015:234-241.

[0156]

[10] Li B,Ren W,Fu D,et al.Benchmarking single-image dehazing andbeyond[J].IEEE Transactions on Image Processing,2018,28:492-505.

[0157]

[11] Zhang Y,Ding L,Sharma G.Hazerd:an outdoor scene dataset andbenchmark for single image dehazing[C] / / 2017 IEEE International Conference onImage Processing(ICIP).IEEE,2017:3205-3209.

[0158]

[12] He K,Sun J,Tang X.Single image haze removal using dark channelprior[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2010,33(12):2341-2353.

[0159]

[13] Tarel J,Hautiere N.Fast visibility restoration from a singlecolor or gray level image[C] / / 2009IEEE 12th International Conference onComputer Vision.IEEE,2009:2201-2208.

[0160]

[14] Cai B,Xu X,Jia K,et al.Dehazenet:An end-to-end system for singleimage haze removal[J].IEEE Transactions on Image Processing,2016,25(11):5187-5198.

[0161]

[15] Li B,Peng X,Wang Z,et al.Aod-net:All-in-one dehazing network[C] / / Proceedings of the IEEE international conference on computer vision.IEEE,2017:4770-4778.

[0162]

[16] Qu Y,Chen Y,Huang J,et al.Enhanced pix2pix dehazing network[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and PatternRecognition.IEEE,2019:8160-8168.

[0163]

[17] Liang J,Cao J,Sun G,et al.Swinir:Image restoration using swintransformer[C] / / Proceedings of the IEEE / CVF International Conference onComputer Vision,2021:1833-1844.

[0164]

[18] Guo C L,Yan Q,Anwar S,et al.Image dehazing transformer withtransmission-aware 3d position embedding[C] / / Proceedings of the IEEE / CVFconference on computer vision and pattern recognition,2022:5812-5820.

[0165]

[19] Zamir S W,Arora A,Khan S,et al.Restormer:Efficient transformerfor high-resolution image restoration[C] / / Proceedings of the IEEE / CVFconference on computer vision and pattern recognition,2022:5728-5739.

[0166]

[20] Chen Z,Wang Y,Yang Y,et al.Psd:Principled synthetic-to-realdehazing guided by physical priors[C] / / Proceedings of the IEEE / CVF Conferenceon Computer Vision and Pattern Recognition.IEEE,2021:7180-7189.

[0167]

[21] Yasarla R,Priebe C,Patel V.Art-ss:An adaptive rejection techniquefor semi-supervised restoration for adverse weather-affected images[C] / / European Conference on Computer Vision.Springer,2022:699-718.

[0168]

[22] Tong M,Wang Y,Cui P,et al.Semi-uformer:Semi-superviseduncertainty-aware transformer for image dehazing[J / OL].arXiv preprint arXiv:2210.16057,2022.

[0169]

[23] Dong Y,Li Y,Dong Q,et al.Semi-supervised domain alignmentlearning for single image dehazing[J].IEEE Transactions on Cybernetics,2022.

[0170]

[24] Jia T,Li J,Zhuo L,et al.Self-guided disentangled representationlearning for single image dehazing[J].Neural Networks,2024,172:106107.

Claims

1. A weakly supervised dehazing method based on uncertainty driving, characterized in that: It includes a weakly supervised defogging network using the U-Net architecture. The weakly supervised defogging network consists of a regional dynamic attention module RDAM, a downsampling layer, a multi-scale feature fusion module MSFFM, an upsampling layer, a 3x3 convolution layer, a Relu activation function, and a 1x1 convolution layer; among which: The regional dynamic attention module RDAM is used to process the input image and the output of the previous layer of RDAM; and construct a dynamic convolution layer using spatial attention features; the regional dynamic attention module RDAM uses two parallel convolution paths to calculate the channel attention and pixel attention respectively, and then uses learnable parameters to weight the two attentions to obtain a channel-pixel collaborative attention feature map, and then obtains a regional dynamic attention feature map through a dynamic convolution layer; The downsampling layer is used to downsample the output of the regional dynamic attention module RDAM in the previous layer to reduce the spatial resolution and obtain the input of the regional dynamic attention module RDAM in the next layer; The multi-scale feature fusion module MSFFM is used to process the features input to the multi-scale feature fusion module MSFFM through the window multi-head self-attention mechanism to capture the long-distance dependencies between pixels and obtain the window multi-head self-attention features; the multi-scale feature fusion module MSFFM performs deconvolution and average pooling operations on the window multi-head self-attention feature map to achieve information extraction and fusion at several levels; the module also introduces parallel dilated convolution to enhance the fusion effect of local and global information; The upsampling layer is used to restore the feature map output by the multi-scale feature fusion module MSFFM to the resolution of the feature map output by the multi-scale feature fusion module MSFFM of the previous layer, and perform feature splicing with the feature map output by the multi-scale feature fusion module MSFFM of the previous layer; finally, a 3x3 convolution layer, a Relu activation function and a 1x1 convolution layer are used to output the dehazed image.

2. According to claim 1, a weakly supervised defogging method based on uncertainty driving is characterized in that: In the regional dynamic attention module RDAM, channel attention and pixel attention are calculated by the following formulas respectively: fc=fi n ·σ(Conv(δ(Conv(AvgPool(fi n )))))); fi n represents the input feature map of the regional dynamic attention module, fc represents the channel attention feature; AvgPool(·) represents the average pooling operation; Conv(·) is the convolution layer; δ(·) is the ReLU activation function; σ(·) is the Sigmoid activation function, which is used to limit the weight to the range of (0, 1); By inputting the feature map fi n Average pooling is performed, and then two convolutional layers and nonlinear activation are calculated; fp=fi n ·σ(Conv(δ(Conv(fi n )))); fp represents the pixel attention feature, directly input feature map fi n It is obtained by performing two convolutional layers and activation function calculations; Then, the channel attention features and pixel attention features are adaptively fused using the learnable parameter σ(θ): FMI x =σ(θ)·fc+(1-σ(θ))·fp; σ(θ) is a learnable parameter obtained through the Sigmoid function, with a value range of (0, 1); fmi x Represents the channel-pixel collaborative attention feature map, which represents the weighted sum of channel attention and pixel attention, and is used for subsequent dynamic convolution; In order to achieve regional adaptive convolution, the regional dynamic attention module RDAM is equipped with a dynamic convolution layer based on spatial attention; First, calculate the spatial attention feature f s : f s =f in ·σ(Conv(concat(AvgPool(f in ),MaxPool(f in )))); Enhance the spatial feature expression of the input feature map through spatial attention and calculate the regional weight W i : W i =Softmax(Linear(AvgPool(f s )),i=1,2,3,...,k; where f s represents the spatial attention feature; AvgPool(·) represents the average pooling operation; concat(·) represents the feature concatenation operation; MaxPool(·) represents the maximum pooling operation, Linear(·) represents the fully connected layer; Softmax(·) is the Softmax activation function; k is the number of regions; then the weight Wi is applied to the k convolution kernels to generate a set of regional adaptive dynamic convolution kernels conv i , the specific calculation process is: conv i =W i conv0,i=1,2,3,...,k; Among them, conv0 is the initial convolution kernel; the obtained dynamic convolution kernel is used to construct the dynamic convolution layer DyConv(); Finally, the channel-pixel collaborative attention feature f mix After the dynamic convolution layer DyConv(), the regional dynamic attention feature f is obtained out for: f out =DyConv(f mix )。 3. According to the uncertainty-driven weakly supervised defogging method of claim 1, it is characterized in that: The specific workflow in the multi-scale feature fusion module MSFFM is as follows: The multi-scale feature fusion module MSFFM processes the input x through a window multi-head self-attention mechanism in To learn the long-distance dependencies between image pixels, we can obtain the window multi-head self-attention feature x W-MSA , and then perform deconvolution and average pooling operations to obtain 2 times up-sampled features x2 and 1 / 2 times down-sampled features x of different scales. 1 / 2 , downsample feature x by 1 / 4 1 / 4 Finally, refinement and fusion are performed to obtain the multi-scale fusion feature Y out ; The specific operations are as follows: X W-MSA =W-MSA(x in ) x in is the feature input to the multi-scale feature fusion module MSFFM; W-MSA(·) is the window multi-head self-attention mechanism operation; then the window multi-head self-attention feature x W-MSA Refine through a separable shared convolution layer and a dilated convolution layer to obtain the original scale refined feature Y; In parallel, multi-head self-attention features of the window x W-MSA Perform deconvolution and average pooling operations: x2=Deconv(x W-MSA ) Deconv(·) is a deconvolution operation used to convert x W-MSA The resolution is doubled to obtain a 2x upsampled feature x2; then it is refined through a convolution layer, a separable shared convolution layer, and a dilated convolution layer to obtain a 2x upsampled scale refined feature Y2; x 1 / 2 =AvgPool2×2(x w-MSA ) AvgPool2×2(·) is a 2x2 average pooling operation used to convert x W-MSA The resolution is reduced by half to obtain a 1 / 2 times downsampled feature x 1 / 2 , and then refined through a transposed convolution layer, a separable shared convolution layer and a dilated convolution layer to obtain a 1 / 2 times downsampling scale refined feature Y 1 / 2 ; x 1 / 4 =AvgPool4×4(x W-MSA ) AvgPool4×4(·) is a 4x4 average pooling operation, which obtains 1 / 4 times the downsampled feature x 1 / 4 , and then refined through two transposed convolutional layers, a separable shared convolutional layer and an expanded convolutional layer to obtain a 1 / 4 times down-sampled scale refined feature Y 1 / 4 ; Then, the above multi-scale refined features are concatenated, and after a convolution layer and a Relu activation function, the multi-scale fusion feature Y is output. out , the calculation process is: AND out =Relu(Conv(concat(And 1 / 4 ,AND 1 / 2 ,And,And2))) Through these operations, the multi-scale feature fusion module MSFFM combines the global and local information of the input image and enhances the feature expression capability of the dehazing task.

4. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the uncertainty-driven weakly supervised defogging method according to any one of claims 1 to 3 are implemented.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the uncertainty-driven weakly supervised defogging method according to any one of claims 1 to 6 are implemented.

Images