Image defogging method and system based on multistage fusion and parallel filtering

Abstract

The invention discloses an image defogging method and system based on multi-level fusion and parallel filtering, and belongs to the field of image processing.The method comprises the steps that firstly, a foggy image is input, and useful low-level and high-level features are extracted through the multi-level feature fusion technology; then, designing a parallel filter to process the blurred image for feature extraction, and ensuring that texture details are recovered while capturing a large-area blurred region; besides, an enhanced distributed attention unit is constructed, uneven haze distribution is effectively processed, and extracted features are adaptively fused through learned spatial weight features. And finally, inputting the fused features into a defogging model to generate a clear output image. The invention further provides a system for realizing the method, evaluation shows that the method obviously improves the defogging effect under the condition that only 6.92 M parameters are used, and the method has a good application prospect.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to an image defogging method and system based on multi-level fusion and parallel filtering. Background Art

[0002] In natural scenes, due to the presence of suspended particles in the atmosphere (such as fog, smoke, etc.), images are often affected by haze, resulting in a decrease in visual quality, manifested as reduced contrast, blurred details and color distortion. This phenomenon can seriously affect the performance of computer vision systems, especially in the fields of unmanned driving, video surveillance and remote sensing image analysis. Therefore, image dehazing technology has become an important research direction in the field of image enhancement.

[0003] Traditional image defogging methods are mainly based on the physical model of the image, and restore the image by estimating the transmittance of the fog and the atmospheric light. However, traditional methods often require certain assumptions about the prior information of the scene, and the defogging effect in complex scenes is insufficient. In recent years, defogging methods based on deep learning have made significant progress. By using convolutional neural networks to automatically learn the feature information required for defogging, the limitations of traditional methods can be overcome, especially when dealing with complex and non-uniform haze scenes.

[0004] Most of the existing deep learning dehazing methods focus on improving the model's capabilities by increasing the depth and width of the network, but this is often accompanied by a sharp increase in computational overhead and model parameters. In addition, the existing methods are still insufficient in processing large areas of haze and restoring details. Summary of the invention

[0005] The purpose of the present invention is to overcome the technical problems existing in the prior art. The present invention proposes an improved defogging scheme based on multi-scale detail enhancement and parallel attention mechanism. By combining multi-scale convolution, large convolution kernel module and attention mechanism, it can not only capture a large range of blurred areas, but also effectively restore the detail information of the image, thereby significantly improving the defogging effect.

[0006] The objective of the present invention is achieved through the following technical solutions:

[0007] In a first aspect, a method for image defogging based on multi-level fusion and parallel filtering is provided, comprising the following steps:

[0008] S1. Input a foggy image and preprocess the foggy image;

[0009] S2, extracting low-level features and high-level features of the preprocessed image using a multi-level feature fusion module;

[0010] S3, convolving the features extracted in step S2 using a massively parallel filter;

[0011] S4, processing the convolved features through a distributed attention unit, wherein channel attention and spatial attention are used to process the uneven haze distribution to obtain adaptively weighted features;

[0012] S5. Input the features obtained in step S4 into the decoder of the defogging model and output a clear image without fog.

[0013] In some embodiments, step S2 specifically includes:

[0014] The multi-level feature fusion module extracts multi-scale features of the image through multiple parallel convolutional structures and enhances the local details of the image through residual connections.

[0015] In some embodiments, step S3 specifically includes:

[0016] The massively parallel filter first performs channel expansion on the input features through convolution kernels of different sizes and expansion rates;

[0017] Then, a convolution based on reflection filling is used to calculate the weight of each pixel to improve the sensitivity to the local pixel area.

[0018] In some embodiments, step S4 specifically includes:

[0019] Calculate the importance of each pixel position of the input feature;

[0020] Generate the weight of each channel through global average pooling and nonlinear transformation;

[0021] The spatial and channel features are fused to further enhance the information of specific areas of the image.

[0022] In some embodiments, step S5 specifically includes:

[0023] The dehazing model is an improved U-Net neural network, which restores high-resolution clear images through deconvolution and feature fusion.

[0024] In a second aspect, an image defogging system based on multi-level fusion and parallel filtering is provided, comprising:

[0025] Image input module, used to input foggy images;

[0026] An image preprocessing module, used for preprocessing the foggy image;

[0027] A multi-level feature fusion module is used to extract low-level features and high-level features of the preprocessed image using the multi-level feature fusion module;

[0028] Massively parallel filters for convolution of features extracted from multi-level feature fusion modules;

[0029] The distributed attention unit is used to process the convolved features, where channel attention, spatial attention and pixel attention are used to process the uneven haze distribution to obtain adaptively weighted features.

[0030] The dehazing decoder module is used to process the features obtained by the distributed attention unit through the decoder of the dehazing model;

[0031] The output module is used to output a clear image without fog.

[0032] In some embodiments, the multi-level feature fusion module adopts a detail enhancement block (DetailEnhancement Block), which includes multiple parallel convolution structures, specifically including ordinary convolution (VC), central difference convolution (CDC), angular divergence convolution (ADC), etc., to capture feature information of different scales and enhance local details of the image through residual connection.

[0033] In some embodiments, the massively parallel filter includes convolutions of different sizes and expansion rates. The convolution operation is performed on the input image to expand the receptive field and capture feature information of large blurred areas while maintaining image details. This is to achieve multi-level feature fusion and ensure that feature information at different levels is effectively captured.

[0034] In some embodiments, the distributed attention unit specifically includes the following submodules:

[0035] A spatial attention module that computes the importance of each pixel position in the input image;

[0036] The channel attention module generates the weight of each channel through global average pooling and nonlinear transformation;

[0037] The pixel attention module is used to fuse spatial and channel features to further enhance the information of specific areas of the image. Each submodule is used to calculate the importance of different features to cope with the unevenness of haze distribution.

[0038] In some embodiments, the defogging model includes:

[0039] Normalization layer;

[0040] The first convolutional layer has the following parameters: 32 kernels, 3×3 kernel size, and 1×1 stride.

[0041] The second convolutional layer has the following parameters: 64 kernels, 3×3 kernel size, and 2×2 stride length.

[0042] The number of convolutional layer kernels is 128, the size is 3×3, and the stride is 2×2;

[0043] Multi-scale detail enhancement block, parameter configuration is: convolution kernel 3×3, step size 1×1;

[0044] The first deconvolution layer, upsampling layer, has the following parameters: 64 kernels, 3×3 kernel size, and 2×2 stride length;

[0045] The second deconvolution layer is an upsampling layer with the following parameters: 32 kernels, 3×3 kernel size, and 2×2 stride length.

[0046] The third convolutional layer has the following parameters: 3 kernels, 3×3 kernel size, and 1×1 stride.

[0047] Feature fusion module, parameter configuration is: convolution kernel 3×3, step size 1×1.

[0048] Specifically, the improved U-Net neural network uses different types of convolution kernels for parallel processing in the designed MSDEB (multi-level feature fusion module) to expand the receptive field; uses GELU as the activation function instead of the Relu activation function to improve the nonlinear expression ability of the model; replaces the feature attention part in the FAB module with the channel attention mechanism of dual-channel fusion; adopts a dilated convolution layer with a 3×3 kernel and the dilation value is set to 2 to expand the receptive field for multi-stream feature extraction.

[0049] It should be further explained that the technical features corresponding to the above embodiments can be combined or replaced with each other to form a new technical solution without conflict.

[0050] Compared with the prior art, the present invention has the following beneficial effects:

[0051] (1) This method uses multi-level feature fusion technology to extract useful low-level and high-level features. Next, a parallel filter is designed to process the blurred image for feature extraction, ensuring that texture details are restored while capturing large blurred areas. In addition, an enhanced distributed attention unit is constructed to effectively handle the uneven haze distribution, and the extracted features are adaptively fused through the learned spatial weight features. Finally, the fused features are input into the dehazing model to generate a clear output image. By combining multi-level feature fusion, large-scale parallel filters, and distributed attention units, the present invention provides an efficient, robust, and complex scene-suitable image dehazing method that has significant advantages in restoring image details and improving visual quality.

[0052] . After experimental verification, the method of the present invention has shown superior defogging effects on multiple public defogging datasets, and has lower model parameter volume and computational overhead. Evaluation shows that this method significantly improves the defogging effect with only 6.92M parameters, and has good application prospects.

[0053] (2) The system of the present invention includes an input module, multi-level feature fusion, a large-scale parallel filter, a defogging decoder module and an output module, which can realize automatic processing from input image to defogging result, and is suitable for unmanned driving, video surveillance, remote sensing imaging and other fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 A flow chart of an image defogging method based on multi-level fusion and parallel filtering shown in an embodiment of the present invention;

[0055] Figure 2 This is a general network framework diagram of an embodiment of the present invention;

[0056] Figure 3 A multi-level feature fusion module (MSDEB) shown in an embodiment of the present invention;

[0057] Figure 4 A schematic diagram of the structure of a distributed attention block (PAB) network according to an embodiment of the present invention;

[0058] Figure 5 A schematic diagram of the structure of a feature fusion (GCA) network shown in an embodiment of the present invention. DETAILED DESCRIPTION

[0059] The technical solution of the present invention is clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. The components of the embodiments of the present application described and shown in the accompanying drawings can be arranged and designed in various configurations. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0060] It should be noted that the defects existing in the solutions in the above-mentioned prior art are the results obtained by the inventor after practice and careful research. Therefore, the discovery process of the above-mentioned problems and the solutions proposed in the embodiments of the present application for the above-mentioned problems below should all be the contributions made by the inventor to the present application in the process of invention and creation, and should not be understood as technical contents known to technical personnel in this field.

[0061] In view of the technical problems pointed out in the background technology, the embodiments provided by the present invention are as follows:

[0062] The image dehazing method of the present invention is implemented based on the deep learning framework PyTorch. The experimental environment is an NVIDIA RTX 3080 GPU, and the model is trained and tested using dehazing datasets such as RESIDE and Haze4K. The Adam optimizer is used in the training process, and the initial learning rate is set to 0.0001, and the cosine annealing strategy is used to gradually decay to 0.000001. After 30 training cycles, the performance of the model on the public dataset is significantly better than traditional and existing deep learning methods.

[0063] Example 1

[0064] An image defogging system network based on multi-level fusion and parallel filtering is provided, comprising:

[0065] Image input module, used to input foggy images;

[0066] The image preprocessing module is used to preprocess the foggy image, including normalizing the input foggy image, adjusting the size and other preprocessing operations.

[0067] A multi-level feature fusion module is used to extract low-level features and high-level features of the preprocessed image using the multi-level feature fusion module;

[0068] Large-scale parallel filters are used to convolve the features extracted in the multi-level feature fusion module. Specifically, convolution kernels of different sizes are used to extract multi-scale features, expand the receptive field, capture large blurred areas and retain details.

[0069] Distributed attention unit, used to process the convolutional features, where channel attention, spatial attention and pixel attention are used to process the uneven haze distribution, weight the features, enhance the model's response to different regions, and obtain adaptively weighted features;

[0070] The dehazing decoder module is used to process the features obtained by the distributed attention unit through the decoder of the dehazing model; the processed features are gradually restored to the original resolution through the upsampling layer to generate a clear dehazing image.

[0071] The output module is used to output a clear image without fog.

[0072] Example 2

[0073] Based on the above system, an image defogging method based on multi-level fusion and parallel filtering is provided. Figure 1 , including the following steps:

[0074] S1. Input a foggy image and preprocess the foggy image;

[0075] S2, extracting low-level features and high-level features of the preprocessed image using a multi-level feature fusion module;

[0076] S3, convolving the features extracted in step S2 using a massively parallel filter;

[0077] S4, processing the convolved features through a distributed attention unit, wherein channel attention and spatial attention are used to process the uneven haze distribution to obtain adaptively weighted features;

[0078] S5. Input the features obtained in step S4 into the decoder of the defogging model and output a clear image without fog.

[0079] Specifically, step S1 includes:

[0080] Step S11, image size adjustment: ensuring input consistency;

[0081] Step S12, random cropping: obtain random cropping parameters for cropping to increase data diversity;

[0082] Step S13, random rotation: image conversion according to angle parameters (0 degrees, 90 degrees, 180 degrees, 270 degrees)

[0083] Step S14, convert to tensor: convert to tensor through ToTensor;

[0084] Step S15, image normalization: normalize the pixel values ​​to It can speed up the convergence of the model and improve its stability.

[0085] Furthermore, in the multi-level feature fusion module in step S2, the multi-stream features are fused together through the Concat operation, such as Figure 3 shown.

[0086] The massively parallel filter has both multi-scale characteristics and a large receptive field. First, for the feature map processed by detail enhancement convolution, multi-scale features can be extracted through three parallel dilated convolutions with different kernel sizes.

[0087]

[0088] In the formula, It represents a 7×7 depth-wise dilated convolution with a kernel size of 19 and a dilation rate of 3. It represents a 5×5 depth-wise dilated convolution with a kernel size of 13 and a dilation rate of 3. It represents a 3×3 depth-wise dilated convolution with a dilated convolution kernel size of 7 and a dilation rate of 3. The large and medium dilated convolutions have long-distance modeling capabilities and a large receptive field of self-attention in Transformer, which can focus on larger blurred areas, while the small dilated convolution can focus on smaller blurred areas, and the value of restoring texture details becomes three times.

[0089]

[0090] In the formula, Represents point-by-point convolution, and feeds the features into a multilayer perceptron, which converts the feature dimension of the image to the same as . The multilayer perceptron contains two point-by-point convolutions and uses GELU as the activation function. Finally, the output of the multilayer perceptron is added to the initial input. The multilayer perceptron can not only combine three different types of features, but also play a certain fitting role in the dehazing features.

[0091] The feature attention module (FAM) consists of a channel attention and a spatial attention, which are placed in sequence to calculate the attention weights in the channel and spatial dimensions. Channel attention calculates a channel vector, i.e., to recalibrate the features. Spatial attention calculates a spatial importance map (SIM), i.e., to adaptively represent the importance of different regions. FAM improves the dehazing performance by performing unequal treatment on different channels and pixels.

[0092] In the distributed attention unit, a multi-level feature fusion part is embedded and channel and pixel attention are processed in parallel, such as Figure 4 In the feature extraction part, the size of The standard convolutional layer of size is A standard convolutional layer of size The expansion rate is set to 2. These three convolutional layers are deployed in parallel to extract images with multiple receptive fields (i.e. , and ). Let represent the input feature map and extract the multi-scale features.

[0093]

[0094] In the formula, It means using a dilation rate of 2. After computing the multi-scale features, we concatenate them channel by channel and then use The convolutional layer reduces the number of channels from 3c to c. Using local residual learning and Convolutional layers are used to compute the output.

[0095]

[0096] In the attention part, different types of attention mechanisms are mixed, including a channel attention module and a pixel attention module. We use global average pooling (GAP), PWConv-GELU-PWConv and Sigmoid function to extract global channel gated features. PWConv represents point-wise convolution and Conv represents convolution with kernel size = 1.

[0097]

[0098] PWConv-GELU-PWConv is used to fit features, Sigmoid function is used to extract global pixel gating features, and Conv represents convolution with kernel size = 3.

[0099]

[0100] The two different attention results are concatenated along the channel dimension, and then an MLP with PWConv-GELU-PWConv is applied to reduce the concatenated feature channel dimension to the same dimension as the input, and finally the output of the MLP is added to the input.

[0101]

[0102] In addition, another problem of FAM is that there is no information exchange between the two attention weights. Therefore, we use a fusion scheme to fully mix the channel attention weights and spatial attention weights, such as Figure 4 As shown, information interaction is guaranteed.

[0103]

[0104] In the formula represents the sigmoid operation, Represents a channel shuffle operation, Indicates size The information encoded in the shallow features of the kernel's convolutional layer is very different from that encoded in the deep features. The fusion scheme adaptively fuses the low-level features of the encoder part and the corresponding high-level features through the learned spatial weight modulation features.

[0105] Furthermore, if Figure 5As shown in Figure 1, the low-level features of the encoder part and the corresponding high-level features are input into PAB to calculate the weights, and then combined using the weighted summation method. We also increase the input features through skip connections to alleviate the gradient vanishing problem and simplify the learning process. Finally, the fused features are The projection of the convolutional layer obtains the final features.

[0106]

[0107] In this embodiment, the defogging model is an improved DEA-Net neural network, and its structure is as follows: Figure 2 As shown, it includes:

[0108] Normalization layer;

[0109] The first convolutional layer has the following parameters: 32 kernels, 3×3 kernel size, and 1×1 stride.

[0110] The second convolutional layer has the following parameters: 64 kernels, 3×3 kernel size, and 2×2 stride length.

[0111] The number of convolutional layer kernels is 128, the size is 3×3, and the stride is 2×2;

[0112] Multi-scale detail enhancement block, parameter configuration is: convolution kernel 3×3, step size 1×1;

[0113] The first deconvolution layer, upsampling layer, has the following parameters: 64 kernels, 3×3 kernel size, and 2×2 stride length;

[0114] The second deconvolution layer is an upsampling layer with the following parameters: 32 kernels, 3×3 kernel size, and 2×2 stride length.

[0115] The third convolutional layer has the following parameters: 3 kernels, 3×3 kernel size, and 1×1 stride.

[0116] Feature fusion module, parameter configuration is: convolution kernel 3×3, step size 1×1.

[0117] In addition, this embodiment improves the image defogging performance by optimizing multiple modules in the improved DEA-Net neural network, as follows:

[0118] Depthwise separable convolution: The multi-level feature fusion module uses depthwise separable convolution, combined with common convolution (VC), center difference convolution (CDC) and angle difference convolution (ADC), to extract multi-scale features, and combines residual connection to enhance the expression ability of features of different scales.

[0119] Distributed attention unit: Through the parallel attention mechanism module (including spatial attention module, channel attention module, pixel attention module), it replaces the simple convolution method in the classic convolution module, effectively captures the feature information of different regions, and enhances the refinement of feature extraction. It is suitable for complex lighting conditions and uneven fog distribution in image defogging tasks.

[0120] Multi-layer fusion feature mapping: Combining the multi-scale detail enhancement module with the large-scale parallel filter, the convolution kernels of different sizes and dilation rates are used to capture local and global features. At the same time, multi-level feature mapping is performed through the feature fusion module, combining a large receptive field with rich details of local information.

[0121] Through the above improvements, the U-Net neural network constructs an efficient network architecture suitable for image defogging tasks, and effectively realizes the extraction, fusion and enhancement of fog-free features between encoding and decoding. Experimental verification shows that the network significantly improves the visual quality of defogging images and has good application prospects in application scenarios such as unmanned driving and video surveillance.

[0122] The model of the method of the present invention performs significantly better than traditional and existing deep learning methods on public data sets. In the experiment, seven existing models were selected for comparison, including DCP, NLD, DehazeNet, AOD-Net, CAP, PFF-Net and GCANet. The quality of the dehazed image is evaluated using peak signal-to-noise ratio PSNR, structural similarity SSIM, network parameter quantity Param and calculation accuracy FLOPs. Among them, the larger the values ​​of peak signal-to-noise ratio PSNR and structural similarity SSIM, the better, and the smaller the values ​​of network parameter quantity Param and calculation accuracy FLOPs, the better the result. The experimental results show that the indicators of the model of the present invention are better than other methods in both indoor and outdoor scenes. Compared with the second best model, the average peak signal-to-noise ratio PSNR of the model of the present invention in outdoor scenes is increased by 3.01, and the peak signal-to-noise ratio PSNR in indoor scenes is increased by 4.57.

[0123] The above specific implementation methods are detailed descriptions of the present invention. It cannot be determined that the specific implementation methods of the present invention are limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions and substitutions can be made without departing from the concept of the present invention, which should be regarded as belonging to the protection scope of the present invention.

Claims

1. An image defogging method based on multi-level fusion and parallel filtering, characterized in that: The following steps are involved: S1. Input a foggy image and preprocess the foggy image; S2, extracting low-level features and high-level features of the preprocessed image using a multi-level feature fusion module; S3, convolving the features extracted in step S2 using a massively parallel filter; S4, processing the convolved features through a distributed attention unit, wherein channel attention, spatial attention, and pixel attention are used to process the uneven haze distribution to obtain adaptively weighted features; S5. Input the features obtained in step S4 into the decoder of the defogging model and output a clear image without fog.

2. The image defogging method according to claim 1, characterized in that: The step S2 specifically includes: The multi-level feature fusion module extracts multi-scale features of the image through multiple parallel convolutional structures and enhances the local details of the image through residual connections.

3. The image defogging method according to claim 1, characterized in that: The step S3 specifically includes: The massively parallel filter first performs channel expansion on the input features through convolution kernels of different sizes and expansion rates; Then, a convolution based on reflection filling is used to calculate the weight of each pixel to improve the sensitivity to the local pixel area.

4. The image defogging method according to claim 1, characterized in that: The step S4 specifically includes: Calculate the importance of each pixel position of the input feature; Generate the weight of each channel through global average pooling and nonlinear transformation; The spatial and channel features are fused to further enhance the information of specific areas of the image.

5. The image defogging method according to claim 1, characterized in that: The step S5 specifically includes: The dehazing model is an improved U-Net neural network, which restores high-resolution clear images through deconvolution and feature fusion.

6. An image defogging system based on multi-level fusion and parallel filtering, characterized in that: include: Image input module, used to input foggy images; An image preprocessing module, used for preprocessing the foggy image; A multi-level feature fusion module is used to extract low-level features and high-level features of the preprocessed image using the multi-level feature fusion module; Massively parallel filters for convolution of features extracted from multi-level feature fusion modules; The distributed attention unit is used to process the convolved features, where channel attention, spatial attention and pixel attention are used to process the uneven haze distribution to obtain adaptively weighted features. The dehazing decoder module is used to process the features obtained by the distributed attention unit through the decoder of the dehazing model; The output module is used to output a clear image without fog.

7. The image defogging system according to claim 6, characterized in that: The multi-level feature fusion module includes multiple parallel convolution structures.

8. The image defogging system according to claim 6, characterized in that: The massively parallel filters contain convolutions of different sizes and dilation rates.

9. The image defogging system according to claim 6, characterized in that: The distributed attention unit specifically includes the following submodules: A spatial attention module that computes the importance of each pixel position in the input image; The channel attention module generates the weight of each channel through global average pooling and nonlinear transformation; Pixel attention module is used to fuse spatial and channel features to further enhance the information of specific areas of the image.

10. The image defogging system according to claim 6, characterized in that: The defogging model includes: Normalization layer; The first convolutional layer has the following parameters: 32 kernels, 3×3 kernel size, and 1×1 stride. The second convolutional layer has the following parameters: 64 kernels, 3×3 kernel size, and 2×2 stride length. The number of convolutional layer kernels is 128, the size is 3×3, and the stride is 2×2; Multi-scale detail enhancement block, parameter configuration is: convolution kernel 3×3, step size 1×1; The first deconvolution layer, upsampling layer, has the following parameters: 64 kernels, 3×3 kernel size, and 2×2 stride length; The second deconvolution layer is an upsampling layer with the following parameters: 32 kernels, 3×3 kernel size, and 2×2 stride length. The third convolutional layer has the following parameters: 3 kernels, 3×3 kernel size, and 1×1 stride. Feature fusion module, parameter configuration is: convolution kernel 3×3, step size 1×1.

Images