Dehazing method based on dual-branch residual feature fusion

Abstract

The present invention discloses a defogging method based on dual-branch residual feature fusion, which can directly learn the mapping relationship between foggy images and fog-free images, extract the residual features of uneven fog and achieve defogging. The context space domain attention structure and the channel attention autoencoder structure respectively focus on the pixel space and channel space of the feature map, adaptively fuse the feature information extracted by the two branches and output the final fog layer residual map, and introduce a discriminant network to make the defogging image closer to the real look and feel. Experiments show that the present invention can provide a defogging effect that is superior to existing methods, but the processing process is time-consuming. The efficiency of the network outputting the haze layer will be optimized in the future so that it can be applied to engineering practices such as autonomous driving.

Description

Technical Field

[0001] The invention belongs to the technical field of image restoration of computer vision, and in particular relates to a defogging method based on dual-branch residual feature fusion. Background Art

[0002] Images captured in bad weather will be disturbed by impurities such as haze, resulting in reduced image contrast and detail visibility, which affects the subsequent computer vision algorithm. Therefore, it is of great practical significance to restore the details of haze images. Traditional image dehazing methods are mainly divided into the following two categories:

[0003] 1) Based on image enhancement: Ignoring the theoretical reasons for image degradation, image features are enhanced to improve the color saturation and contrast of the image to achieve the purpose of dehazing. For example, wavelet transform, Retinex algorithm and histogram equalization.

[0004] 2) Image restoration: The current defogging methods based on physical models are as follows: Defogging is achieved by establishing a medium propagation image through the local statistical irrelevance between the surface chromaticity of the object and the medium propagation. The defogging efficiency is high, but the defogging effect is poor on highly blurred images; the method based on the dark channel prior gives priority to obtaining the atmospheric light value, and optimizes the transmittance map through guided filtering for image defogging. The defogging effect is good, but color distortion is prone to occur when processing large white areas; the median filter is used to estimate the medium transmittance to achieve defogging. The method is simple, but the restored image is prone to halo effect. The above defogging methods have certain scene limitations due to the limitations of prior content or artificial constraints.

[0005] At present, the defogging methods based on deep learning are as follows: the medium transmission estimation network based on convolutional neural network achieves end-to-end defogging by estimating the mapping relationship between the blurred image and its medium transmission; the defogging image is directly generated through a lightweight convolutional network without estimating the transmission rate and atmospheric light intensity; the residual defogging network based on the gentle convolution expansion group solves the image artifact problem caused by the expansion convolution block and deconvolution; the end-to-end feature fusion attention network is used to directly restore the fog-free image, which has excellent defogging effect on synthetic fog images, but poor defogging effect on real fog images; the fast deep multi-region superposition defogging network has high defogging efficiency for real foggy images, but the defogging performance is relatively ordinary. Since the fog datasets of some algorithms are synthetic fog images, the synthesis method is based on the atmospheric scattering physical model, which has significant differences in feature distribution from the real fog images in nature. Although the defogging effect is better than the traditional image enhancement method, the defogging effect on uneven fog and dense fog images is poor, and some areas are prone to residual fog. Summary of the invention

[0006] The purpose of the present invention is to provide a defogging method based on dual-branch residual feature fusion, which extracts the features of the foggy image through the attention mechanism of the spatial domain attention branch and the channel domain attention branch, and then fuses the two parts of the features to give higher weights to important features, thereby improving the ability to extract effective features and improving the defogging effect of the foggy image.

[0007] The technical solution adopted by the present invention is based on a double-branch residual feature fusion defogging method, which is specifically implemented according to the following steps:

[0008] Step 1: Take multiple scenes, collect original images of scenes without haze and artificial haze images of the same scene, form matching pairs of foggy and non-fog images, and select image pairs from open source datasets to form an image dataset;

[0009] Step 2: Divide the image data set into a training set and a test set in a ratio of 9:1, and perform data augmentation on the training set to obtain an augmented training set;

[0010] Step 3, input the image pairs in the augmented training set into the two-branch residual feature fusion network for training, and obtain an optimized two-branch residual feature fusion network;

[0011] Step 4: Input the artificial haze image in the test set into the optimized two-branch residual feature fusion network, perform feature fusion based on convolution blocks and residual structure, extract the residual features of the haze layer, and output the haze impurity layer;

[0012] Step 5: The artificially created haze image in the test set is jump-connected with the haze impurity layer to output a dehazed image, and the peak signal-to-noise ratio and structural similarity of the dehazed image are compared with the corresponding haze-free image in the test set, and the optimal two-branch residual feature fusion network is determined according to the comparison results;

[0013] Step 6: Input the image to be dehazed into the optimal two-branch residual feature fusion network and output the dehazed image.

[0014] The specific process of data augmentation for the training set in step 2 is: randomly cropping, horizontally flipping, and region blocking the training set image pairs to obtain an augmented training set.

[0015] The dual-branch residual feature fusion network includes a contextual spatial domain attention branch that performs pixel attention on the high-frequency fog area of ​​the foggy image and a channel domain attention encoding and decoding branch that pays attention to the channel direction of the high-frequency haze features. The spatial domain attention branch assigns weights to the contextual pixel space of the haze features by inserting the spatial domain attention module into the multi-scale dilated convolution group; the channel domain attention encoding and decoding branch assigns weights to different channel feature maps by setting the ResNet autoencoder structure and introducing the channel attention decoding structure. It also includes a feature fusion branch, which uses adaptive weights to fuse the fog layer feature information of pixel attention and channel attention, and outputs an uneven fog residual layer.

[0016] The spatial domain attention module is used to effectively focus on dense fog pixels, expressed as:

[0017] SA c =δ(f 7×7 [G Max (F c1 );G Ave (F c1 )] (5)

[0018]

[0019] In the formula, δ is the Sigmoid activation function, F C1 is the input feature map, f 7×7 is a 7×7 convolution, Multiply corresponding inputs element-wise.

[0020] The following spatial domain attention branch assigns weights to the context pixel space of the haze feature by inserting the spatial domain attention module into the multi-scale dilated convolution group. The specific process is as follows: through the feature map average pooling and maximum pooling and splicing in the channel direction, the feature map dimension changes from C×H×W to 2×H×W; a convolution operation with a convolution kernel of 7×7, a step size of 1, and zero padding on the boundaries is performed to obtain the input feature map F c1 , and use the Sigmoid function to make the activation distribution between 0 and 1, and then multiply the single-channel feature map pixels after dimensionality reduction to complete the pixel weight assignment, so that the pixels at different spatial positions are replaced with probability values ​​in the range of 0 to 1, which is the weight information of pixels at different positions; input feature map F c1 Multiplying it by the spatial pixel weight is the weighted context pixel spatial feature of the haze feature.

[0021] The channel attention mechanism of the channel attention decoding structure is expressed as:

[0022]

[0023] CA c =δ(f1×1 [θ(F C2 )]) (8)

[0024] In the formula, X c (i, j) represents the pixel value of the X position of the cth channel, θ is the global average pooling function, f 1×1 is a 1×1 convolution operation, and δ is a Sigmoid activation function.

[0025] The channel domain attention encoding and decoding branch assigns weights of different channel feature maps by setting the ResNet autoencoder structure and introducing the channel attention decoding structure. The specific usage process is: use global average pooling to convert the channel global spatial information into basic channel elements, and the feature map dimension changes from C×H×W to C×1×1. 1×1 convolution is also called point-by-point convolution. The convolution kernel is 3×3. The channel feature map F is extracted. c2 , through 1×1 convolution and Sigmoid function, the parameters of each channel are replaced by probability values ​​in the range of 0 to 1, which is the weight of the channel, and F c2 With weight CA c The multiplication is the weighted channel feature.

[0026] The feature fusion branch expression is:

[0027]

[0028]

[0029] In the formula, is the gated fusion subnet, F1 and F2 are the fog layer feature information extracted by the contextual spatial domain attention branch and the channel domain attention encoding and decoding branch. and is the weight of the corresponding channel, and after fusion, F′ is the final fog layer feature.

[0030] The initial parameters of the two-branch residual feature fusion network in step 3 are as follows: small batches of 8 are used for parallel training, the Adam optimizer is used for gradient descent and update, 1000 epochs are set as the cycle batch to further optimize the weights, the learning rate uses 1e-5 as the initial value, and it is scaled every 500 epochs with scaling ratios of 0.01 and 0.001 respectively.

[0031] In step 5, the peak signal-to-noise ratio and structural similarity of the dehazed image are compared with the corresponding haze-free image in the test set. When the peak signal-to-noise ratio is 20-25 and the structural similarity is 0.7-0.8, the optimized two-branch residual feature fusion network is output as the optimal two-branch residual feature fusion network.

[0032] The beneficial effects of the present invention are:

[0033] (1) The unique pixel attention module can effectively remove the uneven haze distribution on different haze images;

[0034] (2) A context aggregation module is designed based on a multi-scale dilated convolution group, which can aggregate context feature information and exponentially improve the feature receptive field without increasing the number of training parameters;

[0035] (3) To address the problem of insufficient paired fog image datasets, the ResNet structure is applied to ImageNet for weight parameter pre-training to improve the robustness of the model;

[0036] (4) The feature fusion module adaptively fuses the fog layer features extracted by the two branches, assigning higher weights to important features and improving the ability to extract effective features. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is a flow chart of the defogging method based on dual-branch residual feature fusion of the present invention;

[0038] Figure 2 Schematic diagram of the double-branch residual feature fusion network structure in the present invention;

[0039] Figure 3 Schematic diagram of the attention branch structure in the context space domain of the present invention;

[0040] Figure 4 Schematic diagram of the channel domain attention encoding and decoding branch structure in the present invention;

[0041] Figure 5 Schematic diagram of the change of the objective evaluation index training process of the ablation experiment in an embodiment of the present invention;

[0042] Figure 6 It is a comparison diagram of subjective evaluation and thermal map of ablation experiment in an embodiment of the present invention;

[0043] Figure 7 This is a comparison chart of the effects of various algorithms when using images from the OTS dataset for dehazing;

[0044] Figure 8 This is a comparison chart of the effects of various algorithms when using images in the O-Haze dataset for dehazing;

[0045] Fig. 9 This is a comparison chart of the effects of various algorithms when using images in the NH-Haze dataset for dehazing. DETAILED DESCRIPTION

[0046] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0047] The present invention is based on a dual-branch residual feature fusion defogging method, such as Figure 1 As shown, the specific implementation steps are as follows:

[0048] Step 1: Take multiple scenes, collect original images of scenes without haze and artificial haze images of the same scene, form matching pairs of foggy and non-fog images, and select image pairs from open source datasets to form an image dataset;

[0049] Step 2: Divide the image dataset into a training set and a test set in a ratio of 9:1, and randomly crop, horizontally flip, and regionally block the training set image pairs to obtain an augmented training set.

[0050] Step 3, input the image pairs in the augmented training set into the two-branch residual feature fusion network for training, and obtain an optimized two-branch residual feature fusion network;

[0051] The dual-branch residual feature fusion network includes a contextual spatial domain attention branch that performs pixel attention on the high-frequency fog area of ​​the foggy image and a channel domain attention encoding and decoding branch that pays attention to the channel direction of the high-frequency haze features. The spatial domain attention branch assigns weights to the contextual pixel space of the haze features by inserting the spatial domain attention module into the multi-scale dilated convolution group; the channel domain attention encoding and decoding branch assigns weights to different channel feature maps by setting the ResNet autoencoder structure and introducing the channel attention decoding structure. It also includes a feature fusion branch, which uses adaptive weights to fuse the fog layer feature information of pixel attention and channel attention, and outputs an uneven fog residual layer.

[0052] The spatial domain attention module is used to effectively focus on dense fog pixels, expressed as:

[0053] SA c =δ(f 7×7 [G Max (F c1 );G Ave (F c1 )] (5)

[0054]

[0055] In the formula, δ is the Sigmoid activation function, F C1 is the input feature map, f 7×7 is a 7×7 convolution, Multiply corresponding inputs element-wise.

[0056] The following spatial domain attention branch assigns weights to the context pixel space of the haze feature by inserting the spatial domain attention module into the multi-scale dilated convolution group. The specific process is as follows: through the feature map average pooling and maximum pooling and splicing in the channel direction, the feature map dimension changes from C×H×W to 2×H×W; a convolution operation with a convolution kernel of 7×7, a step size of 1, and zero padding on the boundaries is performed to obtain the input feature map F c1 , and use the Sigmoid function to make the activation distribution between 0 and 1, and then multiply the single-channel feature map pixels after dimensionality reduction to complete the pixel weight assignment, so that the pixels at different spatial positions are replaced with probability values ​​in the range of 0 to 1, which is the weight information of pixels at different positions; input feature map F c Multiplying it by the spatial pixel weight is the weighted context pixel spatial feature of the haze feature.

[0057] The channel attention mechanism of the channel attention decoding structure is expressed as:

[0058]

[0059] CA c =δ(f 1×1 [θ(F C2 )]) (8)

[0060] Where, X c (i, j) represents the pixel value of the X position of the cth channel, θ is the global average pooling function, f 1×1 is a 1×1 convolution operation, and δ is a Sigmoid activation function.

[0061] The channel domain attention encoding and decoding branch assigns weights of different channel feature maps by setting the ResNet autoencoder structure and introducing the channel attention decoding structure. The specific usage process is: use global average pooling to convert the channel global spatial information into basic channel elements, and the feature map dimension changes from C×H×W to C×1×1. 1×1 convolution is also called point-by-point convolution. The convolution kernel is 3×3. The channel feature map F is extracted. c2 , through 1×1 convolution and Sigmoid function, the parameters of each channel are replaced by probability values ​​in the range of 0 to 1, which is the weight of the channel, and F c2 With weight CA c The multiplication is the weighted channel feature.

[0062] The feature fusion branch expression is:

[0063]

[0064]

[0065] In the formula, is the gated fusion subnet, F1 and F2 are the fog layer feature information extracted by the contextual spatial domain attention branch and the channel domain attention encoding and decoding branch. and is the weight of the corresponding channel, and after fusion, F′ is the final fog layer feature.

[0066] Step 4: Input the artificial haze image in the test set into the optimized two-branch residual feature fusion network, perform feature fusion based on convolution blocks and residual structure, extract the residual features of the haze layer, and output the haze impurity layer;

[0067] Step 5: The artificially created haze image in the test set is jump-connected with the haze impurity layer to output a dehazed image, and the peak signal-to-noise ratio and structural similarity of the dehazed image are compared with the corresponding haze-free image in the test set, and the optimal two-branch residual feature fusion network is determined according to the comparison results;

[0068] Step 6: Input the image to be dehazed into the optimal two-branch residual feature fusion network and output the dehazed image.

[0069] Example

[0070] We placed some burning straw and paper scraps on an open plain, ignited them, and collected images of smoke and haze rising. After the smoke dissipated, we took the corresponding fog-free images of the same scene and matched them one by one to create an independent dataset. At the same time, we added some image pairs from the open source dataset as data supplement, and created a total of 643 pairs of image datasets.

[0071] 90% of the image pairs in the image dataset are selected, and data augmentation operations are performed on small datasets by randomly using methods such as horizontal flipping, random cropping, regional occlusion, and direct output. This avoids problems such as overfitting and poor generalization ability caused by a small dataset.

[0072] The augmented dataset is input into the end-to-end dehazing network (two-branch residual feature fusion network) in the form of an image matrix, and small batches of mini-batches are trained in parallel with a batch size of 8. This occupies less video memory and can greatly constrain the distribution norm, avoiding problems such as inconsistent image styles and slow training convergence caused by large batches.

[0073] The Adam optimizer is used for gradient descent and update, and 1000 epochs are set as the cycle batch to further optimize the weights. The learning rate uses 1e-5 as the initial value, and is scaled every 500 epochs to 0.001 of the original value. After another 500 epochs, it is scaled back to 0.01 of the original value.

[0074] The imaging device receives two parts of light, one is the reflected light of the actual scene, and the other is the atmospheric light that enters the device after being scattered by suspended particles in the air. The expression is:

[0075] I(x)=J(x)t(x)+A·(1-t(x)), (1)

[0076] t(x)=exp(-β·d(x)), 0≤t(x)≤1. (2)

[0077] The residual error of the designed fog feature layer is r(x), which is obtained by subtracting the fog image from the fog-free image.

[0078] r(x)=I(x)-J(x), (3)

[0079] r(x)=(J(x)-A)(t(x)-1). (4)

[0080] The fog layer feature r(x) is learned to directly estimate the mapping relationship between the fog image and the fog-free image, avoiding the superposition error caused by multiple estimates of t(x) and A. The overall algorithm flow of the present invention is as follows: Figure 1 shown.

[0081] In the data output network, the context spatial domain attention branch is first used, and then three consecutive multi-scale dilated convolution blocks are used to exponentially expand the receptive field and improve the context aggregation ability. The spatial domain attention block is inserted to perform pixel attention on the uneven fog features, and the feature information of the pixel attention is fused through the residual connection to output the high-frequency and uneven fog feature map. The module structure is as follows: Figure 2 And as shown in Table 1.

[0082] The second branch passes through the channel domain attention encoding and decoding module, uses the pruned ResNet-101 network, freezes the pre-trained parameters of ImageNet for transfer learning, and fine-tunes the weight convolution kernel of the model; the automatic encoding and decoding structure is designed inspired by the U-Net structure, the channel attention mechanism is introduced, and multiple upsampling is used to achieve reconstruction and restoration of high-frequency feature information; the module structure is as follows Figure 3 shown.

[0083] Finally, the feature fusion branch is used to perform weighted fusion of the feature information output by the two branches by learning adaptive weights, and a gated subnet is designed as a feature fusion module to output the final fog layer residual feature map, i.e., the haze impurity layer. The overall network structure is shown in the figure. Figure 4 shown.

[0084] The artificially created haze image in the test set is jump-connected to the haze impurity layer to output the dehazed image. The peak signal-to-noise ratio and structural similarity of the dehazed image are compared with those of the corresponding haze-free image in the test set. When the peak signal-to-noise ratio is 20-25 and the structural similarity is 0.7-0.8, the optimized two-branch residual feature fusion network is output as the optimal two-branch residual feature fusion network.

[0085] The image to be dehazed is input into the optimal two-branch residual feature fusion network and the dehazed image is output.

[0086] In the dual-branch residual feature fusion network used in the present invention:

[0087] 1) Contextual Space Domain Attention Branch

[0088] The convolution layer of the basic convolution block uses a convolution kernel size of 3×3, a step size of 1, and a zero-filling operation on the border. Since the result of image defogging is greatly affected by a single image instance, the normalization effect of the entire batch is poor. Therefore, the instance normalization (InstanceNorm) method that standardizes the height and width of the image is used after the convolution layer, which can effectively accelerate network convergence and maintain the independence of image instances. Adding the nonlinear activation function ReLu (Rectified Linear Units) can improve the nonlinear relationship between the layers of the neural network. The traditional Sigmoid activation function has the problem of gradient disappearance. The ReLu activation function can clear the negative gradient of the input to zero, which can avoid gradient disappearance to a certain extent. A basic block is composed of a convolution layer, an InstanceNorm layer, and a ReLu layer.

[0089] The traditional CNN-based image dehazing network will process the feature information of spatial pixels equally, but the natural fog image has the problem of uneven fog distribution. The weight for very thin fog during convolution operation should be significantly different from the weight for dense fog area. Therefore, the spatial domain attention module is introduced to perform weighted assignment on pixels at different spatial positions, so that it pays more attention to the pixel distribution in different areas.

[0090] like Figure 3 As shown, the spatial domain attention module can effectively focus on dense fog pixels and local high-frequency areas of the image, expressed as

[0091] SA c =δ(f 7×7 [G Max (F c1 );G Ave (F c1 )], (5)

[0092]

[0093] In the formula, δ is the Sigmoid activation function, F C1 is the input feature map, f 7×7 is a 7×7 convolution, Multiply the corresponding elements of the input in element order. Perform average pooling and maximum pooling on the feature map and concatenate in the channel direction, and the feature map dimension changes from C×H×W to 2×H×W; perform a convolution operation with a convolution kernel of 7×7, a step size of 1, and zero padding on the boundaries, and use the Sigmoid function to assign pixel weights to the single-channel feature map after dimensionality reduction, so that pixels at different spatial positions are replaced with probability values ​​in the range of 0 to 1, which is the weight information of pixels at different positions; input F c Multiplying it with the spatial pixel weight PA is the weighted spatial pixel feature.

[0094] Table 1 Context aggregation module parameters

[0095]

[0096] As shown in Table 1, the context aggregation module has 4 layers, which use 3×3 dilated convolutions with different dilation rates, which are (1, 2, 5, 1), and the boundary is padded with zeros to keep the size of the feature map unchanged. The last layer is a 1×1 ordinary convolution layer; the Relu activation function is added after the convolution layer to add more nonlinear factors. The context aggregation module is combined with the pixel attention mechanism and residual connection is used to form a basic residual block. In order to balance the training effect and efficiency, three basic residual blocks are used to extract the fog layer features, and the intermediate feature map channels of the basic residual block output are set to 32.

[0097] 2) Channel Domain Attention Autoencoding Branch

[0098] The ResNet model has excellent feature extraction performance. The cropped ResNet-101 model is used as an autoencoder structure to extract fog layer features. The model has been pre-trained on the ImageNet dataset. The ImageNet image dataset is a large dataset containing 1.4 million images and 1,000 categories. Due to the small number of paired natural fog datasets, the pre-trained autoencoder can bring substantial prior knowledge to the small training dataset; the design of the self-decoding structure and the introduction of the channel attention mechanism to assign weights to different channels provide additional flexibility for processing information of different channel types.

[0099] like Figure 4 As shown, channel attention makes different channel features have different weight information, and the channel attention mechanism can be expressed as

[0100]

[0101] CA c =δ(f 1×1 [θ(F C2 )]), (8)

[0102] In the formula, X c(i, j) represents the pixel value of the X position of the cth channel, θ is the global average pooling function, f 1×1 is a 1×1 convolution operation, and δ is a Sigmoid activation function. Global Average Pooling is used to convert the global spatial information of the channel into basic channel elements. The dimension of the feature map changes from C×H×W to C×1×1. 1x1 convolution is also called pointwise convolution. It does not mix the information of different spatial features, and can distinguish channel feature learning from spatial feature learning. Through 1×1 convolution and Sigmoid function, the parameters of each channel are replaced with probability values ​​in the range of 0 to 1, which are the weights of the channel. c With weight CA c The multiplication is the weighted channel feature.

[0103] 3) Feature fusion branch

[0104] Fusion of features of different levels and types is very beneficial for both low-level and high-level tasks. The gated subnet is designed as a feature fusion module to adaptively learn the residual feature weights extracted by the two branches. The expression is:

[0105]

[0106]

[0107] In the formula, is the gated fusion subnet, F1 and F2 are the fog layer feature information extracted by the two branches, and is the weight of the corresponding channel, and after fusion, F′ is the final fog layer feature.

[0108] 4) Loss Function

[0109] The L1 function is also called the mean absolute error (MAE). It is a regression model that compares the difference pixel by pixel and takes the absolute value.

[0110]

[0111] Where N is the total number of pixels, c is the number of channels, represents the dehazed image, Represents a haze-free label map. L1 regularization creates sparse features and resets the weights of most useless features to 0, which has a feature selection effect.

[0112] Perceptual Loss is also called feature reconstruction loss. Comparing the feature maps obtained by convolution of the real image with those obtained by convolution of the generated image makes the content of the deep information closer to the global structure.

[0113]

[0114] In the formula, C j , H j , W j Respectively represent the feature map channel, height, width, φ extracted from the j-th layer feature j represents the j-th layer pre-trained VGG-16 feature extraction network, I * For the dehazed image, I gt This is a fog-free label image.

[0115] Adversarial Loss is the discriminator loss, which determines the probability of the network determining whether it is a fog-free image.

[0116]

[0117] In the formula, D(G(I gt )) is the reconstructed dehazed image G(I gt ) is the probability of a true fog-free image.

[0118] Setting different weight combinations for each loss function can effectively reduce the error of a single loss function.

[0119]

[0120] In the formula, in order to balance the weights of different losses, α=1, β=0.01, and γ=0.05 are set.

[0121] 5) Training environment

[0122] Hardware equipment: CPU: i5-8400@2.80GHz; GPU: NVIDIAGeForce GTX1080Ti; Memory 16G

[0123] Software equipment: Operating system: Windows 10 64-bit; Running framework: Pytorch 1.6

[0124] The initial learning rate α for training is 1e-5, the Adam optimizer is used, and the cosine annealing strategy is adopted to periodically adjust the learning rate;

[0125] 6) Module ablation analysis

[0126] In order to verify the effectiveness of the structure of each part of the network, based on the NH-Haze dataset, ablation analysis was performed on the contextual spatial domain attention branch, the channel attention encoding and decoding branch, and the two-branch residual fusion. Since subjective judgment cannot explain the effectiveness of the method of the present invention, the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) of each cyclic model are compared. SSIM is an indicator to measure the similarity between two images. The dehazed image is compared with the standard haze-free image. The larger the SSIM value, the smaller the distortion of the dehazed image. PSNR is an important indicator to measure image quality. It is the ratio of the maximum signal to the noise intensity. The larger the PSNR value, the closer the dehazed image is to the standard haze-free image. Figure 5 and Figure 6 shown.

[0127] 7)Comparison with other algorithms:

[0128] The test set data uses images that are not involved in the training in the synthetic dataset OTS, the real dataset NH-Haze and D-Haze. Since the datasets of the AODNet, GCANet and FFANet network structures do not include D-Haze and NH-Haze, in order to qualitatively compare the advantages and disadvantages of the algorithms, the respective networks were retrained using a unified dataset and training scheme. The final test images were uniformly cropped to [1024×1024] for result display, and the PSNR and SSIM objective evaluation criteria were used to analyze the dehazing effect. The comparison chart is as follows Figure 7 , Figure 8 , Fig. 9 As shown:

[0129] Table 2 Experimental results of each data set

[0130]

[0131] As shown in Table 2, all methods have good dehazing performance on the OTS dataset: the overall color of the DCP dehazed image is brighter, especially the large white area and sky part of the dehazed image have different degrees of color distortion; AODNet has poor dehazing effect in the image details, and the image color brightness is low; GCANet dehazed image has serious color shift, but the overall dehazing effect is relatively good. FFANet dehazed image has the best PSNR evaluation, and is closest to the standard non-hazed image in color and detail; DMPHN is more delicate in visual perception and richer in dark details compared to FFANet; the visual performance of the dehazed image of the present invention is slightly inferior, and the PSNR evaluation is slightly lower than FFANet.

[0132] On the O-Haze dataset: the DCP dehazing effect is poor, and the image color contrast is low; the AODNet dehazing function is almost ineffective; the GCANet dehazing effect is good, but the dehazed image color is reddish overall; the FFANet dehazing effect is good, but the dehazed image has serious color distortion as a whole; the DMPHN dehazing effect has obvious color shift, and the overall image color is whitish, but the dehazing effect is good in some areas; the present invention has excellent dehazing effect on local detail textures, and the overall look and feel is good with high color fidelity.

[0133] On the NH-Haze dataset: the dehazing functions of DCP, AODNet, and GCANet are basically ineffective, a large amount of residual fog remains in local areas of the FFANet dehazed image, and the detailed texture recovery performance is poor; the DMPHN dehazed image has a good overall appearance and is visually close to the standard haze-free image, but there are problems of detail loss and severe blur in local areas; the dehazed image of the present invention has only a small amount of residual fog in local areas, no obvious color distortion, and better detailed texture, and performs best in the objective evaluation of PSNR and SSIM.

[0134] In an embodiment of the present invention, the traditional convolutional neural network defogging method has a poor defogging effect on the real non-uniformly distributed fog image. The method proposed in the present invention can directly learn the mapping relationship between the foggy image and the fog-free image, extract the non-uniform fog residual features and achieve defogging. The context space domain attention structure and the channel attention autoencoder structure respectively focus on the pixel space and channel space of the feature map, adaptively fuse the feature information extracted by the two branches and output the final fog layer residual map, and introduce the discriminant network to make the defogging image closer to the real look and feel. Experiments show that the present invention can provide a defogging effect that is superior to the existing methods, but the processing process is time-consuming. The efficiency of the network outputting the haze layer will be optimized in the future so that it can be applied to engineering practices such as autonomous driving.

Claims

1. The dehazing method based on dual-branch residual feature fusion is characterized by: Follow these steps to implement: Step 1: Take multiple scenes, collect original images of scenes without haze and artificial haze images of the same scene, form matching pairs of foggy and non-fog images, and select image pairs from open source datasets to form an image dataset; Step 2: Divide the image data set into a training set and a test set in a ratio of 9:1, and perform data augmentation on the training set to obtain an augmented training set; Step 3, input the image pairs in the augmented training set into the two-branch residual feature fusion network for training, and obtain an optimized two-branch residual feature fusion network; The dual-branch residual feature fusion network includes a contextual spatial domain attention branch for pixel attention to the high-frequency fog area of ​​the foggy image and a channel domain attention encoding and decoding branch for paying attention to the channel direction of the high-frequency haze feature. The contextual spatial domain attention branch assigns weights to the contextual pixel space of the haze feature by inserting the spatial domain attention module into the multi-scale dilated convolution group; the channel domain attention encoding and decoding branch assigns weights to different channel feature maps by setting the ResNet autoencoder structure and introducing the channel attention decoding structure, and also includes a feature fusion branch. The feature fusion branch uses adaptive weights to fuse the fog layer feature information of pixel attention and channel attention, and outputs an uneven fog residual layer; The contextual spatial domain attention branch assigns weights to the context pixel space of the haze feature by inserting the spatial domain attention module into the multi-scale dilated convolution group. The specific process is as follows: through the feature map average pooling and maximum pooling and splicing in the channel direction, the feature map dimension changes from C×H×W to 2×H×W; a convolution operation with a convolution kernel of 7×7, a step size of 1, and zero padding on the boundaries is performed to obtain the input feature map , and use the Sigmoid function to make the activation distribution between 0 and 1, and then multiply the single-channel feature map pixels after dimensionality reduction to complete the pixel weight assignment, so that the pixels at different spatial positions are replaced with probability values ​​in the range of 0 to 1, which is the weight information of pixels at different positions; input the feature map Multiplying it with the spatial pixel weight is the weighted context pixel spatial feature of the haze feature; The channel domain attention encoding and decoding branch assigns weights of different channel feature maps by setting the ResNet autoencoder structure and introducing the channel attention decoding structure. The specific use process is: using global average pooling to convert the channel global spatial information into basic channel elements, the feature map dimension changes from C×H×W to C×1×1, 1×1 convolution is also called point-by-point convolution, the convolution kernel is 3×3, and the channel feature map is extracted. , through 1×1 convolution and Sigmoid function, the parameters of each channel are replaced by probability values ​​in the range of 0 to 1, which is the weight of the channel. With weight The multiplication is the weighted channel feature; The initial parameters of the dual-branch residual feature fusion network are as follows: a small batch of 8 is used for mini-batch parallel training, an Adam optimizer is used for gradient descent and update, 1000 epochs are set as the cycle batch to further optimize the weights, a learning rate of 1e-5 is used as the initial value, and the learning rate is scaled every 500 epochs, with scaling ratios of 0.01 and 0.001 respectively; Step 4: Input the artificial haze image in the test set into the optimized two-branch residual feature fusion network, perform feature fusion based on convolution blocks and residual structure, extract the residual features of the haze layer, and output the haze impurity layer; Step 5: The artificially created haze image in the test set is jump-connected with the haze impurity layer to output a dehazed image, and the peak signal-to-noise ratio and structural similarity of the dehazed image are compared with the corresponding haze-free image in the test set, and the optimal two-branch residual feature fusion network is determined according to the comparison results; Step 6: Input the image to be dehazed into the optimal two-branch residual feature fusion network and output the dehazed image.

2. The dehazing method based on dual-branch residual feature fusion according to claim 1 is characterized in that: In step 2, the specific process of data augmentation for the training set is: randomly cropping, horizontally flipping, and regionally blocking the training set image pairs to obtain an augmented training set.

3. The defogging method based on dual-branch residual feature fusion according to claim 1 is characterized in that: In step 3, the spatial domain attention module is used to effectively focus on dense fog pixels, expressed as: (5) ，(6) In the formula, is the Sigmoid activation function, is the input feature map, is a 7×7 convolution, Multiply corresponding inputs element-wise.

4. The dehazing method based on dual-branch residual feature fusion according to claim 1 is characterized in that: In step 3, the channel attention mechanism of the channel attention decoding structure is expressed as: (7) (8) In the formula, Indicates Channel The pixel value of the position, is the global average pooling function, is a 1×1 convolution operation, is the Sigmoid activation function.

5. The dehazing method based on dual-branch residual feature fusion according to claim 1, characterized in that: In step 3, the feature fusion branch expression is: (9) = (10) In the formula, is the gated fusion subnet, and The fog layer feature information extracted by the context space domain attention branch and the channel domain attention encoding and decoding branch. and is the weight of the corresponding channel, after fusion This is the final fog layer feature.

6. The dehazing method based on dual-branch residual feature fusion according to claim 1, characterized in that: In step 5, the peak signal-to-noise ratio and structural similarity of the dehazed image are compared with the corresponding haze-free image in the test set. When the peak signal-to-noise ratio is 20-25 and the structural similarity is 0.7-0.8, the optimized two-branch residual feature fusion network is output as the optimal two-branch residual feature fusion network.

Images