A single image defogging method based on random mask convolution and attention mechanism

Abstract

The application discloses a single image defogging method based on random mask convolution and attention mechanism, comprising the following steps: cutting the disclosed clear / foggy image dataset, and obtaining a training set after random horizontal or vertical flipping; a single image defogging network based on random mask convolution and attention mechanism is constructed, and the network as a whole adopts an encoder-decoder structure. In the encoder, random mask convolution is used to extract foggy image features, and the foggy image features are further processed through a coordinate attention module, a residual block and a dense feature fusion module. In the decoder, the image features generated in the encoding stage are decoded using deconvolution, a coordinate attention module and a dense feature fusion module to restore the original image feature size; the application can better restore the structure and color information of the fog image, so that the restored image is more in line with the requirements of human eye vision, and the effectiveness of the application method can be proved by the subjective and objective evaluation results.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image defogging technology based on convolutional neural network, and particularly relates to a single image defogging method based on random mask convolution and attention mechanism. BACKGROUND

[0002] In the field of computer vision, due to the low anti-interference ability of existing image acquisition devices to the real environment, fog weather will seriously affect the clear image acquisition, and further affect the computer vision application system. The influence of fog medium on light absorption, attenuation and scattering leads to serious degradation of the image captured in fog weather, so that the visual tasks that need to be performed on the clear image, such as image segmentation, target detection, unmanned driving, intelligent monitoring, military reconnaissance, satellite remote sensing, etc., appear performance degradation or failure. Therefore, it is of great practical significance to study an efficient image defogging algorithm to enable the computer to recover the corresponding clear image under fog conditions, and further improve the robustness of the visual system.

[0003] At present, the research on image defogging methods mainly includes three categories: the first is the defogging method based on image enhancement, the second is the defogging method based on atmospheric scattering model, and the third is the defogging method based on deep learning. The method based on image enhancement focuses on improving the visual satisfaction of fog image, without considering the physical process of image degradation, resulting in inaccurate recovery of local image color and detail information; the method based on atmospheric scattering model does not directly modify the fog image, but extracts and collects multiple input useful feature information while combining the atmospheric scattering model to generate high-quality images, but still has phenomena such as poor detail recovery effect and color distortion; the defogging method based on deep learning directly relies on the strong learning ability of deep learning to obtain the potential relationship between the fog image and the clear image, and the defogging effect is more in line with the requirements of human visual perception.

[0004] Document 1 (Qin X, Wang Z, Bai Y, et al. FFA-Net: Feature fusion attention network for single image dehazing [C] / / Proceedings of the AAAI conference on artificial intelligence. 2020, 34(07): 11908-11915.) uses channel attention and pixel attention to build a model to improve the performance of image dehazing, but lacks robustness when the model processes real scene hazy images. Document 2 (Wu H, Qu Y, Lin S, et al. Contrastive learning for compact single image dehazing [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2021: 10551-10560.) is an image dehazing method based on contrastive learning. This method proposes contrast regularization to balance positive and negative samples, and uses an automatic encoder-based compact network to ensure that the position of the restored image in the representation space is closer to the clear image and farther from the blurred image. However, the design of this network structure will result in a long inference time of the model. Document 3 (Shao Y, Li L, Ren W, et al. Domain adaptation for image dehazing [C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2020: 2808-2817.) is a dehazing method based on a generative adversarial network, which can reduce the domain difference and make the model more robust in real scenes. However, the training steps of the network are complex and time-consuming.

[0005] Deep learning-based image dehazing methods have achieved great success and have become the current mainstream dehazing method. However, they must rely on large-scale clear / hazy image data, high computational resource requirements, and more deep, wide, and complex network structures to improve image dehazing quality, greatly increasing the time of model training and inference. At the same time, the equal treatment of all pixels in the image by various modules in the network will lead to information loss, inaccurate dehazing estimation, blurred edges and details, and decreased visual quality. SUMMARY

[0006] To address the issues of long training and inference times and the equal treatment of all pixels during feature extraction in existing deep learning-based image dehazing methods, this invention proposes a single-image dehazing method based on random mask convolution and attention mechanisms. This method uses random mask convolution to extract image features from the training set and uses an attention mechanism to further enhance feature extraction. Then, residual blocks and dense feature fusion modules are used to effectively fuse contextual information, enabling the network to efficiently focus on fog-related features in the image, while also achieving high dehazing efficiency.

[0007] A single-image dehazing method based on random mask convolution and attention mechanism specifically includes the following steps:

[0008] S1. Randomly crop images to size 256×256 from the publicly available image dehazing dataset RESIDE and randomly flip them horizontally or vertically to obtain the training set.

[0009] S2. Construct random mask convolutional layers as components of the network encoding part to effectively extract the required feature information and avoid redundant features; specifically including the following steps:

[0010] S21. Create random mask templates, wherein the size and number of random mask templates match the size and number of convolutional kernels in each convolutional layer of the convolutional neural network;

[0011] S22. Set the number of masks in each mask template;

[0012] S23. Combine random mask templates with convolutional kernels to construct random mask convolutional layers, and keep the number and position of the masks unchanged during each epoch of training.

[0013] S3. Construct an image dehazing convolutional neural network based on random mask convolution and attention mechanisms. The image dehazing convolutional neural network includes an encoder and a decoder. In the encoder, the feature image of the hazy image is first extracted through the random mask convolutional layer constructed in step S2. Then, the feature image of the hazy image is processed and fused by introducing coordinate attention, a residual module, and a dense feature fusion module to obtain the high-level semantic features of the hazy image. In the decoder, deconvolution is used to upsample the high-level semantic features to obtain the detailed information of the dehazed image. The obtained image detailed information is then further processed and fused by the coordinate attention and dense feature fusion modules. The residual block consists of two convolutional layers and a PRelu activation function layer, and the input is directly added to the output of the residual block. The dense feature fusion module fuses features from multiple different levels. The feature fusion process is as follows:

[0014]

[0015] in Dn-1is the output of the n-th dense feature fusion module, n denotes dense feature fusion, q n is the output of the n-th attention module, denotes the output from the first n-1 dense feature fusion modules in the decoder / encoder;

[0016] The encoder first expands the input feature image channel number to 16 using ordinary convolution with a step size of 1 and a convolution kernel size of 11x11, and then performs 4 times of down-sampling operation on the image containing fog features using random mask convolution with a step size of 2 and a convolution kernel size of 3x3; after each down-sampling, the channel number of the image containing fog features is doubled, and after each down-sampling, the coordinate attention, the dense feature fusion module and the residual block are used to further fuse the context information of the feature image;

[0017] The decoder first uses deconvolution with a step size of 2 to up-sample the encoded high-level semantic features, and the channel number of the clear feature image after each up-sampling is halved, from 256 to 128 in the first time; then the coordinate attention and the dense feature fusion module are used to restore the clear image features for the up-sampled feature image; the above process needs to be performed 4 times to gradually restore the clear image features, and after the fourth up-sampling, the channel number of the clear feature image is reduced to 16; the last convolution uses ordinary convolution with a step size of 1 and a convolution kernel size of 11x11 to restore the image without fog;

[0018] The encoder-decoder adopts a skip connection between the same level, and the features extracted by the residual block of the encoder are connected to the coordinate attention layer in the decoder through the skip connection;

[0019] The image dehazing convolutional neural network introduces coordinate information through the coordinate attention mechanism, calculates the relative importance between each element and other elements in the feature map, and transfers and aggregates these weight information, and the specific steps are as follows:

[0020] First, use global pooling to encode each channel along the horizontal and vertical directions respectively, and get z w , z h , z w , z h Embedding z h , z w in the network space, then merge z h , z w in the channel dimension, and then use 1x1 convolution F1 to transform:

[0021] f = δ (F1 ([z h , z w ]))

[0022] where [·,·] represents a merge operation, δ represents a nonlinear activation function, and f represents a merged intermediate feature map; f is then divided into and r represents a down-sampling size, C represents the number of channels of the feature map, H represents the height of the feature map, and W represents the width of the feature map; two 1x1 convolutions F h and F w are used to transform f h and f w into tensors with the same input channel size as the input, to obtain g h , g w , respectively; finally, the output of the coordinate attention is:

[0023]

[0024] where y c is sent to the dense feature fusion module for further processing.

[0025] S4, related hyperparameters and optimization algorithms required when setting the training image dehazing convolutional neural network are set, and the optimization direction of the network is constrained by the mean square error loss function, the network is iteratively trained to continuously reduce the loss, and finally the weight parameters of the network are obtained and saved.

[0026] The loss function used when training the image dehazing convolutional neural network is specifically:

[0027]

[0028] In the formula, n b is the batch size selected in the training process, N(I i ) and J i represent the dehazed image and the corresponding haze-free image, respectively, and || ||2 represents the standard L2 regularization.

[0029] S5, an image dehazing model based on random mask convolution and attention mechanism is constructed using the image dehazing convolutional neural network constructed in step S3 and the network training weight parameters obtained in step S4, and the image dehazing model is used to dehaze the haze-containing images in the test set to obtain dehazed images.

[0030] Advantages:

[0031] 1. The application provides a single image defogging method based on random mask convolution and attention mechanism. For the public data set, the application cuts the data set and randomly flips it horizontally or vertically to enhance the training data set; uses a U-Net network as the basic network architecture, and combines random mask convolution, coordinate attention, residual block and dense feature fusion module to build the network, so that the network has strong feature extraction capability and can process multi-scale information, while improving the perception and context understanding ability of the model.

[0032] 2. The application uses random mask convolution to avoid extracting too much redundant information and effectively extracts image features while reducing network parameters; the coordinate attention mechanism can not only capture cross-channel information, but also capture direction perception and position sensitive information, which can help the network more accurately identify the object of interest; the dense feature fusion module and the residual block fusion network fuse the features at each level, making the restored clear image more realistic and significantly improving the visual quality. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 The application is a deep learning model structure based on random mask convolution and attention mechanism;

[0034] Figure 2 The application is a random mask convolution layer structure diagram;

[0035] Figure 3 The application is a coordinate attention structure diagram;

[0036] Figure 4 The application is a dense feature fusion module structure diagram;

[0037] Figure 5 The application is a flowchart of an embodiment;

[0038] Figure 6 The application is a defogging effect comparison diagram of the defogging method. DETAILED DESCRIPTION

[0039] To further describe the single image defogging method based on random mask convolution and attention mechanism proposed by the application, an embodiment of the application is further described in combination with the drawings to strengthen the understanding of the purpose, technical features and application effect of the application. The embodiment is only a further explanation of the application, and researchers in the field can further improve and adjust the non-essential technology on the basis of the application, which is also within the protection scope of the application.

[0040] A single image defogging method based on random mask convolution and attention mechanism, specifically comprising the following steps:

[0041] S1, randomly crop images from the public image defogging dataset RESIDE to 256x256 size and randomly flip horizontally or vertically to obtain the training set.

[0042] S2, construct a random mask convolutional layer as a network coding part component, as shown in Figure 2 for effective extraction of the required feature information and to avoid redundant features; specifically comprising the following steps:

[0043] S21, create a random mask template, wherein the size and number of the random mask template match the size and number of the convolutional kernels in each convolutional layer in the convolutional neural network; as shown in Figure 2 k1 in the figure, the white small squares represent masks of 1, indicating that the corresponding convolutional kernel weights will participate in network training, and the black small squares represent masks of 0, indicating that the corresponding convolutional kernel weights will not participate in network training;

[0044] S22, set the convolutional kernel size of the random mask convolutional layer to 3x3, and set the number of masks corresponding to each convolutional kernel to 3;

[0045] S23, combine the random mask template with the convolutional kernel to construct the random mask convolutional layer, and in each epoch training process, the number and position of the masks remain unchanged.

[0046] S3, construct an image defogging convolutional neural network based on random mask convolution and attention mechanism; the image defogging convolutional neural network includes an encoder and a decoder, in the encoder, first extract the feature image of the fog image through the random mask convolutional layer constructed in step S2, then process and fuse the feature image of the fog image through the introduction of coordinate attention, residual module and dense feature fusion module to obtain the high-level semantic features of the fog image; in the decoder, the high-level semantic features are upsampled to obtain the detailed information of the defogging image using deconvolution, and the obtained image detailed information is further processed and fused through the coordinate attention and dense feature fusion module.

[0047] The encoder is used to capture different levels of semantic information in the image, first uses ordinary convolution with a step size of 1 and a convolutional kernel size of 11x11 to expand the input feature image channel number to 16, and then uses random mask convolution with a step size of 2 and a convolutional kernel size of 3x3 to perform 4 times down-sampling operation on the fog feature image; after each down-sampling, the fog feature image channel number is doubled, and after each down-sampling, the coordinate attention, dense feature fusion module and residual block are used to further fuse the context information of the feature image;

[0048] The decoder first uses deconvolution with a step size of 2 to up-sample the encoded high-level semantic features, and the number of clear feature image channels is halved each time, from 256 to 128 in the first time; then the up-sampled feature image is recovered using the coordinate attention and dense feature fusion module to restore the clear image feature; the above process needs to be performed 4 times to gradually restore the clear image feature, and after the fourth up-sampling, the number of clear feature image channels is reduced to 16; the last convolution uses ordinary convolution with a step size of 1 and a convolution kernel size of 11x11 to restore the haze-free image;

[0049] The encoder-decoder structure takes a skip connection between the same levels, as shown in Figure 1 The features extracted by the residual block of the encoder are connected to the input of the coordinate attention module in the decoder through a skip connection;

[0050] The coordinate attention mechanism in the image haze removal convolutional neural network calculates the relative importance between each element and other elements in the feature map by introducing coordinate information, and transmits and aggregates these weight information, and the specific steps are as follows:

[0051] First, use global pooling to encode each channel along the horizontal and vertical directions respectively, and get z w and z h respectively. w h After embedding z h and z w into the network space, z h and z w are merged in the channel dimension, and then a 1x1 convolution F1 is used for transformation operation:

[0052] f=δ(F1([z h , z w ]))

[0053] Where [·, ·] represents the merging operation, δ represents the nonlinear activation function, and f represents the merged intermediate feature map; then, f is divided into and along the horizontal and vertical directions respectively, where r represents the down-sampling size, which is used to reduce the feature map channel and reduce the calculation amount, C represents the channel number of the feature map, H represents the height of the feature map, and W represents the width of the feature map; then, two 1x1 convolutions F h and F w are used to transform f h and f w into tensors with the same input channel size as the input, respectively, to get g h and g w ; finally, the output of the coordinate attention is:

[0054]

[0055] wherein y c The input is sent to the dense feature fusion module for further processing.

[0056] The residual block in the image defogging convolutional neural network directly transmits the input to the output of the block, thereby improving the information loss problem in the information transmission process, and the specific structure is as follows:

[0057] For the input X, a convolutional layer with a convolution kernel of 3*3 and a step of 1 is used to extract the input features, then a PRelu activation function layer is used, then a convolutional layer with a convolution kernel of 3*3 and a step of 1 is used to further extract the features, and finally the input X is added to the output of the residual block;

[0058] The dense feature fusion module fuses features from multiple different levels to comprehensively utilize low-order or high-level position space information. The feature fusion process is represented as follows:

[0059]

[0060] wherein is the output of the nth dense feature fusion module, D n represents dense feature fusion, q n is the output of the nth attention module, represents the output from the first n-1 dense feature fusion modules in the decoder / encoder; the specific iterative updating steps are as follows:

[0061] (1) Calculate the output of the coordinate attention module at the nth layer at the tth iteration and the difference between the output of the dense feature fusion module before the tth iteration

[0062]

[0063] wherein is a convolution operation, and the function is to downsample to have the same size as q L-t

[0064] (2) Update the value of

[0065]

[0066] wherein is a deconvolution operation, and the function is to upsample to have the same size as .

[0067] ​​​S4, the related hyperparameters and optimization algorithm required when setting the training image defogging convolutional neural network are set, and the optimization direction of the network is constrained by the mean square error loss function, the network is trained by cyclic iteration to make the loss continuously decrease, and finally the weight parameters of the network are obtained and saved; the epoch of the network is set to 50, the batchsize is set to 16, the optimization algorithm uses adam, wherein beta1=0.9, beta2=0.999, and the initial learning rate is set to 10 -4 .

[0068] The mean square error loss function used when training the image defogging convolutional neural network is specifically:

[0069]

[0070] In the formula, n b is the batch size selected in the training process, N(I i ) and J i represent the defogging image and the corresponding haze-free image respectively, and || ||2 represents the standard L2 regularization.

[0071] S5, the single image defogging model based on random mask convolution and attention mechanism is constructed by using the image defogging convolutional neural network constructed in step S3 and the network training weight parameters obtained in step S4, and the image defogging model is used to defog the haze-containing images in the test set to obtain the defogging images.

[0072] Simulation experiment:

[0073] The subjective defogging quality graph of the application and the subjective defogging quality graph of the method of document 1 are as shown in Figure 6 , the first row is a haze-containing image, the second row is a real clear image, the third row is the subjective defogging quality graph of the method of document 1, and the fourth row is the subjective defogging quality graph of the method of the application. At the same time, in order to quantitatively evaluate the defogging effect of the application, the application is tested on the SOTS test set, and objective evaluation indexes such as PSNR, SSIM and test time are used for evaluation.

[0074] Table 1 Image defogging quantitative evaluation index

[0075] Method PSNR SSIM Test time(s) Document 1 method 32.263 0.979 0.117 Inventive method 33.012 0.982 0.092

[0076] From Figure 6 it can be seen that: the detail recovery effect of the method of document 1 is not ideal, while the image after defogging by the image defogging method designed by the application is clearer and more natural.

[0077] From table 1, it can be seen that the designed method of the application obtains better effect in quantitative evaluation, which shows that the method of the application can restore images with better perceptual quality, and has faster processing speed.

[0078] The foregoing description of the methods and systems of the present application enable those skilled in the art to reproduce the application. Based on the content of the present application, other embodiments and results obtained by researchers without making creative efforts should be within the scope of the present application.

Claims

1. A single image defogging method based on random mask convolution and attention mechanism, characterized in that, The method comprises the following steps: S1, randomly cropping images from a public image defogging dataset RESIDE to 256x256 size and performing random horizontal or vertical flipping to obtain a training set; S2, constructing a random mask convolution layer as a network coding part component for effectively extracting required feature information and avoiding redundant features; S3, constructing an image defogging convolutional neural network based on random mask convolution and attention mechanism; the image defogging convolutional neural network comprises an encoder and a decoder; in the encoder, first, the random mask convolution layer constructed in step S2 is used to extract a feature image of a fog image, then the feature image of the fog image is processed and fused by introducing a coordinate attention, a residual module and a dense feature fusion module to obtain high-level semantic features of the fog image; in the decoder, the high-level semantic features are upsampled by deconvolution to obtain detailed information of a defogged image, and the obtained image detailed information is further processed and fused by the coordinate attention and the dense feature fusion module, wherein the residual block is composed of two convolution layers and a PReLU activation function layer, and the input is directly added to the output of the residual block; the dense feature fusion module fuses features from multiple different levels, and the feature fusion process is as follows: S4, setting related hyperparameters and optimization algorithms required when training the image defogging convolutional neural network, and jointly constraining the optimization direction of the network by a mean square error loss function, and iteratively training the network to continuously reduce the loss, and finally obtaining the weight parameters of the network and saving them, wherein the calculation formula of the mean square error loss function is as follows: wherein is the output of the nth dense feature fusion module, D n denotes dense feature fusion, q n is the output of the nth attention module, denotes the output from the first n-1 dense feature fusion modules in the decoder / encoder; S5, constructing a single image defogging model based on random mask convolution and attention mechanism by using the image defogging convolutional neural network constructed in step S3 and the network training weight parameters obtained in step S4, and using the image defogging model to defog the foggy images in the test set to obtain defogged images. where n b is the batch size selected during training, N(I i ) and J i represent the hazy image and the corresponding haze-free image, respectively, and || ||2 denotes the standard L2 regularization. The specific steps of constructing the random mask convolution layer in step S2 are as follows:

2. The single image defogging method based on random mask convolution and attention mechanism according to claim 1, characterized in that, S21, creating a random mask template, wherein the size and number of the random mask template match the size and number of the convolution kernel in each convolution layer in the convolutional neural network; S22, setting the number of masks in each mask template; S23, combining the random mask template with the convolution kernel to construct a random mask convolution layer, and keeping the number and position of the masks unchanged during each epoch training process. The specific network structure of the encoder and the decoder in step S3 is as follows:

3. The single image defogging method based on random mask convolution and attention mechanism according to claim 1, characterized in that, The encoder is used to capture different levels of semantic information in the image, first, an ordinary convolution with a step of 1 and a convolution kernel size of 11x11 is used to expand the channel number of the input feature image to 16, and then a random mask convolution with a step of 2 and a convolution kernel size of 3x3 is used to perform 4 times of downsampling operation on the foggy feature image; after each downsampling, the channel number of the foggy feature image is doubled; after each downsampling, the coordinate attention, the dense feature fusion module and the residual block are used to further fuse the context information of the feature image; ​ The decoder first uses deconvolution with a step size of 2 to up-sample the encoded high-level semantic features, and the number of clear feature image channels is halved each time, from 256 to 128 in the first time, and then the up-sampled feature image is restored to a clear image feature using the coordinate attention and dense feature fusion module; The above process needs to be performed 4 times to gradually restore the clear image feature, and after the fourth up-sampling, the number of clear feature image channels is reduced to 16; the last convolution uses ordinary convolution with a step size of 1 and a convolution kernel size of 11x11 to restore the haze-free image; The encoder-decoder adopts a skip connection between the same levels, and the features extracted by the residual block of the encoder are connected to the coordinate attention layer in the decoder through the skip connection.

4. The single image defogging method based on random mask convolution and attention mechanism according to claim 1, characterized in that, The coordinate attention is introduced in step S3, and the coordinate attention mechanism calculates the relative importance between each element and other elements in the feature map by introducing coordinate information, and transmits and aggregates according to the weight information. The specific steps are as follows: First, use global pooling to encode each channel along the horizontal and vertical directions respectively, and get z w ,z h Embed z w , z h Containing coordinate information into network space, z h , z w Merge in the channel dimension, and then use 1x1 convolution F1 to transform operation: f = δ(F1([z h , z w ])) where [·, ·] represents a merge operation, δ represents a nonlinear activation function, and f represents a merged intermediate feature map; then, f is divided into and r represents a down-sampling size, C represents a channel number of a feature map, H represents a height of a feature map, and W represents a width of a feature map; then, two 1x1 convolutions F h and F w are used to transform f h and f w into tensors with the same input channel size as the input, to obtain g h and g w respectively; finally, the output of the coordinate attention is: wherein y c The input is sent to the dense feature fusion module for further processing.

Images