Unsupervised defogging system and method based on visual quality driving

Abstract

The application discloses a kind of unsupervised defogging system and method based on visual quality driving, by constructing end-to-end network learning between foggy image and fog-free image mapping relationship, directly generate fog-free image;The system is composed of information interaction module and iteration module, wherein, information interaction module is used to realize the efficient fusion of input shallow feature, complete the interaction of detail information and structure information, learn to mine deep semantic information;Iteration module 200 is used to estimate K (x) first, and the K (x) value is substituted into the recomposed atmospheric scattering model to calculate the fog-free image, and the defogging result is iterated and refined multiple times to retain image details.Compared with the prior art, the defogging image of the present application has more natural color and better defogging effect, and the network training process only requires foggy images, which can fundamentally avoid domain shift phenomenon.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer image processing, and particularly relates to an unsupervised image defogging method. BACKGROUND

[0002] In the haze weather condition, a large number of fine particles are suspended in the air. The reflection light of the scene object and these particles have absorption and scattering effects, which causes color degradation and detail blur of the foggy image.

[0003] At present, most methods learn the related features of haze in the foggy image through a neural network, and directly restore the final haze-free image. The dehazing method based on supervised learning (Li B, Peng X, Wang Z, et al. An all-in-one network for dehazing and beyond [J]. arXiv preprint arXiv: 1707.06543, 2017.) has achieved certain dehazing effect, but the success of this method is largely dependent on the paired data set. These models trained on synthetic images have the phenomenon of incomplete dehazing for real foggy images. In order to improve the generalization performance of the network, the dehazing method based on weak supervision learning (Li L, Dong Y, Ren W, et al. Semi-supervised image dehazing [J]. IEEE Transactions on Image Processing, 2019, 29: 2766-2779; Shao Y, Li L, Ren W, et al. Domain adaptation for image dehazing [C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2020: 2808-2817.) is proposed, which can alleviate the domain shift problem caused by the data distribution difference between real images and synthetic images to a certain extent, but it is still limited by the synthetic data set, it is difficult to obtain useful information from the thick haze covered image, and it takes a lot of time and GPU resources to train a model. Therefore, the dehazing method based on unsupervised learning has more practical application value, the network is trained on real foggy images, which completely solves the color distortion and incomplete dehazing caused by the use of synthetic training set due to domain shift, and effectively improves the performance and efficiency of the network. SUMMARY

[0004] Inspired by unsupervised thinking, the present application aims to propose a visual quality driven unsupervised image defogging system and method, which only uses a foggy image to train a network model, and through mining more abstract deep semantic information, the output error is greatly reduced through multiple iteration calculation of defogging results.

[0005] The present application realizes the following technical solutions:

[0006] An unsupervised defogging system based on visual quality driving, by constructing an end-to-end network to learn the mapping relationship between foggy images and fog-free images, directly generating fog-free images; the system is composed of an information interaction module and an iteration module, wherein:

[0007] The information interaction module is used to realize efficient fusion of input shallow features, complete interaction of detail information and structure information, and learn to mine deep semantic information;

[0008] The iteration module is used for K(x i ) estimation, substituting the K(x i ) value into the reorganized atmospheric scattering model to calculate the fog-free image, and multiple iterations of the defogging result to retain image details.

[0009] An unsupervised defogging method based on visual quality driving, the method comprising the following specific steps:

[0010] Step 1, obtaining a total training set and a test set, wherein the training data set is an indoor training set ITS and an outdoor training set OTS, and the test set is a comprehensive target test set SOTS and a mixed subjective test set HSTS;

[0011] Step 2, constructing an unsupervised defogging network, which includes an information interaction module and an iteration module;

[0012] In the information interaction module, the input features x m and x n are subjected to convolution operation to generate features x mn containing different types of information; then, feature fusion is performed on each pair of information features, and a ReLU activation function is used after each convolution layer for non-linear transformation; after multiple information interaction processes and feature re-extraction, high-level semantic information in the image is learned;

[0013] In the iteration module, the output of the information interaction module is used as the input of the iteration module; first, a 3x3 convolution layer is used to extract features from the input features, and the intermediate variable K(x i ) of the model is estimated.

[0014] The total loss function L in network training is set to include a content preservation loss function, a dark channel loss function, a contrast loss function, a saturation loss function and a sharpening loss function to realize label-free training of the network.

[0015] The total loss function L expression is as follows:

[0016] L = L mse + L dcp + L con + L sat + L sharp

[0017] Wherein, L mse is the content preservation loss function, L dcp is the dark channel loss function, L con is the contrast loss function, L sat is the saturation loss function, and L sharp is the sharpening loss function.

[0018] Step 3, using the constructed unsupervised defogging network, realizing single image defogging, estimating the intermediate variable of the intermediate variable model of the atmospheric scattering model, substituting the intermediate variable value of the intermediate variable model of the atmospheric scattering model into the recombined atmospheric scattering model to calculate the haze-free image, and repeatedly iterating and refining the defogging result to retain image details.

[0019] Compared with the prior art, the present application can achieve the following beneficial technical effects:

[0020] 1) The defogged image is more natural in color and better in defogging effect, and the network training process only needs a foggy image, so that the domain shift phenomenon can be fundamentally avoided;

[0021] 2) Three different forms of regular path queries are processed, and the result connection problem of mixed form queries is solved. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 It is a structure schematic diagram of the unsupervised defogging system based on visual quality driving of the present application;

[0023] Figure 2 It is a process schematic diagram of the unsupervised defogging method based on visual quality driving of the present application;

[0024] Figure 3 It is a structure schematic diagram of the information interaction module;

[0025] Figure 4 It is a structure schematic diagram of the iteration module;

[0026] Figure 5The experimental results are shown in the following comparison figures: (a) Hazy image, (b) DCP, (c) DehazeNet, (d) AODNet, (e) GCANet, (f) GridDehazeNet, (g) YNet, and (h) the method of this invention. Detailed Implementation

[0027] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0028] The basic principles of this invention are briefly described below:

[0029] Based on the physical scattering characteristics, a haze image model is established, expressed as follows:

[0030] I(x i )=J(x i )t(x i )+A(1-t(x i )) (1)

[0031] Where, x i Let I(x) be the position of the i-th pixel in the image. i ) represents a haze image, J(x) i ) represents the restored sharp image, A represents the atmospheric light value, and t(x) represents the image. i () represents the transmission matrix;

[0032] Based on formula (1), the atmospheric scattering model is reorganized, and the expression is as follows:

[0033] J(x i )=K(x i )I(x i )-K(x i )+b (2)

[0034] Where b is a constant, K(x) i ) is an intermediate variable in the atmospheric scattering model, and t(x) in equation (1) is used as an intermediate variable. i ) and A are unified as K(x) i The specific relationships are as follows:

[0035]

[0036] Based on the reconstructed atmospheric scattering model, the key to defogging is calculating K(x) i This method achieves dehazing without distorting the image.

[0037] like Figure 1As shown, it is a visual quality driven unsupervised defogging structure schematic diagram of the application. By constructing an end-to-end network to learn the mapping relationship between the foggy image and the fog-free image, the fog-free image can be directly generated. The system is composed of an information interaction module 100 and an iteration module 200, wherein the information interaction module 100 is used to complete the interaction of the detail information and the structure information, and learn the deep semantic information such as the texture, color and shape of the image; the iteration module 200 is used to firstly estimate K(x i ), and then perform multiple iteration operations on the defogging result to retain the image details. As shown in Figure 2 , it is a visual quality driven unsupervised defogging method flowchart of the application, and the specific processing process is as follows:

[0038] Step 1, obtain the total training set and the test set:

[0039] The public data set RESIDE is used in the application, the training data set is an indoor training set ITS (Indoor Training Set) and an outdoor training set OTS (Outdoors Train Set), the ITS contains 1399 clear images and 13990 foggy images, the OTS contains 8477 clear images and 296695 foggy images, 2061 clear images and corresponding 72135 foggy images are selected as the training set of the outdoor foggy image. The test set is a comprehensive target test set (SOTS) and a mixed subjective test set (HSTS), the SOTS contains 500 indoor images (not overlapped with the training image), and the foggy image is synthesized according to the same process as the training data, the HSTS contains 10 synthesized outdoor foggy images and 10 real world clear images;

[0040] Step 2, construct an unsupervised defogging network:

[0041] As shown in Figure 2As shown, the unsupervised dehazing network is a convolutional neural network structure, including an information interaction module (IIM) and an iteration module (IM). The information interaction module (IIM) 100 is used to efficiently fuse shallow input features, completing the interaction of detailed and structural information to mine deep semantic information. The iteration module (IM) 200 is used to perform multiple iterations to refine the dehazing result. This module only requires one convolutional layer, reducing the error of the output result without excessively increasing the number of convolutional layers, thus ensuring the network's efficiency. Furthermore, when pre-extracting features from the input hazy image, the unsupervised dehazing network uses convolutional layers with receptive fields of different sizes to extract shallow features more relevant to dehazing. By setting the convolutional kernel size, different receptive field sizes are obtained, capturing features at different scales and comprehensively extracting the image's detailed and structural information. In this step, the convolutional layers are sequentially set to kernel sizes of 3×3, 5×5, 3×3, and 5×5.

[0042] like Figure 3 The diagram shown is a schematic of the information interaction module structure. In this information interaction module, the m-th layer of the network inputs feature x. m and the input features x of the nth layer n Perform convolution operations to generate fused features x that contain information from different categories. mn The features from layer 1 and layer 2 are convolved to obtain the fused feature x. 12 The features from layers 3 and 4 are convolved to obtain the fused feature x. 34 The features from layer 1 and layer 3 are convolved to obtain the fused feature x. 13 The features from layers 2 and 4 are convolved to obtain the fused feature x. 24 The convolutional layers used here have a kernel size of 7×7, a stride of 1, padding of 3, and 36 output channels. Each convolutional layer is followed by a ReLU activation function for non-linear transformation. Through multiple information interaction processes and feature re-extraction, high-level semantic information in the image can be learned. To maintain information integrity, the network does not use pooling operations at all, effectively avoiding the loss of details.

[0043] like Figure 4 The diagram shown is a schematic of the iterative module structure.

[0044] To embed the reconstructed atmospheric scattering model into the network design, the intermediate variable K(x) of the model is first estimated. i ), will K(x i The values ​​are substituted into the reconstructed atmospheric scattering model to calculate the fog-free image.

[0045] The input to the iterative module 200 is the output of the information interaction module 100. It first uses a 3×3 convolutional layer to extract features from the input features and calculates K(x). i The convolutional layer has 24 output channels. The output feature map is divided into 8 groups of 3 adjacent channels. The g-th group is denoted as K. g (x i The convolutional layer is followed by the Tanh activation function, and the output value ranges from [-1, 1].

[0046] Iteration module 200 estimates K(x) i An enhancement strategy was employed to make the details of the enhanced image clearer. The process of applying an enhancement strategy to an image is as follows:

[0047] J1(x i )=K1(x i )I(x i )-K1(x i )+1 (4)

[0048] J n (x i ) = K n (x i )(I(x i )+J n-1 (x i ))-K n (x i )+1-J n-1 (x i ), n=2,3,...,8 (5)

[0049] Where, x i For the i-th pixel of the input image, J1(x) i J represents the results of the first defogging attempt. n (x i ) represents the dehazing result of the nth iteration.

[0050] The specific design of the loss function is as follows:

[0051] To train the network proposed in this invention, the total loss function consists of five parts, including the content preservation loss function, the dark channel loss function, the contrast loss function, the saturation loss function, and the sharpening loss function.

[0052] Content Preservation Loss Function L mse Represented as:

[0053]

[0054] Where, x i J is the number of pixels in the input image. n(x i ) represents the dehazing result of the nth iteration, I(x) i ) represents a haze image, and M represents the total number of pixels in the image.

[0055] Dark channel loss function L dcp Represented as:

[0056] L dcp =|MaxPool3d(Z(x) i )-I(x i ))-I(x i (7)

[0057] Where MaxPool3d(.) is the three-dimensional max pooling function, I(x i Z(x) is the input foggy image. i ) is a sheet of size I(x) i Images that are identical and have all pixel values ​​of 0.

[0058] Contrast loss function L con Represented as:

[0059]

[0060] Among them, C I(xi) For the contrast of haze images, C J(xi) For a clear image, M represents the total number of pixels in the image.

[0061] Saturation loss function L sat Represented as:

[0062]

[0063]

[0064] L sat =S J -S I (11)

[0065] Among them, S I and S J The saturation values ​​for the hazy and clear images are x and x, respectively. i Let q ∈ (R, G, B) be the position of the i-th pixel, and let I be one of the three channels of RGB. q (x i J represents a channel in a haze image. q (x i Let be a channel of a clear image. max(.) finds the maximum value and min(.) finds the minimum value.

[0066] Sharpening loss function Lsharp Represented as:

[0067]

[0068] Wherein, J(x) i ) is a clear image, I(x) i G(·) represents a haze image, G(·) is the value obtained by performing the Sobel operator operation on the image, and M is the number of pixels.

[0069] Therefore, the total loss function L is expressed as:

[0070] L = L mse +L dcp +L con +L sat +L sharp (13)

[0071] Step 3: Use the constructed unsupervised dehazing network to achieve dehazing of a single image: First, estimate the intermediate variable K(x) of the atmospheric scattering model. i Then, the intermediate variable K(x) of the atmospheric scattering model is... i The values ​​are substituted into the reconstructed atmospheric scattering model to calculate the haze-free image. The haze-free result is refined through multiple iterations to preserve image details.

[0072] Table 1 shows the comparison results between prior art and the present invention in terms of PSNR (peak signal-to-noise ratio) (dB) / SSIM (image similarity).

[0073] Table 1

[0074]

[0075]

[0076] The algorithm of this invention is compared with existing mainstream dehazing algorithms in terms of PSNR (peak signal-to-noise ratio) (dB) / SSIM (image similarity). The comparison results shown in Table 1 show that the method of this invention can effectively remove haze from the image without color distortion, and can restore richer edge and contour information, making the restored image more realistic and natural.

[0077] The PSNR (dB) and SSIM values ​​are two metrics that reflect the similarity between the restored image and the original image, as well as the structural similarity.

[0078] This invention trains a network model using only hazy images. The proposed network framework consists of an information interaction module and an iteration module. The information interaction module densely connects shallow features of different categories to extract more abstract semantic information. The iteration module employs an enhancement strategy, calculating the dehazing result through multiple iterations, which can significantly reduce output error. Furthermore, to achieve label-free training of the network, a content preservation loss function, a dark channel loss function, a contrast loss function, a saturation loss function, and a sharpening loss function are designed to jointly optimize the network, achieving dehazing from multiple aspects and significantly improving image visual quality.

[0079] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the embodiments of the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A visual quality-driven unsupervised dehazing system that directly generates hazy images by constructing an end-to-end network to learn the mapping relationship between hazy and hazy images; characterized in that, The system consists of an information interaction module and an iteration module, wherein: The information interaction module is used to achieve efficient fusion of shallow input features, complete the interaction of detailed information and structural information, and learn to mine deep semantic information. The iterative module is used to process intermediate variables in the atmospheric scattering model. It is estimated that... The values ​​are substituted into the reconstructed atmospheric scattering model to calculate the haze-free image. The dehazing results are refined through multiple iterations to preserve image details. The system includes the following specific steps: Step 1: Obtain the total training set and test set. The training dataset consists of the indoor training set ITS and the outdoor training set OTS, and the test set consists of the comprehensive target test set SOTS and the mixed subjective test set HSTS. Step 2: Construct an unsupervised dehazing network, which includes an information interaction module and an iteration module; In the information interaction module, input features and Perform convolution operations to generate features that contain information from different categories. Subsequently, feature fusion is performed on each pair of information features. After each convolutional layer, a ReLU activation function is used for nonlinear transformation. Through multiple information interaction processes and feature re-extraction, high-level semantic information in the image is learned. In the iterative module, the output of the information interaction module is used as the input of the iterative module. First, a 3×3 convolutional layer is used to extract features from the input features. estimate; Define the overall loss function during network training. This includes content preservation loss functions, dark channel loss functions, contrast loss functions, saturation loss functions, and sharpening loss functions to achieve unlabeled training of the network; Total loss function The expression is as follows: ； in, For content preservation loss function, The dark channel loss function is... Let be the contrast loss function. The saturation loss function is... This is the sharpening loss function; Step 3: Using the constructed unsupervised dehazing network, dehaze a single image and use it as an intermediate variable in the atmospheric scattering model. Estimate the intermediate variables of the atmospheric scattering model. The values ​​are substituted into the reconstructed atmospheric scattering model to calculate the haze-free image. The haze-free result is refined through multiple iterations to preserve image details.

2. The unsupervised dehazing system based on visual quality drive as described in claim 1, characterized in that, The iterative module estimates An enhancement strategy was employed during the evaluation process. The process of applying the enhancement strategy to the image is as follows: ； ； in, For the first input image Pixels The result of the first defogging attempt. For the first The result of the next iteration of defogging.

3. The unsupervised dehazing system based on visual quality drive as described in claim 2, characterized in that, By analyzing input features and The feature maps are obtained by stacking, and then abstract features that are more relevant to dehazing are extracted from the stacked feature maps through convolution operations. .

4. The unsupervised dehazing system based on visual quality drive as described in claim 1, characterized in that, In the information interaction module, when performing feature pre-extraction on the input foggy image, convolutional layers with receptive fields of different sizes are used to extract shallow features that are more relevant to dehazing from the input image. By setting the convolutional kernel size, different receptive fields are obtained to capture features at different scales and comprehensively extract the detailed and structural information of the image.

5. The unsupervised dehazing system based on visual quality drive as described in claim 1, characterized in that, in: Content Preservation Loss Function Represented as: ； in, For the restored first A clear image of every single pixel. For the first A foggy image with individual pixels. This represents the total number of pixels in the image. Dark channel loss function Represented as: ； in, It is a three-dimensional max pooling function. For the input foggy image, For a size and Identical images with all pixel values ​​being 0; Contrast loss function Represented as: ； in, For the first The contrast of a foggy image at each pixel level. To restore the contrast of the image, This represents the total number of pixels in the image. Saturation loss function Represented as: ； ； ； in, and The saturation values ​​are for the hazy image and the restored image, respectively. For the first The position of each pixel For images with fog, To restore the image, It refers to one of the three channels in the RGB spectrum. A clear image of the restored channel q. A foggy image for channel q; Sharpening loss function Represented as: ； in, To restore the image, For images with fog, The values ​​obtained by performing the Sobel operator operation on the image. The number of pixels.

6. The unsupervised dehazing system based on visual quality drive as described in claim 1, characterized in that, The intermediate variables of the atmospheric scattering model The expression is as follows: ； in, Image of smog. Atmospheric light value, It is a constant. For the transmission matrix, These are intermediate variables in the atmospheric scattering model.

Images