A semi-supervised dehazing method based on fusion of depth map and feature mask

Abstract

The invention belongs to the field of image restoration, and particularly relates to a semi-supervised defogging method based on the fusion of a depth map and a feature mask, comprising obtaining a depth map and processing the map through two cascaded basic defogging modules to obtain depth information; obtaining original image features and inputting the map into an autoencoder to obtain coarse defogging features; projecting the coarse defogging features through a separator to obtain a coarse defogging image; adding the coarse defogging image to the depth information and then inputting the map into a feature mask generation module to obtain a depth feature mask, which is multiplied with the coarse defogging image to obtain a refined defogging image; splicing haze distribution features and original image features according to channel dimensions and inputting the map into a noise image extraction module to output a noise image; inputting the noise image and the refined defogging image into an adaptive channel attention layer to obtain a haze-free image; the invention can better restore image details and suppress noise, and obtain a visually cleaner haze-free image.

Description

Technical Field

[0001] The present invention belongs to the field of image restoration, and in particular relates to a semi-supervised defogging method based on fusion of a depth map and a feature mask. Background Art

[0002] Vision is the main way for humans to perceive the world and acquire information, and images are the information carriers of the visual path. Compared with text, audio and other carriers, they can contain richer information. With the development of science and technology, various image acquisition devices have been widely used in social life. Under severe weather conditions, image and video sensing equipment often finds it difficult to acquire high-quality image information, and the images will have color distortion, blurred features, reduced contrast and other visual quality degradation. These degraded images are not conducive to subsequent advanced visual tasks in many existing computer vision technologies. Therefore, designing effective algorithms to process images polluted by haze and restore the original details and structural texture of the images has become a hot topic for researchers at home and abroad.

[0003] In foggy weather, before the reflected light from an object reaches the image acquisition device, these particles will absorb and scatter the light reflected from the scene, causing the quality of the captured image to degrade, which is not conducive to subsequent advanced visual tasks.

[0004] In recent years, many researchers have studied foggy images and defogging algorithms from multiple directions, and many defogging algorithms have emerged. According to the number of input images, it can be divided into single-image or multi-image defogging algorithms. It is difficult to obtain data for multi-image defogging algorithms, so there are few practical applications. Single-image defogging algorithms are widely used. From the perspective of image processing principles, single-image defogging algorithms can be divided into two categories: defogging algorithms based on physical models and defogging algorithms based on non-physical models. Among them, algorithms based on physical models can be divided into: based on depth information, based on prior information, based on partial differential equations, and based on deep learning; non-physical model defogging algorithms can be divided into methods based on image enhancement and deep learning.

[0005] Methods based on physical prior knowledge can perform well when the prior assumptions are met. However, the strong reliance on priors makes these methods unreliable when the priors are not true. Data-driven methods perform better than traditional physical prior methods, but these methods require a large number of training samples to represent haze scenes under different outdoor conditions, and building large-scale real data sets is expensive and time-consuming. At present, artificially synthesized haze images do not consider the impact of air particles on light of different wavelengths and the impact of wind on imaging, resulting in differences in domain distribution between synthetic images and real images, which makes it difficult to apply models trained based on synthetic data sets to real haze scenes. Existing data-driven defogging methods do not make good use of the scene depth information contained in the image, and their performance on real foggy images is not satisfactory. The foggy images processed by them have residual fog and incomplete defogging in dense fog areas and large depth of field areas. Summary of the invention

[0006] The present invention proposes a semi-supervised defogging method based on fusion of depth map and feature mask, which specifically includes the following steps:

[0007] The depth map of the image is obtained through the depth prediction module, and the depth map is processed through two cascaded basic defogging modules to obtain depth information;

[0008] The image is subjected to feature extraction by the feature extraction module to obtain the original image features, which are then input into the autoencoder composed of the basic defogging module to obtain the coarse defogging features;

[0009] The rough dehazed features are mapped back to the RGB space through the separator, and the haze distribution features are separated to obtain the rough dehazed image and haze distribution features;

[0010] The coarse dehazed image is added to the depth information and then input into the feature mask generation module composed of the channel attention and pixel attention cascade to obtain the depth feature mask;

[0011] Multiply the deep feature mask with the rough dehazed image to obtain a refined dehazed image;

[0012] The haze distribution features and the original image features are concatenated according to the channel dimension and input into the noise image extraction module to output the noise image;

[0013] A defogging layer consisting of four cascaded basic defogging modules is constructed. The noisy image is used as the input of each basic defogging module, and the refined defogging image is used as the input of the first basic defogging module. The image output by the last basic defogging module is the haze-free image.

[0014] Furthermore, the feature extraction module is a pre-trained VGG-16 network.

[0015] Furthermore, the features input to the basic dehazing module are processed by a cascaded 3×3 convolutional layer, a ReLU activation layer, and a 3×3 convolutional layer, and then input to a multiplier and multiplied by the input features as the output of the basic dehazing module.

[0016] Furthermore, the features of the input channel attention are processed by the cascaded average pooling layer, 1×1 convolution layer, ReLU activation layer, 1×1 convolution layer, and Sigmoid activation function in sequence, and then multiplied by the input features of the input multiplier as the output of the input channel attention.

[0017] Furthermore, the features of the input pixel attention are processed by a cascaded 1×1 convolution layer, a ReLU activation layer, a 1×1 convolution layer, and a Sigmoid activation function, and then input to the multiplier and multiplied by the input features as the output of the pixel attention.

[0018] Furthermore, the feature f of the input separator is processed by the cascaded 3×3 convolution layer, ReLU activation layer, 3×3 convolution layer, ReLU activation layer, 1×1 convolution layer, Tanh activation layer, 1×1 convolution layer, and Sigmoid activation function in sequence, and the feature m obtained by the input multiplier is multiplied by the feature f of the input separator to obtain a coarse dehazed image, and the haze distribution feature is obtained by multiplying (1-m) with the feature f of the input separator.

[0019] Furthermore, the noise image extraction module consists of a cascaded basic dehazing module, channel attention and pixel attention.

[0020] Furthermore, in the training process of the defogging network composed of a depth prediction module, a basic defogging module, a feature extraction module, channel attention, familiarity attention and a separator, the training sets used include real fog images and synthetic fog images. The supervised loss is calculated based on the rough defogging image obtained based on the synthetic fog image as input, and the loss is back-propagated to update the defogging network; the unsupervised loss is calculated based on the real fog image and the defogging image output by the defogging network, and the loss is back-propagated to update the defogging network.

[0021] Furthermore, the supervised loss is expressed as:

[0022]

[0023]

[0024]

[0025] Among them, λ1 and λ2 are weight coefficients; N b is the batch size chosen during training; J i represents the i-th standard haze-free image in a training batch; I irepresents the i-th batch of original foggy images; N(I i ) represents the dehazed image obtained after the i-th original foggy image is processed by the dehazing network; φ() represents the pre-trained VGG-16 model; ||·||2 represents the standard L2 norm.

[0026] Furthermore, the unsupervised loss is expressed as:

[0027]

[0028]

[0029]

[0030] Among them, λ3 and λ4 are weight coefficients; are the horizontal and vertical gradient operators, respectively, represents the horizontal gradient of the i-th channel of the image, Represents the vertical gradient of the i-th channel of the image; N u is the batch size set during training, i.e. Batch-Size; D(J i ) represents image J i Dark channel map of ; ||·||1 represents the standard L1 norm.

[0031] The present invention aims to solve the problems of overall color deviation and color distortion caused by mutual amplification of estimation parameter errors and failure of prior conditions in traditional physical prior defogging algorithms, and is suitable for most scenarios and is more robust. In view of the performance gap between real images and synthetic haze images, and the poor effect on real foggy images, existing learning-based methods use a framework for joint training of data sets distributed in different domains and a semi-supervised loss function to improve the existing methods. In view of the problem that current defogging algorithms cannot completely defog dense fog areas and areas with large depth of field, scene depth estimation is combined with a non-local attention mechanism to form an attention mask to achieve step-by-step defogging, which can better restore image details and suppress noise, and obtain visually cleaner fog-free images. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is a schematic diagram of the semi-supervised learning framework used in the present invention;

[0033] Figure 2 This is a structural block diagram of a semi-supervised defogging method based on the fusion of depth map and feature mask of the present invention. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] The present invention proposes a semi-supervised defogging method based on fusion of depth map and feature mask, which specifically includes the following steps:

[0036] The depth map of the image is obtained through the depth prediction module, and the depth map is processed through two cascaded basic defogging modules to obtain depth information;

[0037] The image is subjected to feature extraction by the feature extraction module to obtain the original image features, which are then input into the autoencoder composed of the basic defogging module to obtain the coarse defogging features;

[0038] The rough dehazed features are mapped back to the RGB space through the separator, and the haze distribution features are separated to obtain the rough dehazed image and haze distribution features;

[0039] The coarse dehazed image is added to the depth information and then input into the feature mask generation module composed of the channel attention and pixel attention cascade to obtain the depth feature mask;

[0040] Multiply the deep feature mask with the rough dehazed image to obtain a refined dehazed image;

[0041] The haze distribution features and the original image features are concatenated according to the channel dimension and input into the noise image extraction module to output the noise image;

[0042] A defogging layer consisting of four cascaded basic defogging modules is constructed. The noisy image is used as the input of each basic defogging module, and the refined defogging image is used as the input of the first basic defogging module. The image output by the last basic defogging module is the haze-free image.

[0043] In this embodiment, in order to obtain a more robust defogging image, this embodiment is improved mainly in the following three aspects:

[0044] (1) The fog distribution mask is derived based on the image depth information and combined with dual attention to extract the residual fog distribution features, giving different weights to different pixel areas;

[0045] (2) A loss function constraint model with good generalization performance is constructed to learn datasets from different domain distributions, improving the model's dehazing ability on real images;

[0046] (3) A semi-supervised training method is used to obtain domain transfer capabilities between the source domain and the target domain and improve generalization performance.

[0047] In a foggy environment, atmospheric particles cause scattering and absorption of light in the process of transmitting from the surface of the scene to the camera. This is because the attenuation of light decays exponentially with the increase in depth of field, which causes the light intensity obtained by the sensor to change unevenly, and then the grayscale value of the image changes. Although the degree of degradation of the effect of foggy images varies, they still contain rich depth information. After obtaining the relative depth of the scene in the image, this embodiment assigns a greater weight to the area with greater depth, so that the model focuses more on defogging in areas with large depth of field. This embodiment divides the process of obtaining a defogged image into two stages: coarse defogging and refined defogging. After the coarse defogging process in the first stage, the residual fog distribution map is extracted from the image, and a mask is generated, so that the second stage network can focus on refined defogging in areas with residual fog.

[0048] In order to improve the defogging performance of the model on real data, it is necessary to introduce real foggy images to participate in the training. The semi-supervised learning method based on supervised and unsupervised alternating iterative training can handle this type of domain adaptation problem very well. Therefore, this embodiment proposes a robust no-reference loss function to act on the unsupervised branch, which can not only play a regularization effect and reduce the model's tendency to overfit the synthetic data set, but also is the key to improving the model's defogging performance on real images.

[0049] Learning-based dehazing methods can map input foggy images to clean images without fog through modeling with the support of a large amount of training data. This data-driven method is constrained by the dataset, and the model trained by artificial synthetic images has limited generalization performance on natural foggy images; semi-supervised learning is used, and the supervised and unsupervised branches share network parameters to improve the dehazing performance of the model on real-world images.

[0050] like Figure 1 The present invention adopts semi-supervised learning that combines supervised learning branches and unsupervised learning branches to perform defogging. During the training process, the original fog-free image is used to synthesize an artificially synthesized foggy image, namely a synthetic fog image. The supervised branch uses a defogging network to defog the artificially synthesized foggy image, calculates the loss between the defogging image and the original fog-free image, and updates the parameters of the defogging network; the unsupervised branch directly uses the real natural foggy image to input the defogging network to obtain the defogging image, calculates the unsupervised loss, and updates the parameters of the defogging network.

[0051] In the supervised branch, the mean square error (MSE) is used to measure the pixel difference between the haze-free image and the dehazed image, which is expressed as:

[0052]

[0053] Among them, N b is the input batch size during supervised training, i.e., Brach-Size during the supervised training phase; J i represents the i-th standard haze-free image in a training batch; N represents the dehazing network, I i represents the i-th original foggy image in a training batch; ||·||2 represents the standard L2 norm.

[0054] In order to retain high-level style features, the perceptual loss function is introduced. By calculating the distance in the feature space, the obtained result has a better visual effect and conforms to the sensory characteristics of humans. The perceptual loss is defined as:

[0055]

[0056] Among them, φ() represents the pre-trained VGG-16 model, which is used to extract image features.

[0057] In summary, the total loss in the supervised phase is defined as:

[0058]

[0059] Among them, λ1 and λ2 are weight coefficients.

[0060] In the unsupervised branch, in order to control the noise of the generated image and eliminate the artifacts that may be caused by image restoration, the total variation loss (TV) is introduced to enable the network to generate spatially smooth images. The loss is expressed as:

[0061]

[0062] in, Represents the horizontal and vertical gradient operators respectively.

[0063] The dark channel prior principle (DC) is an important observation conclusion, which has a high reference value for restoring haze-free images. Introducing it into the unsupervised training stage as a loss function can constrain the network to generate haze-free images that conform to statistical laws. The formula for the dark channel is defined as follows:

[0064]

[0065] Among them, D(J i ) represents the dark channel image of image J, J i c Representative J iThe color channel in , and Ω(x) represents the local block centered at x. L1 regularization is applied to constrain the sparsity of the dark channel of the image, expressed as:

[0066]

[0067] Among them, J i Represents the i-th image in a batch of Batch-Size size, N u represents the Batch-Size set during unsupervised training, and ||·||1 represents the standard L1 norm.

[0068] In summary, the loss function of the unsupervised training phase is defined as:

[0069]

[0070] Among them, λ3 and λ4 are weight coefficients.

[0071] like Figure 2 In this embodiment, two images, namely, synthetic fog image and real fog image, are input during the training process for training. After inputting the defogging network, a feature extractor is first used to extract features from the input image. The feature extractor of this implementation uses a pre-trained VGG-16 network to extract features. The VGG-16 network can use any one of the currently disclosed ones. The VGG-16 network is used as a feature extractor to map the foggy image into the feature space, which is expressed as:

[0072]

[0073] Where F is the feature extractor The feature map obtained; I is the image input to the VGG-16 network, which is a synthetic fog image or a real fog image during the training process. When processing real-time data, that is, in the defogging network that has completed the training, the input is the real fog image to be defogged.

[0074] The feature F is sent to a dehazing network composed of a basic dehazing module (Dehaze Block) to obtain a rough dehazing feature. The input feature of the basic dehazing module is processed by a cascaded 3×3 convolution layer, a ReLU activation layer, and a 3×3 convolution layer, and then multiplied by the input feature of the input multiplier as the output of the basic dehazing module. The process is expressed as:

[0075] B coarse =AE(F)

[0076] Among them, B coarse represents the coarse dehazing feature obtained from the AE (autoencoder) module; then the coarse dehazing feature B coarseThe number of feature channels will be reduced to 3 through a separator S, and the rough dehazed image will be directly obtained. The process is expressed as:

[0077] B, H = S (B coarse )

[0078] At this point, the first stage of coarse dehazing is completed; while the separator directly maps the coarse dehazing features back to the RGB space, it also separates the haze distribution feature H.

[0079] The feature f of the input separator is processed by the cascaded 3×3 convolution layer, ReLU activation layer, 3×3 convolution layer, ReLU activation layer, 1×1 convolution layer, Tanh activation layer, 1×1 convolution layer, and Sigmoid activation function. The feature m obtained by the input multiplier is multiplied by the feature f of the input separator to obtain a rough dehazed image. Let (1-m) be multiplied by the feature f of the input separator to obtain the haze distribution feature.

[0080] In addition, a depth prediction module is used to predict the depth map of the input dehazing network image. The depth map is processed by two cascaded basic dehazing modules to obtain the depth information of the image. The depth information is added to the coarse dehazing image obtained by the separator and then input into the feature mask generation module composed of the channel attention and pixel attention cascade to extract the depth feature mask.

[0081] The input channel attention features are processed by the cascaded average pooling layer, 1×1 convolution layer, ReLU activation layer, 1×1 convolution layer, and Sigmoid activation function, and then multiplied by the input features as the output of the input channel attention.

[0082] The input pixel attention feature is processed by the cascaded 1×1 convolution layer, ReLU activation layer, 1×1 convolution layer, and Sigmoid activation function, and then input to the multiplier and multiplied by the input feature as the output of the pixel attention.

[0083] In the second stage of dehazing, the depth feature mask M obtained by the depth mask branch is multiplied by the rough dehazing feature (Element-wise Multiplication) to obtain the refined dehazing regional feature B′ coarse At the same time, the haze distribution feature H and the original image feature F are spliced ​​according to the channel dimension and input into the noise image extraction module, and the noise image Ar is output as a comparison clue. The process is expressed as:

[0084] Ar=PA(CA(σ(f convv (H, F))))

[0085] Among them, f convrepresents a 3×3 convolutional layer; σ represents the activation function ReLU; the noise image extraction module consists of a cascaded basic dehazing module, channel attention, and pixel attention.

[0086] The noise image Ar and the refined dehazed image B′ coarse The final haze-free image J is obtained after passing through 4 layers of basic defogging modules. The four basic defogging modules are cascaded, the noise image is used as the input of each basic defogging module, the refined defogging image is used as the input of the first basic defogging module, and the image output by the last basic defogging module is the haze-free image.

[0087] Since the supervised branch and the unsupervised branch have common weights and structures, the weights are iteratively updated during the training process. The updating process includes:

[0088] First, two batches of reference images are randomly selected, and the loss value is calculated by the labels and the output dehazed images;

[0089] At the same time, a batch of real foggy images without reference are selected for prediction, and the output fog-free images are directly used to calculate the loss according to the unsupervised loss function;

[0090] Then, the two losses are used to back-propagate and update the network parameters respectively.

[0091] The updated algorithm flow is expressed as:

[0092]

[0093]

[0094] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A semi-supervised dehazing method based on fusion of depth map and feature mask, characterized in that: The specific steps include: The depth map of the image is obtained through the depth prediction module, and the depth map is processed by two cascaded basic defogging modules to obtain depth information; the features input to the basic defogging module are processed by the cascaded 3×3 convolution layer, ReLU activation layer, and 3×3 convolution layer in turn, and then input to the multiplier and multiplied with the input features as the output of the basic defogging module; The image is subjected to feature extraction by the feature extraction module to obtain the original image features, which are then input into the autoencoder composed of the basic defogging module to obtain the coarse defogging features; The rough dehazed features are mapped back to the RGB space through the separator, and the haze distribution features are separated to obtain the rough dehazed image and haze distribution features; The coarse dehazed image is added to the depth information and then input into the feature mask generation module composed of the channel attention and pixel attention cascade to obtain the depth feature mask; Multiply the deep feature mask with the rough dehazed image to obtain a refined dehazed image; The haze distribution features and the original image features are concatenated according to the channel dimension and input into a noise image extraction module consisting of a basic dehazing module, channel attention, and pixel attention, and the noise image is output; A defogging layer consisting of four cascaded basic defogging modules is constructed. The noisy image is used as the input of each basic defogging module, and the refined defogging image is used as the input of the first basic defogging module. The image output by the last basic defogging module is the haze-free image.

2. The semi-supervised defogging method based on fusion of depth map and feature mask according to claim 1, characterized in that: The feature extraction module is a pre-trained VGG-16 network.

3. The semi-supervised defogging method based on fusion of depth map and feature mask according to claim 1, characterized in that: The features of the input channel attention are processed by the cascaded average pooling layer, 1×1 convolution layer, ReLU activation layer, 1×1 convolution layer, and Sigmoid activation function in sequence, and then multiplied by the input features of the input multiplier as the output of the input channel attention.

4. The semi-supervised defogging method based on fusion of depth map and feature mask according to claim 1, characterized in that: The input pixel attention features are processed by the cascaded 1×1 convolution layer, ReLU activation layer, 1×1 convolution layer, and Sigmoid activation function, and then input to the multiplier and multiplied by the input features as the output of the pixel attention.

5. The semi-supervised defogging method based on fusion of depth map and feature mask according to claim 1, characterized in that: The feature f of the input separator is processed by the cascaded 3×3 convolution layer, ReLU activation layer, 3×3 convolution layer, ReLU activation layer, 1×1 convolution layer, Tanh activation layer, 1×1 convolution layer, and Sigmoid activation function. The feature m obtained by the input multiplier is multiplied by the feature f of the input separator to obtain a rough dehazed image. Let (1-m) be multiplied by the feature f of the input separator to obtain the haze distribution feature.

6. The semi-supervised defogging method based on fusion of depth map and feature mask according to claim 1, characterized in that: The noisy image extraction module consists of a cascaded basic dehazing module, channel attention, and pixel attention.

7. The semi-supervised defogging method based on fusion of depth map and feature mask according to claim 1, characterized in that: In the training process of the defogging network composed of a depth prediction module, a basic defogging module, a feature extraction module, a channel attention, a familiarity attention and a separator, the training sets used include real fog images and synthetic fog images. The supervised loss is calculated based on the coarse defogging image obtained based on the synthetic fog image as input, and the defogging network is updated by backpropagation. The unsupervised loss is calculated based on the real fog image and the defogging image output by the defogging network, and the defogging network is updated by backpropagation.

8. The semi-supervised defogging method based on fusion of depth map and feature mask according to claim 7, characterized in that: The supervised loss is expressed as: Among them, λ1 and λ2 are weight coefficients; N b is the batch size chosen during training; J i represents the i-th standard haze-free image output in a training batch; I i represents the i-th original foggy image in a training batch; N(I i ) represents the dehazed image obtained after the i-th original hazy image is processed by the dehazing network; φ( ) represents the pre-trained VGG-16 model; ||·||2 represents the standard L2 norm.

9. The semi-supervised defogging method based on fusion of depth map and feature mask according to claim 7, characterized in that: The unsupervised loss is expressed as: Among them, λ3 and λ4 are weight coefficients; are the horizontal and vertical gradient operators, respectively, represents the horizontal gradient of the i-th image in a training batch, Represents the vertical gradient of the i-th image in a training batch; N u is the batch size set during training, i.e. Batch-Size; D(J i ) represents the dark channel map of the i-th image in a training batch; ||·||1 represents the standard L1 norm.

Images