Image defogging method based on atmospheric scattering model

Abstract

The invention belongs to the technical field of computer vision, and particularly relates to an image defogging method based on an atmospheric scattering model. Comprising the following steps: acquiring an original space fog image and inputting the original space fog image into a trained image defogging model for processing to obtain a defogged clear image; the processing process of the image defogging model on the original space fog image comprises the following steps of: processing the original space fog image by adopting a feature extraction module to obtain a first feature space fog image, a second feature space fog image and a third feature space fog image; inputting the feature space fog image into a haze extraction unit for processing to obtain a haze feature image; a feature decoding module is adopted to process the haze feature graph according to the first feature space fog graph, the second feature space fog graph and the third feature space fog graph, and a feature fusion graph is obtained; carrying out atmospheric scattering model inverse operation on the feature fusion image to obtain a defogged clear image; according to the method, the model defogging precision is remarkably improved, a good defogging effect is achieved, and the image defogging precision is effectively improved.

Description

Technical Field

[0001] The invention belongs to the technical field of computer vision, and in particular relates to an image defogging method based on an atmospheric scattering model. Background Art

[0002] Image dehazing technology is an important means to solve problems such as image blur, contrast reduction and color distortion caused by suspended particles in the atmosphere. Image clarity and quality are crucial in areas such as traffic monitoring, unmanned driving, smart homes, and medical imaging. However, environmental factors such as haze often cause the image quality acquired by the imaging system to drop significantly, which not only affects the visual effect, but also reduces the accuracy of subsequent computer vision tasks.

[0003] Traditional image dehazing algorithms are mostly based on the atmospheric scattering model, which assumes that light scattering and attenuation caused by haze are the main reasons for image quality degradation. Early dehazing methods mainly relied on image enhancement techniques, such as histogram equalization, frequency domain transformation, and dehazing methods based on prior knowledge (such as dark channel prior). Although these methods are simple and effective, they often fail under thick fog conditions, and are prone to problems such as color distortion and over-enhancement, making them difficult to apply to complex real scenes. In recent years, the rapid development of deep learning technology has provided new solutions for image dehazing. Dehazing methods based on convolutional neural networks can learn nonlinear mappings through large-scale data and directly generate clear images from haze images.

[0004] Therefore, how to combine the physical principles of the atmospheric scattering model with the deep learning network to develop a defogging algorithm with physical interpretation has become an important research direction in the current image defogging field. This method can not only improve the defogging effect, but also reduce error accumulation, and improve the robustness and wide applicability of the model. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention proposes an image defogging method based on an atmospheric scattering model, the method comprising: obtaining an original spatial fog image and inputting the image into a trained image defogging model for processing to obtain a clear image after defogging;

[0006] The image defogging model processes the original spatial fog image in the following way:

[0007] S1: Processing the original spatial fog image using a feature extraction module to obtain a first feature spatial fog image, a second feature spatial fog image, and a third feature spatial fog image;

[0008] S2: Input the feature space fog image into the haze extraction unit for processing to obtain a haze feature image;

[0009] S3: using a feature decoding module to process the haze feature map according to the first feature space fog map, the second feature space fog map and the third feature space fog map to obtain a feature fusion map;

[0010] S4: Perform an inverse operation of the atmospheric scattering model on the feature fusion image to obtain a clear image after defogging.

[0011] Preferably, the feature extraction module includes a basic convolution block, a first MSG convolution module, a first downsampling module, a second MSG convolution module, a second downsampling module and a third MSG convolution module.

[0012] Furthermore, the structures of the three MSG convolution modules are the same. The data processing of the MSG convolution module includes: the input feature map is batch normalized and then input into two branches for processing; the first branch is composed of an adaptive convolution layer, an instance normalization layer, a ReLU layer, and a depth convolution layer; the second branch is obtained by replacing the adaptive convolution layer of the first branch with a point convolution layer; after the outputs of the two branches are fused, they are passed through a point convolution layer to obtain the output of the MSG convolution module, namely the feature space fog map.

[0013] Preferably, the haze extraction unit includes an atmospheric light extraction branch and a transmittance extraction branch; the third feature space fog map is respectively input into the atmospheric light extraction branch and the transmittance extraction branch for processing to obtain two branch outputs; the outputs of the two branches are subjected to pixel-level operations to obtain the output of the haze extraction unit, i.e., the haze feature map.

[0014] Furthermore, the atmospheric light extraction branch is composed of global average pooling, a first ordinary convolution block, a first nonlinear activation function, a second ordinary convolution block and a second nonlinear activation function; the transmittance extraction branch is composed of a third ordinary convolution block, a fourth ordinary convolution block, a third nonlinear activation function, a fifth ordinary convolution block and a fourth nonlinear activation function.

[0015] Preferably, the feature decoding module includes a first upsampling module, a first SK Fusion module, a second upsampling module, a second SK Fusion module, a third upsampling module, a third SK Fusion module and a sixth ordinary convolution block; wherein, the input of the first SK Fusion module is the third feature space fog image and the output of the first upsampling module, the input of the second SK Fusion module is the second feature space fog image and the output of the second upsampling module, and the input of the third SK Fusion module is the first feature space fog image and the output of the third upsampling module.

[0016] Furthermore, the structures of the three SK Fusion modules are the same. The data processing process of the SK Fusion module includes: adding the two input feature maps and inputting them into the global average pooling layer, the point convolution layer, and the nonlinear function layer; inputting the output of the nonlinear function layer into the two point convolution layers respectively, and the output of the two point convolution layers is then processed by the nonlinear function layer to obtain the output of the SK Fusion module.

[0017] Preferably, the loss function in the image dehazing model training process is the sum of L1 loss, mean square error loss and structural similarity loss.

[0018] The beneficial effects of the present invention are:

[0019] 1. The defogging method provided by the present invention significantly improves the model defogging accuracy through three stages of image processing, achieves good defogging effect, effectively improves the accuracy of image defogging, and is more generalized in cross-dataset testing.

[0020] 2. The MSG module used in the first part of the network model improves the performance and adaptability of the model, reduces information loss, and maintains a low computational cost through multi-scale feature extraction and information fusion.

[0021] 3. The SK Fusion module used in the third part of the network model provides an effective strategy to enhance the expressiveness and robustness of the model through adaptive feature selection and multi-scale fusion, making it perform better in various visual tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is the overall network structure diagram of the image defogging model in the present invention;

[0023] Figure 2 It is a structural diagram of the feature extraction module in the present invention;

[0024] Figure 3 This is a network structure diagram of the MSG module in the present invention;

[0025] Figure 4 This is a structural diagram of the haze extraction unit in the present invention;

[0026] Figure 5 This is a structural diagram of the feature decoding module in the present invention;

[0027] Figure 6 This is the network structure diagram of the SKFusion module in the present invention;

[0028] Figure 7 This is a comparison chart of the image defogging effect in the present invention;

[0029] Figure 8The comparison results of the present invention and the comparison method on different data sets are shown in FIG. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] The present invention proposes an image defogging method based on an atmospheric scattering model. The method is a learning method based on the atmospheric scattering model. By predicting the features corresponding to the atmospheric light and the transmission map, the physical characteristics of the two factors are considered at the same time to improve the defogging accuracy. At the same time, the idea of ​​multi-scale gated convolution is adopted to further improve the network's defogging performance for images, and the SK fusion block is used to adaptively fuse different feature maps. The method includes: obtaining the original spatial fog image and inputting it into the trained image defogging model for processing to obtain a clear image after defogging;

[0032] like Figure 1 As shown in the figure, the image defogging model processes the original spatial fog image in the following way:

[0033] S1: Use a feature extraction module to process the original spatial fog image to obtain a first feature spatial fog image, a second feature spatial fog image, and a third feature spatial fog image.

[0034] The original spatial fog image is cropped into an image with a length and width of 256 pixels and then input into the feature extraction module for processing. The image defogging model proposed in the present invention includes a feature extraction module, a haze extraction unit, a feature decoding module and an atmospheric scattering model inverse operation unit. Figure 2 As shown, the input layer and the output layer of the feature extraction module are sequentially connected by a basic convolution block, a first MSG convolution module, a first down-sampling module, a second MSG convolution module, a second down-sampling module and a third MSG convolution module.

[0035] The structures of the three MSG convolution modules are the same, and the outputs of the three MSG convolution modules are the first feature space fog image, the second feature space fog image, and the third feature space fog image. Figure 3 As shown in the figure, the data processing of the MSG convolution module includes: the input feature map is batch normalized and then input into two branches for processing; the first branch is composed of an adaptive convolution layer, an instance normalization layer, a ReLU layer, and a depth convolution layer; the second branch is obtained by replacing the adaptive convolution layer of the first branch with a point convolution layer; after the outputs of the two branches are fused, they are passed through a point convolution layer to obtain the output of the MSG convolution module, namely the feature space fog map.

[0036] S2: Input the feature space fog map into the haze extraction unit for processing to obtain a haze feature map.

[0037] like Figure 4 As shown, the haze extraction unit includes an atmospheric light extraction branch and a transmittance extraction branch; the third feature space fog map is respectively input into the atmospheric light extraction branch and the transmittance extraction branch for processing to obtain two branch outputs; the outputs of the two branches are subjected to pixel-level operations to obtain the output of the haze extraction unit, i.e., the haze feature map.

[0038] The atmospheric light extraction branch is composed of global average pooling, the first ordinary convolution block, the first nonlinear activation function, the second ordinary convolution block and the second nonlinear activation function; the transmittance extraction branch is composed of the third ordinary convolution block, the fourth ordinary convolution block, the third nonlinear activation function, the fifth ordinary convolution block and the fourth nonlinear activation function.

[0039] S3: Using a feature decoding module to process the haze feature map according to the first feature space fog map, the second feature space fog map and the third feature space fog map to obtain a feature fusion map.

[0040] like Figure 5 As shown, between the input layer and the output layer of the feature decoding module are: the first upsampling module, the first SKFusion module, the second upsampling module, the second SK Fusion module, the third upsampling module, the third SK Fusion module and the sixth ordinary convolution block.

[0041] The three SK Fusion modules have the same structure, wherein the input of the first SK Fusion module is the third feature space fog image and the output of the first upsampling module, the input of the second SK Fusion module is the second feature space fog image and the output of the second upsampling module, and the input of the third SK Fusion module is the first feature space fog image and the output of the third upsampling module.

[0042] like Figure 6 As shown in the figure, the data processing process of the SK Fusion module includes: adding the two input feature maps and inputting them into the global average pooling layer, the point convolution layer, and the nonlinear function layer; inputting the output of the nonlinear function layer, i.e., the ReLU activation function, into the two point convolution layers respectively, and the output of the two point convolution layers is then processed by the nonlinear function layer, i.e., the Softmax function, to obtain the output of the SK Fusion module.

[0043] S4: Perform an inverse operation of the atmospheric scattering model on the feature fusion image to obtain a clear image after defogging.

[0044] The atmospheric scattering model is expressed as:

[0045] I(x)=t(x)J(x)+(1-t(x))A

[0046] Among them, I(x) represents the pixel intensity at the position x of the foggy image, J(x) represents the pixel intensity at the position x of the clear image, t(x) represents the transmission map corresponding to the position x of the foggy image, and A represents the global atmospheric light.

[0047] Therefore, the inverse operation of the atmospheric scattering model is:

[0048]

[0049] Substitute the feature fusion map as I(x) into the above formula for calculation to obtain a clear image after dehazing.

[0050] When training the image dehazing model, the RESIDE public data set can be used. For the ITS data set, part of the data is randomly divided for training, and the other part of the data is used for verification. For the OTS data set, part of the data is randomly divided for training, and the other part of the data is used for verification. The loss function used in the training process of the present invention is the sum of L1 loss, mean square error loss and structural similarity loss.

[0051] L1 loss: used to calculate the L1 loss between the model output and the target image.

[0052]

[0053] Where n is the number of training samples, y i is the true value (target value), is the predicted value of the model.

[0054] Mean square error loss: measures the difference between the predicted value and the true value. The smaller the MSE value, the more accurate the model's prediction.

[0055]

[0056] Structural Similarity Loss: The value of SSIM is between -1 and 1. The closer the value is to 1, the more similar the two images are.

[0057]

[0058] where μ x and μ y represents the pixel sample mean of x and y, and represents the variance of x and y, μ xy represents the covariance of x and y, c1 and c2 represent two constants.

[0059] Evaluation of the present invention:

[0060] like Figure 7 As shown, the fog map ( Figure 7 Left) Input the trained model, and the network model generates a dehazed image ( Figure 7 Right). It can be seen that the defogging method based on the atmospheric scattering model of the present invention has achieved a good defogging effect; and the MSG module used in the present invention improves the performance and adaptability of the model through multi-scale feature extraction and information fusion, reduces information loss, and maintains a low computing cost. The SK Fusion module uses adaptive feature selection and multi-scale fusion to provide an effective strategy to enhance the expression ability and robustness of the model, so that it performs better in various visual tasks.

[0061] The dehazing method of the present invention is compared with two existing model structures, Dehazeformer and gUNet. The Dehazeformer model is detailed in the literature Y.Song, Z.He, H.Qian, X.Du, Vision transformers for single image dehazing, IEEE Trans.Image Process.32(2023)1927–1941. The gUNet model is detailed in the literature Y.Song, Y.Zhou, H.Qian, X.Du, Rethinking performance gains in image dehazing networks, 2022. The comparison results are shown in Figure 8 . The two network structures of the model structure Dehazeformer and gUNet use a separate transmission map estimation network and an atmospheric light estimation network to obtain the estimated values ​​of the intermediate variables, and then use the inverse transformation of the atmospheric scattering model to obtain the dehazed image. The difference compared with the present invention is that the present invention extracts the fog map separately. PSNR (peak signal-to-noise ratio) and SSIM (structural similarity) are used to compare the dehazing accuracy of the model. The higher the values ​​of these two indicators, the better the dehazing accuracy. Figure 8 As shown, the results for different data sets (SOTS-indoor, SOTS-outdoor) show that the experimental effect of the present invention is better, highlighting that the two-stage model structure effectively improves the accuracy of the model.

[0062] In summary, the present invention significantly improves the model defogging accuracy through three stages of image processing, achieves good defogging effect, and effectively improves the image defogging accuracy.

[0063] The above embodiments further illustrate the purpose, technical solutions and advantages of the present invention in detail. It should be understood that the above embodiments are only preferred implementation modes of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made to the present invention within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An image defogging method based on an atmospheric scattering model, characterized in that: include: Obtain the original spatial fog image and input it into the trained image defogging model for processing to obtain a clear image after defogging; The image defogging model processes the original spatial fog image in the following way: S1: Processing the original spatial fog image using a feature extraction module to obtain a first feature spatial fog image, a second feature spatial fog image, and a third feature spatial fog image; S2: Input the feature space fog image into the haze extraction unit for processing to obtain a haze feature image; S3: using a feature decoding module to process the haze feature map according to the first feature space fog map, the second feature space fog map and the third feature space fog map to obtain a feature fusion map; S4: Perform an inverse operation of the atmospheric scattering model on the feature fusion image to obtain a clear image after defogging.

2. The image defogging method based on the atmospheric scattering model according to claim 1, characterized in that: The feature extraction module includes a basic convolution block, a first MSG convolution module, a first down-sampling module, a second MSG convolution module, a second down-sampling module and a third MSG convolution module.

3. The image defogging method based on the atmospheric scattering model according to claim 2, characterized in that: The three MSG convolution modules have the same structure. The data processing of the MSG convolution module includes: the input feature map is batch normalized and then input into two branches for processing; the first branch is composed of an adaptive convolution layer, an instance normalization layer, a ReLU layer, and a depth convolution layer; the second branch is obtained by replacing the adaptive convolution layer of the first branch with a point convolution layer; after the outputs of the two branches are fused, they are passed through a point convolution layer to obtain the output of the MSG convolution module, namely the feature space fog map.

4. The image defogging method based on the atmospheric scattering model according to claim 1, characterized in that: The haze extraction unit includes an atmospheric light extraction branch and a transmittance extraction branch; The third feature space fog map is input into the atmospheric light extraction branch and the transmittance extraction branch for processing, respectively, to obtain two branch outputs; pixel-level operations are performed on the outputs of the two branches to obtain the output of the haze extraction unit, namely the haze feature map.

5. The image defogging method based on the atmospheric scattering model according to claim 4, characterized in that: The atmospheric light extraction branch is composed of global average pooling, the first ordinary convolution block, the first nonlinear activation function, the second ordinary convolution block and the second nonlinear activation function; the transmittance extraction branch is composed of the third ordinary convolution block, the fourth ordinary convolution block, the third nonlinear activation function, the fifth ordinary convolution block and the fourth nonlinear activation function.

6. The image defogging method based on the atmospheric scattering model according to claim 1, characterized in that: The feature decoding module includes a first upsampling module, a first SK Fusion module, a second upsampling module, a second SK Fusion module, a third upsampling module, a third SK Fusion module and a sixth ordinary convolution block; wherein, the input of the first SK Fusion module is the third feature space fog image and the output of the first upsampling module, the input of the second SK Fusion module is the second feature space fog image and the output of the second upsampling module, and the input of the third SK Fusion module is the first feature space fog image and the output of the third upsampling module.

7. The image defogging method based on the atmospheric scattering model according to claim 6, characterized in that: The three SKFusion modules have the same structure. The data processing process of the SK Fusion module includes: adding the two input feature maps and inputting them into the global average pooling layer, the point convolution layer, and the nonlinear function layer; inputting the output of the nonlinear function layer into the two point convolution layers respectively, and the output of the two point convolution layers is then processed by the nonlinear function layer to obtain the output of the SK Fusion module.

8. The image defogging method based on the atmospheric scattering model according to claim 1, characterized in that: The loss function in the image dehazing model training process is the sum of L1 loss, mean square error loss and structural similarity loss.

Images