Image defogging method based on atmospheric scattering model and frequency domain feature extraction

Abstract

The invention discloses an image defogging method based on an atmospheric scattering model and frequency domain feature extraction. The method comprises the following steps: 1) inputting a blurred image # imgabs0 # and a clear image # imgabs1 # into a neural network for training, and outputting a predicted transmittance # imgabs2 #; 2) estimating atmospheric light # imgabs4 # of the blurred image # imgabs3 # according to dark channel prior; 3) calculating clear images # imgabs7 # corresponding to the # imgabs5 # and the # imgabs6 # according to an atmospheric scattering model; 4) updating the parameters of the neural network by using the # imgabs8 # and the input clear image # imgabs9 # through a loss function; 5, the neural network with the updated parameters is used for defogging of a single foggy image.Compared with a traditional single physical model-based method and an end-to-end deep learning method, the defogging task of the single foggy image can be efficiently and rapidly achieved through the designed neural network based on the atmospheric scattering model.

Description

Technical Field

[0001] The present invention relates to the technical field of foggy image defogging, and in particular to an image defogging method based on an atmospheric scattering model and frequency domain feature extraction. Background Art

[0002] Traditional physical model-based image dehazing methods: These methods depend on the atmospheric scattering model, but the atmospheric scattering model depends on relative assumptions and specific target scenes, which leads to reduced robustness in complex actual scenes. The image dehazing methods based on deep learning are different from the previous physical model-based dehazing methods. The deep learning methods are data-driven, usually using neural networks to estimate the transmission map and atmospheric light in the physical scattering model, or directly learning the mapping of blurred to clear images. However, these methods may cause artifacts because inaccurate estimates of both the transmission map and the global atmospheric light lead to cumulative errors. In addition, it is difficult or expensive to collect ground truth of the transmission map and the global atmospheric light in the real world. In addition, there are many problems with a single data-driven neural network. Since haze is usually local and this type of distortion will be mixed in the entire image, forming coupling. In addition, noise belongs to the high-frequency distortion signal that needs to be eliminated. Therefore, image dehazing needs to deal with distortions with local, coupled and frequency-inconsistent characteristics. The traditional deep learning dehazing method extracts features of foggy images in the spatial domain and cannot handle these problems well. Summary of the invention

[0003] The purpose of the present invention is to overcome the shortcomings of the prior art and propose an image defogging method based on an atmospheric scattering model and frequency domain feature extraction. Different from the traditional single physical model-based method and end-to-end deep learning method, the designed neural network based on the atmospheric scattering model can efficiently and quickly achieve the defogging task of a single foggy image.

[0004] To achieve the above object, the present invention adopts the following technical solution:

[0005] An image defogging method based on an atmospheric scattering model and frequency domain feature extraction comprises the following steps:

[0006] Step S1, input the blurred image I(x) and the corresponding clear image J(x) into the neural network for training and output the corresponding predicted transmittance T(x);

[0007] Step S2, estimating the atmospheric light A(x) corresponding to the blurred image I(x) according to the dark channel priori;

[0008] Step S3, calculating the clear image J′(x) corresponding to T(x) and A(x) according to the atmospheric scattering model;

[0009] Step S4: J′(x) and the input clear image J(x) are updated through the loss function (MSE) to update the convolution layer parameters and attention block parameters of the neural network. The calculation formula of the loss function (MSE) is: where Y i is J(x), is J′(x);

[0010] Step S5: The neural network with updated parameters is used to defog a single foggy image, and the defogging effect of the neural network is evaluated by calculating the average peak signal-to-noise ratio and structural similarity.

[0011] Specifically, the predicted transmittance T(x) corresponding to the output in step S1 is expressed as:

[0012] T(x)=e -βd (x) (1)

[0013] In the above formula, β is the scattering coefficient; d(x) is the scene depth.

[0014] Specifically, the atmospheric scattering model in step S3 is expressed as:

[0015] I(x)=J(x)T(x)+(1-T(x))A(x) (2)

[0016] In the above formula, x represents the pixel position; I(x) represents the blurred image; J(x) represents the clear image; T(x) represents the transmission map; A(x) represents the atmospheric light.

[0017] Specifically, since U-Net can better perform feature fusion and retain low-frequency features (such as contours and edges), it is used as the main structure of the neural network, and the neural network includes channel attention and pixel attention.

[0018] Furthermore, the neural network in step S1 is trained and outputs the corresponding predicted transmittance T(x). After downsampling, the neural network converts the feature map to the frequency domain using Fourier transform, and converts the feature map to the spatial domain through inverse Fourier transform during upsampling. In this way, the distortion of frequency-inconsistent features can be processed, and high-frequency noise can be removed more effectively.

[0019] Specifically, the formula for calculating the average peak signal-to-noise ratio in step S5 is:

[0020]

[0021] In the above formula, a is the total number of output images; H and W are the height and width of the image respectively; n is the number of bits per pixel; X is the network output image; and Y is the corresponding clear image.

[0022] Specifically, the formula for calculating the average structural similarity in step S5 is:

[0023]

[0024] In the above formula, a is the total number of output images; μ x , μ y are the pixel averages of image x and image y respectively; x and σ y Represent the pixel standard deviations of image x and image y respectively; c1 and c2 are constants, where c1=0.0001 and c2=0.0009.

[0025] Compared with the prior art, the present invention has the following beneficial effects:

[0026] 1. The method of the present invention can efficiently and quickly realize the defogging task of a single foggy image through the designed neural network based on the atmospheric scattering model, avoiding the problem that a single physical model or data-driven neural network cannot well handle the distortion with local, coupled and frequency inconsistent characteristics.

[0027] 2. The method of the present invention provides important theoretical and technical guidance for image defogging in application fields such as autonomous driving, video surveillance, and military reconnaissance. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 The present invention is a flow chart of an image defogging method based on an atmospheric scattering model and frequency domain feature extraction.

[0029] Figure 2 Schematic diagram of the structure of the neural network of the present invention.

[0030] Figure 3 Schematic diagram of the structure of the neural network attention block of the present invention.

[0031] Figure 4 This is a comparison chart of the clear images output by the foggy image defogging method of the present invention and the traditional method. DETAILED DESCRIPTION

[0032] In order to facilitate the understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. Several embodiments of the present invention are given in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive.

[0033] Example

[0034] In this embodiment, the indoor training set (ITS) and outdoor training set (OTS) in the RESIDE dataset are used as training data. The neural network training environment is a CPU 12th Gen Intel(R) Core(TM) i7-12700KF 3.60GHz, 32GRAM, GPU NVIDIA Geforce RTX 3080, and the program is implemented using Pytorch in python3.9 and windowl1 22000.1696 operating system.

[0035] like Figure 1 As shown, the present invention provides an image defogging method based on an atmospheric scattering model and frequency domain feature extraction, comprising the following steps:

[0036] Step S1, input the blurred image I(x) and the corresponding clear image J(x) into the neural network for training and output the corresponding predicted transmittance T(x);

[0037] Step S2, estimating the atmospheric light A(x) corresponding to the blurred image I(x) according to the dark channel priori;

[0038] Step S3, calculating the clear image J′(x) corresponding to T(x) and A(x) according to the atmospheric scattering model;

[0039] Step S4: J′(x) and the input clear image J(x) are updated through (MSE) to update the convolutional layer parameters and attention block parameters of the neural network. The calculation formula of the loss function (MSE) is: where Y i is J(x), is J′(x);

[0040] Step S5: The neural network with updated parameters is used to defog a single foggy image, and the defogging effect of the neural network is evaluated by calculating the average peak signal-to-noise ratio and structural similarity.

[0041] Specifically, the predicted transmittance T(x) corresponding to the output in step S1 is expressed as:

[0042] T(x)=e -βd (x) (1)

[0043] In the above formula, β is the scattering coefficient; d(x) is the scene depth.

[0044] Specifically, the atmospheric scattering model in step S3 is expressed as:

[0045] I(x)=J(x)T(x)+(1-T(x))A(x) (2)

[0046] In the above formula, x represents the pixel position; I(x) represents the blurred image; J(x) represents the clear image; T(x) represents the transmission map; A(x) represents the atmospheric light.

[0047] Specifically, Figure 2 As shown in FIG, since U-Net can better perform feature fusion and retain low-frequency features (such as contours and edges), it is used as the main structure of the neural network, as shown in FIG. Figure 3 As shown, the attention block structure of the neural network includes channel attention and pixel attention.

[0048] Furthermore, the neural network in step S1 is trained and outputs the corresponding predicted transmittance T(x). After downsampling, the neural network converts the feature map to the frequency domain using Fourier transform, and converts the feature map to the spatial domain through inverse Fourier transform during upsampling. In this way, the distortion of frequency-inconsistent features can be processed, and high-frequency noise can be removed more effectively.

[0049] Specifically, the formula for calculating the average peak signal-to-noise ratio in step S5 is:

[0050]

[0051] In the above formula, a is the total number of output images; H and W are the height and width of the image respectively; n is the number of bits per pixel; X is the network output image; and Y is the corresponding clear image.

[0052] Specifically, the formula for calculating the average structural similarity in step S5 is:

[0053]

[0054] In the above formula, a is the total number of output images; μ x , μ y are the pixel averages of image x and image y respectively; x and σ y Represent the pixel standard deviations of image x and image y respectively; c1 and c2 are constants, where c1=0.0001 and c2=0.0009.

[0055] The average peak signal-to-noise ratio (PSNR) and average structural similarity (SSIM) between the output results of the foggy image defogging method of the present invention and the traditional method and the clear image are calculated, and the calculation results are shown in Table 1 below.

[0056] Table 1. PSNR and SSIM calculation results of the method of the present invention and the traditional method

[0057]

[0058] like Figure 4 The figure shows the comparison of the clear image output by the foggy image defogging method of the present invention and the traditional method. Figure 4 It can be seen that the dehazing method based on the traditional physical model has low robustness in actual situations, cannot restore the sky area well, and the image becomes darker as a whole. The comparison between the U-net output results and the clear image in Table 1 shows that the output color of a single neural network is distorted and accompanied by artifacts; after adding the attention mechanism, the color distortion and artifacts can be better solved, but at the same time, some pixels have abnormal colors; after adding the attention mechanism and performing frequency domain feature extraction to U-net, the structural similarity and peak signal-to-noise ratio of the image are further improved, and the problem of abnormal pixel color can also be solved, which reflects the advantages of the attention mechanism and the conversion of feature maps into frequency domain feature extraction.

[0059] It should be understood that the above description of the preferred embodiment is relatively detailed and cannot be regarded as limiting the scope of patent protection of the present invention. Under the enlightenment of the present invention, ordinary technicians in this field can also make substitutions or modifications without departing from the scope of protection of the claims of the present invention, which all fall within the scope of protection of the present invention. The scope of protection requested for the present invention shall be based on the attached claims.

Claims

1. An image defogging method based on atmospheric scattering model and frequency domain feature extraction, characterized in that: The following steps are involved: Step S1, input the blurred image I(x) and the corresponding clear image J(x) into the neural network for training and output the corresponding predicted transmittance T(x); Step S2, estimating the atmospheric light A(x) corresponding to the blurred image I(x) according to the dark channel priori; Step S3, calculating the clear image J'(x) corresponding to T(x) and A(x) according to the atmospheric scattering model; Step S4: Update the convolutional layer parameters and attention block parameters of the neural network by using the loss function MSE to compare J'(x) with the input clear image J(x). The calculation formula of the loss function MSE is: MSE = where Yi is J(x), For J' (x) ; Step S5: The neural network with updated parameters is used to defog a single foggy image, and the defogging effect of the neural network is evaluated by calculating the average peak signal-to-noise ratio and structural similarity.

2. The image defogging method based on atmospheric scattering model and frequency domain feature extraction according to claim 1 is characterized in that: The predicted transmittance T(x) corresponding to the output in step S1 is expressed as: T(x)=e -βd(x) (1) In the above formula, β is the scattering coefficient; d(x) is the scene depth.

3. The image defogging method based on atmospheric scattering model and frequency domain feature extraction according to claim 1 is characterized in that: The atmospheric scattering model in step S3 is expressed as: I(x)=J(x)T(x)+(1-T(x))A(x) (2) In the above formula, x represents the pixel position; I(x) represents the blurred image; J(x) represents the clear image; T(x) represents the transmission map; A(x) represents the atmospheric light.

4. The image defogging method based on atmospheric scattering model and frequency domain feature extraction according to claim 1 is characterized in that: The neural network architecture is U-net, and the neural network includes channel attention and pixel attention.

5. The image defogging method based on atmospheric scattering model and frequency domain feature extraction according to claim 1 is characterized in that: In step S1, the neural network is trained and outputs the corresponding predicted transmittance T(x). After downsampling, the neural network converts the feature map to the frequency domain using Fourier transform, and converts the feature map to the spatial domain through inverse Fourier transform during upsampling.

6. The image defogging method based on atmospheric scattering model and frequency domain feature extraction according to claim 1, characterized in that: The formula for calculating the average peak signal-to-noise ratio in step S5 is: In the above formula, a is the total number of output images; H and W are the height and width of the image respectively; n is the number of bits per pixel; X is the network output image; and Y is the corresponding clear image.

7. The image defogging method based on atmospheric scattering model and frequency domain feature extraction according to claim 1, characterized in that: The formula for calculating the average structural similarity in step S5 is: In the above formula, a is the total number of output images; μ x , μ y are the pixel averages of image x and image y respectively; x and σ y Represent the pixel standard deviations of image x and image y respectively; c1 and c2 are constants, where c1=0.0001 and c2=0.0009.

Images