A polarization image dehazing method based on 3D convolution

Abstract

This invention discloses a polarization image defogging method based on 3D convolution, comprising: 1. obtaining a synthetic polarization image dataset using a polarization image generation channel; 2. constructing a deep convolutional neural network based on 3D convolution using polarization images, and training the deep convolutional neural network with four polarization images at different polarization angles as input to obtain a defogging model; 3. using the trained model to defog the polarization image to be defogged, thereby obtaining a restored, fog-free image. This invention can achieve 3D convolution-based polarization image defogging, effectively improving the defogging effect in complex and changing scenes, thereby providing clearer images for many advanced visual tasks.

Description

Technical Field

[0001] The present invention belongs to the fields of computer vision, image processing and analysis, and in particular is a defogging method using polarized images based on a 3D convolutional network. Background Art

[0002] Fog or haze is a common atmospheric phenomenon. Under such weather conditions, the outdoor air contains a large number of tiny suspended particles, which refract and scatter atmospheric light. The refracted and scattered light mixes with the light reflected from the target scene to be observed, significantly reducing the visibility of the scene. This causes a significant decrease in the contrast of images captured by outdoor image acquisition equipment, and may even lead to image color distortion and loss of significant details. Some advanced computer vision tasks, such as object detection and image segmentation, require high-quality images as input. However, in severe weather conditions such as fog or haze, the quality of the acquired images degrades, significantly affecting the processing performance of these visual tasks. Therefore, image dehazing is crucial.

[0003] Image dehazing has garnered increasing attention from researchers in recent years, and numerous high-performing image dehazing models have been proposed. Currently, existing frameworks can be broadly divided into two categories: traditional dehazing methods based on artificial prior knowledge and dehazing methods based on deep learning algorithms. Traditional dehazing methods rely on artificial prior knowledge derived from clear image statistics and utilize atmospheric scattering models to recover haze-free images. A well-known example is the dark channel prior (DCP) algorithm proposed by He, which assumes that pixels in haze-free images have values ​​close to zero in at least one color channel. While traditional dehazing methods have made some progress, these assumptions and prior knowledge are specific to specific scenes and weather conditions, resulting in limited generalization. In other words, significant environmental changes can significantly degrade the model's dehazing performance. Deep learning-based methods train a dehazing model using extensive training data and then test the trained model on test data. Deep learning-based methods can also be divided into two categories: one that indirectly recovers the haze-free image by learning parameters from the atmospheric model through a network, and the other that utilizes deep learning networks to achieve end-to-end processing, directly outputting a haze-free image from a haze-containing image as input.

[0004] However, these learning-based methods still have shortcomings: 1. Most deep learning-based methods take a single image (RGB image) as input and use the atmospheric scattering model for training and testing. However, two key parameters in the model need to be estimated simultaneously, which leads to problems of ill-posedness and poor generalization ability; 2. In order to solve the ill-posed problem and improve generalization ability, more and more defogging methods based on multiple images have emerged. Among them, the method of using multiple polarization images with different polarization angles can make full use of scene information and achieve good results. However, most of these methods are based on the assumption that the transmitted light is not significantly polarized or require specific clues such as sky areas or similar objects, resulting in poor defogging effect on foggy images in the real world. Summary of the Invention

[0005] In order to address the shortcomings of the existing technology, the present invention provides a polarization image defogging method based on 3D convolution, in order to improve the quality of images taken in foggy environments and enhance the defogging effect in complex and changing scenes, thereby meeting the requirements of images required for advanced visual tasks and providing clearer images for many advanced visual tasks.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0007] The polarization image defogging method based on 3D convolution of the present invention is characterized in that it is performed according to the following steps:

[0008] Step 1: Obtain a synthetic polarization image dataset;

[0009] Step 1.1, obtain the fog-free image J(z) with the scene depth map d(z) and semantic segmentation map S;

[0010] Step 1.2: The atmospheric scattering factor β and the global atmospheric light A are calculated within a certain range. ∞ and the degree of polarization DoP of global atmospheric light A Randomly assign values, and then use formula (1) to generate the foggy image I(z) at pixel z:

[0011] I(z)=T(z)+A(z)=J(z)t(z)+A ∞ (1-t(z)) (1)

[0012] In formula (1), z represents the spatial coordinate of the pixel, T(z) and A(z) represent the transmitted light and atmospheric light at the pixel point z, respectively, J(z) represents the fog-free image, t(z) represents the transmission map at the pixel point z, and t(z) = e -βd(z) , where d(z) represents the scene depth map at pixel z;

[0013] Step 1.3: Calculate the degree of polarization DoP of the transmitted light TT =g(S), where S represents the semantic segmentation map and g represents the random mapping function;

[0014] Step 1.4: Calculate the degree of polarization DoP of the foggy image I using formula (2):

[0015] I·DoP=T·DoP T +A·DoP A (2)

[0016] In formula (2), DoPA represents the polarization degree of atmospheric light A;

[0017] Step 1.5: Use formula (3) to calculate the polarization angle: Polarization image of

[0018]

[0019] In formula (3), represents the orientation of the polarizer for transmitting the component parallel to the plane of incidence, and

[0020] Step 2: Build a polarization image dehazing model based on 3D convolution based on the U-Net architecture, which includes: POL-3D encoder, spatial redundancy reduction module SSR, and POL decoder;

[0021] Step 2.1: Construct the POL-3D encoder consisting of M 3D convolutional layers, wherein the m-th level of 3D convolutional layers includes, in sequence: a convolutional layer, an instance normalization layer, and a ReLU activation function layer; and the convolution kernel size of each convolutional layer is a tuple containing 3 integers, representing the convolution kernel size in the depth, height, and width dimensions respectively;

[0022] Step 2.2: Perform data dimension upscaling and fusion operations on the polarization images of different polarization angles, and then perform fusion to obtain a 4-dimensional high-dimensional feature map, which is input into the POL-3D encoder and sequentially passes through M 3D convolutional layers to obtain M feature maps with different numbers of channels, polarization angles, heights, and widths. The four dimensions of the high-dimensional feature map include the number of channels, polarization angle, image height, and width.

[0023] Step 2.3: After the spatial redundancy reduction module SSR processes the M feature maps, M valid feature maps F1, F2, ..., F are obtained. m ,…,F M ;

[0024] Step 2.4: The POL decoder decodes M valid feature maps F1, F2, ..., F m ,…,F MAfter processing, the final dehazing prediction map is output;

[0025] Step 3: Train a polarization image dehazing model based on 3D convolution;

[0026] Based on the polarization image and its corresponding real fog-free image, the ADAM optimizer is used to train the 3D convolution-based polarization image defogging model, and the mean absolute error L1 Loss is used as the loss function to calculate the loss between the defogging prediction image and the real fog-free image to update the model parameters until the loss function converges, thereby obtaining the optimal 3D convolution-based defogging model using polarization image, which is used to defog the synthetic polarization foggy image and the real polarization foggy image.

[0027] The polarization image dehazing method based on 3D convolution described in the present invention is also characterized in that the spatial redundancy reduction module SSR in step 2.3 is composed of M octave convolution layers and their corresponding maximum pooling layers, each octave convolution layer includes a preprocessing block, an octave convolution block and a post-processing block, and is processed according to the following steps:

[0028] Step 2.3.1: The preprocessing block in the m-th octave convolutional layer consists of two branches, one of which consists of a convolutional layer and an instance normalization layer in sequence to decompose high-frequency features; the other branch consists of an average pooling layer, a convolutional layer, and an instance normalization layer in sequence to decompose low-frequency features; the convolution kernels of the two convolutional layers are the same size, but the number of output channels is different; m∈M;

[0029] The mth feature map passes through the two branches of the preprocessing block in the mth octave convolution layer, and outputs the mth high-frequency feature map accordingly and the mth low-frequency feature map

[0030] In step 2.3.2, the octave convolution block in the m-th octave convolution layer consists of four convolution layers, two instance normalization layers, one average pooling layer, and one upsampling layer;

[0031] The mth high-frequency feature map After being processed by the first convolution layer of the octave convolution block in the mth octave convolution layer, the mth high-frequency to high-frequency feature map is obtained At the same time, the mth high-frequency feature map After being processed by an average pooling layer, it is input into the second convolutional layer for processing to obtain the mth high-frequency to low-frequency feature map

[0032] The mth low-frequency feature map After processing by the third convolutional layer, the mth low-frequency to low-frequency feature map is obtained At the same time, the mth low-frequency feature map After being input into the fourth convolution layer and the upsampling layer for processing, the mth low-frequency to high-frequency feature map is obtained.

[0033] Will and After fusion, it is input into the first instance normalization layer and outputs the mth high-frequency feature map

[0034] Will and After fusion, it is input into the second instance normalization layer and outputs the mth low-frequency feature map

[0035] Step 2.3.3, the post-processing block in the m-th octave convolutional layer consists of two convolutional layers, an upsampling layer, and an instance normalization layer;

[0036] The mth high-frequency feature map After a convolutional layer in the post-processing block, the mth high-frequency to high-frequency feature map is obtained

[0037] The mth low-frequency feature map After passing through another convolutional layer in the post-processing block, it is input into an upsampling layer to obtain the mth low-frequency to high-frequency feature map

[0038] and After fusion, the mth feature map is obtained after processing through an instance normalization layer.

[0039] The mth feature map Input the maximum pooling corresponding to the mth octave convolution layer for processing to obtain the mth effective feature map F m Thus, after the feature maps of M different channel numbers, polarization angles, heights and widths are processed by the spatial redundancy reduction module SSR, M effective feature maps F1, F2, ..., F are obtained. m ,…,F M .

[0040] The POL decoder in step 2.4 is composed of M deconvolution layers;

[0041] When m=1,2,…,M-1, each deconvolution layer includes a 2D convolution layer and an instance normalization layer; when m=M, the deconvolution layer only includes a 2D convolution layer;

[0042] When m=M, the Mth feature map F M After being processed by the bilinear interpolation function, it is input into the first deconvolution layer and the first deconvolution layer feature map F'1 is output;

[0043] When m=M-1,M-2,…,2, the mth feature map F m And the Mm-th deconvolution layer feature map F' M-m After fusion, the fused feature map is processed using the bilinear interpolation function and then input into the M-m+1th deconvolution layer, and the M-m+1th deconvolution layer feature map F' is output. M-m+1 ;

[0044] When m=1, the first feature map is processed by the Mth deconvolution layer to obtain the Mth deconvolution layer feature map F' m This is the final dehazing prediction map.

[0045] The electronic device of the present invention includes a memory and a processor, and is characterized in that the memory is used to store a program that supports the processor to execute any of the polarization image defogging methods, and the processor is configured to execute the program stored in the memory.

[0046] The present invention provides a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and the computer program is characterized in that when the computer program is executed by a processor, the steps of any of the polarization image defogging methods are executed.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] 1. This paper constructs a polarization image deep neural network based on 3D convolution, uses polarization images with different polarization angles as input, and combines it with a 3D convolution encoder to solve the problem of using 2D convolution for model design in traditional multi-image defogging networks, which ignores the correlation between grouped images, thereby improving the quality of restored fog-free images.

[0049] 2. The polarization image deep neural network based on 3D convolution constructed by the present invention introduces polarization information and uses multiple polarization images with different polarization angles in the same scene to obtain richer scene information. It solves the problem of using a single image as input and relying on image features extracted from training data, resulting in poor generalization ability, thereby improving the generalization ability of the dehazing network and being able to adapt to complex and changing environments.

[0050] 3. The polarization image deep neural network based on 3D convolution constructed by the present invention introduces a spatial redundancy reduction (SRR) module based on octave convolution, which decomposes the feature map output by the convolution layer into features of different spatial frequencies. By sharing information between adjacent positions, the spatial resolution of the low-frequency group can be safely reduced, solving the spatial redundancy problem caused by the dense parameters of the encoder, thereby reducing the number of network parameters and making the network more lightweight.

[0051] 4. The polarization image deep neural network based on 3D convolution constructed by the present invention aggregates the feature map output by each layer of the encoder with the feature map after passing through the spatial redundancy reduction module to obtain a more refined prediction result, thereby improving the dehazing effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 This is a flow chart of the present invention for defogging using polarized images based on a 3D convolutional defogging network;

[0053] Figure 2 This is a schematic diagram of the structure of the deep neural network for polarization image defogging based on 3D convolution in the present invention;

[0054] Figure 3 The defogging results of the method of the present invention and other defogging methods on the synthetic data set are shown in FIG.

[0055] Figure 4 The figure shows the dehazing results of the present invention and other dehazing methods on real-world datasets. DETAILED DESCRIPTION

[0056] In this embodiment, a polarization image defogging method based on 3D convolution is designed to solve the problem of the existing network lacking polarization datasets and extracting useful information from grouped polarization images (multiple images taken at different polarization angles from the same viewing angle). By constructing a polarization image deep neural network based on 3D convolution, a defogging model is obtained that can effectively remove fog without the need for specific clues, thereby improving the quality of images taken in foggy environments and meeting the requirements of images required for advanced visual tasks. Specifically, Figure 1 As shown, the following steps are followed:

[0057] Step 1: Obtain a synthetic polarization image dataset;

[0058] Step 1.1, find a suitable original data set for synthesizing the polarization image data set;

[0059] The original dataset needs to meet the following two requirements: (1) a fog-free image J(z) with a scene depth map d(z); (2) a semantic segmentation map S;

[0060] Step 1.2: The atmospheric scattering factor β and the global atmospheric light A are calculated within a certain range.∞ and the degree of polarization DoP of global atmospheric light A Randomly assign values, and then use formula (1) to generate the foggy image I(z) at pixel z:

[0061] I(z)=T(z)+A(z)=J(z)t(z)+A ∞ (1-t(z)) (1)

[0062] In formula (1), z represents the spatial coordinate of the pixel, T(z) and A(z) represent the transmitted light and atmospheric light at the pixel point z, respectively, J(z) represents the fog-free image, t(z) represents the transmission map at the pixel point z, and t(z) = e -βd(z) , where d(z) represents the scene depth at pixel z; in this embodiment, the atmospheric scattering factor β ranges from [0.01, 0.02], and the global atmospheric light A ∞ The value range is [0.85,0.95], the polarization degree DoP of the global atmospheric light A The value range is [0.05, 0.4].

[0063] Step 1.3: Calculate the degree of polarization DoP of the transmitted light T T =g(S), where S represents the semantic segmentation map and is provided by the original data set, and g represents the random mapping function; in this embodiment, the degree of polarization DoP of the transmitted light T The value range is [0.025,0.2].

[0064] Step 1.4: Calculate the degree of polarization DoP of the foggy image I using formula (2):

[0065] I·DoP=T·DoP T +A·DoP A (2)

[0066] In formula (2), DoP A Indicates the polarization degree of atmospheric light A; in this embodiment, I, T, A can be decomposed into I / / and I ⊥ , T / / and T ⊥ , A / / and A ⊥ , where / / and ⊥ indicate that the component is parallel or perpendicular to the incident plane. Thus, the polarization degree of I, T, and A can be defined as

[0067] Step 1.5: Use formula (3) to calculate the polarization angle: Polarization image of

[0068]

[0069] In formula (3), represents the orientation of the polarizer for transmitting the component parallel to the plane of incidence, and

[0070] In this embodiment, due to the special requirements of the polarization-based synthetic dataset generation pipeline, it is impossible to generate the dataset of the present invention from the foggy images provided by the existing benchmark. At the same time, the Foggy Cityscapes-DBF dataset meets all the requirements. Therefore, the present invention uses the fog-free image J and depth map z provided by the dataset to generate the scattering coefficient β and the global atmospheric light A. ∞ , and use the semantic segmentation map S to generate DOP T , and finally the fog map I is calculated.

[0071] Step 2: Build a polarization image dehazing model based on 3D convolution based on the U-Net architecture, which includes: POL-3D encoder, spatial redundancy reduction module SSR, and POL decoder;

[0072] Step 2.1. Construct a POL-3D encoder consisting of M 3D convolutional layers, where the m-th level 3D convolutional layer includes, in sequence: a convolutional layer, an instance normalization layer, and a ReLU activation function layer; and the convolution kernel size of each convolutional layer is a tuple containing 3 integers, representing the convolution kernel size in the depth, height, and width dimensions respectively; in this embodiment, the value of M is 5. When m = 1, 5, the convolution kernel size is set to (3, 3, 3); when m = 2, ..., 4, the convolution kernel size is set to (2, 3, 3).

[0073] Step 2.2, the polarization images with different polarization angles are subjected to data dimension upgrading and fusion operations and then fused to obtain a 4-dimensional high-dimensional feature map, which is input into the POL-3D encoder and sequentially passed through M 3D convolutional layers to obtain M feature maps with different numbers of channels, polarization angles, heights and widths, wherein the four dimensions of the high-dimensional feature map include the number of channels, polarization angles, image height and width; in this embodiment, four polarization images with different polarization angles (0°, 45°, 90°, 135°) are used as input, and the high-dimensional feature map can be represented as a four-dimensional tensor: C×P×H×W, where the initial values ​​of C, P, H, and W are 3, 4, 256, and 256, respectively. The number of output channels of the five 3D convolutional layers is set to 64, 128, 256, 512, and 512, respectively.

[0074] Step 2.3, construct the spatial redundancy reduction module SSR, which is composed of M octave convolution layers and their corresponding maximum pooling layers. Each octave convolution layer includes a preprocessing block First OctConv, an octave convolution block OctConv and a post-processing block LastOctConv. In this embodiment, the number of input and output channels of the five octave convolution layers are (64, 64), (128, 128), (256, 256), (512, 512), (512, 512), and the convolution kernel sizes of the preprocessing block, the octave convolution block and the post-processing block are (1, 1, 1), (3, 3, 3), and (3, 3, 3), respectively.

[0075] Step 2.3.1, the preprocessing block in the m-th octave convolution layer consists of two branches, one of which is composed of a convolution layer and an instance normalization layer in sequence, which is used to decompose high-frequency features; the other branch is composed of an average pooling layer, a convolution layer and an instance normalization layer in sequence, which is used to decompose low-frequency features; wherein the convolution kernels of the two convolution layers are the same size and the number of output channels is different; m∈M; in this embodiment, the convolution kernel sizes of the two convolution layers are both (3,3,3), and the number of output channels is controlled by the factor α, and the number of output channels of the convolution layer that decomposes high-frequency features is αc out , the number of output channels of the convolutional layer that decomposes low-frequency features is (1-α)c out , α is set to 0.5, the convolution kernel size of the average pooling layer is (1,2,2), and the convolution step size is (1,2,2).

[0076] The mth feature map passes through the two branches of the preprocessing block in the mth octave convolution layer, and outputs the mth high-frequency feature map and the mth low-frequency feature map respectively;

[0077] In step 2.3.2, the octave convolution block in the m-th octave convolution layer consists of four convolution layers, two instance normalization layers, one average pooling layer, and one upsampling layer;

[0078] The mth high-frequency feature map After processing the first convolution layer in the m-th octave convolution layer, the m-th high-frequency to high-frequency feature map is obtained At the same time, the mth high-frequency feature map After being processed by an average pooling layer, it is input into the second convolutional layer for processing to obtain the mth high-frequency to low-frequency feature map

[0079] The mth low-frequency feature map After being processed by the third convolutional layer, the mth low-frequency to low-frequency feature map is obtained. At the same time, the mth low-frequency feature map The fourth convolution layer is input in sequence, and the upsampling layer is processed to obtain the mth low-frequency to high-frequency feature map. In this embodiment, the convolution kernel size of the average pooling is (1, 2, 2), the convolution step size is (1, 2, 2), the upsampling magnification factor is (1, 2, 2), and the algorithm uses the nearest algorithm.

[0080] Will and After fusion, it is input into the first instance normalization layer and outputs the mth high-frequency feature map

[0081] Will and After fusion, it is input into the second instance normalization layer and outputs the mth low-frequency feature map

[0082] Step 2.3.3, the post-processing block in the m-th octave convolutional layer consists of two convolutional layers, an upsampling layer, and an instance normalization layer;

[0083] The mth high-frequency feature map After a convolutional layer in the post-processing block, the mth high-frequency to high-frequency feature map is obtained

[0084] The mth low-frequency feature map After passing through another convolutional layer in the post-processing block, it is input into an upsampling layer to obtain the mth low-frequency to high-frequency feature map

[0085] and After fusion, the mth feature map is obtained after processing through an instance normalization layer.

[0086] The mth feature map Input the maximum pooling corresponding to the mth octave convolution layer for processing to obtain the mth effective feature map F m Thus, after the feature maps of M different channel numbers, polarization angles, heights and widths are processed by the spatial redundancy reduction module SSR, M effective feature maps F1, F2, ..., F are obtained. m ,…,F M ;

[0087] Step 2.4: Construct a POL decoder consisting of M deconvolution layers. By using a bilinear interpolation function, the low-level, mid-level, and high-level feature maps generated by the POL-3D encoder are combined with the output of each deconvolution layer after passing through the spatial redundancy reduction module to output the final dehazed prediction map.

[0088] When m=1,2,…,M-1, each deconvolution layer includes a 2D convolution layer and an instance normalization layer; when m=M, the deconvolution layer includes only one 2D convolution layer; in this embodiment, M is 5, and the numbers of input and output channels of the five deconvolution layers are (512,512), (1024,256), (512,128), (256,64) and (128,3), respectively. The convolution kernel size is 3 and the convolution step is 1.

[0089] When m=M, the Mth feature map F M After being processed by the bilinear interpolation function, it is input into the first deconvolution layer and the first deconvolution layer feature map F'1 is output;

[0090] When m=M-1,M-2,…,1, the mth feature map F m And the Mm-th deconvolution layer feature map F' M-m After fusion, the fused feature map is processed using the bilinear interpolation function and then input into the M-m+1th deconvolution layer, and the M-m+1th deconvolution layer feature map F' is output. M-m+1 ;

[0091] When m=1, the Mth deconvolution layer processes the first feature map, and the resulting Mth deconvolution layer feature map F' M This is the final dehazing prediction map.

[0092] Step 3: Train a polarization image dehazing model based on 3D convolution;

[0093] Based on the polarization image and its corresponding real fog-free image, the ADAM optimizer is used to train the polarization image defogging model based on 3D convolution, and the mean absolute error (L1 Loss) is used as the loss function to calculate the loss between the defogging prediction image and the real fog-free image to update the model parameters until the loss function converges. In this way, the optimal 3D convolution-based defogging model using polarization images is obtained, which is used to defog the synthetic polarization foggy image and the real polarization foggy image. In this embodiment, during the training phase, the network is trained for 300 epochs, and the initial learning rate is set to 1e -4 , it decays by 0.5 every 50 epochs.

[0094] In this embodiment, 8925 polarization images with four different polarization angles (0°, 45°, 90°, and 135°) generated by the polarization image generation channel and their corresponding real fog-free images are used for training. During the training process, the input polarization image size is randomly cropped to 256×256, and the image-free image output by the defogging model based on 3D convolution is converted to 256×256. The L1 loss is calculated with J, and the calculated loss is combined with the ADAM optimizer to guide the network training to obtain a dehazing model based on 3D convolution using polarized images.

[0095] In this embodiment, an electronic device includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the above method, and the processor is configured to execute the program stored in the memory.

[0096] In this embodiment, a computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method are executed.

[0097] Table 1 shows the evaluation indicators of the polarization image defogging method based on 3D convolution of the present invention, using "PSNR" and "SSIM" respectively. For fair comparison, all defogging methods were retrained on the synthetic data set. During the test, the present invention and other defogging methods used polarization images and ordinary RGB images respectively. "PSNR" is the peak signal-to-noise ratio, which is the ratio of the maximum power of the signal to the noise power that may affect its representation accuracy. The larger the value, the smaller the distortion of the image defogging. "SSIM" is the structural similarity. This indicator examines the similarity of images from three aspects: brightness, contrast, and structure. Its value range is [0,1]. The larger the value, the more similar the defogging image is to the real fog-free image. According to the quantitative analysis of Table 1, it can be seen that the method of the present invention has achieved the best results in both indicators.

[0098] Table 1

[0099] Methods PSNR(↑) SSIM(↑) AOD-Net 20.58 0.80 PFFNet 28.63 0.89 GridDehazeNet 29.23 0.91 4KDehazing-Net 29.47 0.90 FFA-Net 29.93 0.92 GCANet 29.98 0.91 Ours 30.21 0.92

[0100] Figure 3The results of the polarization image dehazing method based on 3D convolution of the present invention and other current dehazing methods on the synthetic data set are shown in Figure 2. Ours represents the polarization image dehazing method based on 3D convolution of the present invention; AOD-Net proposes to use an end-to-end trainable network for dehazing based on the reformulation of the atmospheric scattering model, and replaces the two key unknowns in the atmospheric scattering model with one; PFFNet is inspired by the end-to-end dehazing idea and adopts a network based on the U-Net architecture. It adds a ResNet-based conversion module between the encoder and decoder to improve the learning of complex features at different levels; GridDehazeNet is inspired by the grid-based image segmentation network and proposes another end-to-end trainable network that is independent of the atmospheric model, including It includes preprocessing, attention-based multi-scale backbone and post-processing modules. The attention mechanism helps the Backbone module adjust the multi-scale feature fusion contribution by activating / deactivating part of GridDehazeNet; 4KDehazing-Net proposes a 4K resolution image dehazing framework with multi-guided bilateral learning, which can process 4K (3840×2160) resolution images at 125 frames per second; FFA-Net proposes to combine channel and pixel attention, and emphasizes residual learning and feature fusion; GCANet uses smooth void convolution and gated network to fuse multi-level features to aggregate contextual information.

[0101] Figure 4 The results of the polarization image dehazing method based on 3D convolution of the present invention and other current dehazing methods on real datasets are shown. It can be seen that, unlike the comparison on synthetic data, the method of the present invention shows obvious advantages on real datasets. This is because the scattering in the real world is a complex physical process with spatial variation, which makes it difficult to learn the influence of spatial variation from ordinary RGB images. The lack of physics-based learning features makes these methods prone to artifacts for pixels with large spatial variation interference. In contrast, the method of the present invention alleviates this problem by mining the correlation between four polarization images with different polarization angles.

Claims

1. A polarization image defogging method based on 3D convolution, characterized in that: The steps are as follows: Step 1: Obtain a synthetic polarization image dataset; Step 1.1, obtain the fog-free image J(z) with the scene depth map d(z) and semantic segmentation map S; Step 1.2: The atmospheric scattering factor β and the global atmospheric light A are calculated within a certain range. ∞ and the degree of polarization DoP of global atmospheric light A Randomly assign values, and then use formula (1) to generate the foggy image I(z) at pixel z: I(z)=T(z)+A(z)=J(z)t(z)+A ∞ (1-t(z)) (1) In formula (1), z represents the spatial coordinate of the pixel, T(z) and A(z) represent the transmitted light and atmospheric light at the pixel point z, respectively, J(z) represents the fog-free image, t(z) represents the transmission map at the pixel point z, and t(z) = e -βd (z), where d(z) represents the scene depth map at pixel z; Step 1.3: Calculate the degree of polarization DoP of the transmitted light T T =g(S), where S represents the semantic segmentation map and g represents the random mapping function; Step 1.4: Calculate the degree of polarization DoP of the foggy image I using formula (2): I·DoP=T·DoP T +A·DoP A (2) In formula (2), DoP A Indicates the polarization degree of atmospheric light A; Step 1.5: Use formula (3) to calculate the polarization angle: Polarization image of In formula (3), represents the orientation of the polarizer for transmitting the component parallel to the plane of incidence, and Step 2: Build a polarization image dehazing model based on 3D convolution based on the U-Net architecture, which includes: POL-3D encoder, spatial redundancy reduction module SSR, and POL decoder; Step 2.1: Construct the POL-3D encoder consisting of M 3D convolutional layers, wherein the m-th level of 3D convolutional layers includes, in sequence: a convolutional layer, an instance normalization layer, and a ReLU activation function layer; and the convolution kernel size of each convolutional layer is a tuple containing 3 integers, representing the convolution kernel size in the depth, height, and width dimensions respectively; Step 2.2: Perform data dimension upscaling and fusion operations on the polarization images of different polarization angles, and then perform fusion to obtain a 4-dimensional high-dimensional feature map, which is input into the POL-3D encoder and sequentially passes through M 3D convolutional layers to obtain M feature maps with different numbers of channels, polarization angles, heights, and widths. The four dimensions of the high-dimensional feature map include the number of channels, polarization angle, image height, and width. Step 2.3: After the spatial redundancy reduction module SSR processes the M feature maps, M valid feature maps F1, F2, ..., F are obtained. m ,...,F M ; Step 2.4: The POL decoder decodes M valid feature maps F1, F2, ..., F m ,…,F M After processing, the final dehazing prediction map is output; Step 3: Train a polarization image dehazing model based on 3D convolution; Based on the polarization image and its corresponding real fog-free image, the ADAM optimizer is used to train the 3D convolution-based polarization image defogging model, and the mean absolute error L1Loss is used as the loss function to calculate the loss between the defogging prediction image and the real fog-free image to update the model parameters until the loss function converges, thereby obtaining the optimal 3D convolution-based defogging model using polarization images, which is used to defog synthetic polarization foggy images and real polarization foggy images.

2. The polarization image defogging method based on 3D convolution according to claim 1, characterized in that: The spatial redundancy reduction module SSR in step 2.3 is composed of M octave convolution layers and their corresponding maximum pooling layers. Each octave convolution layer includes a pre-processing block, an octave convolution block, and a post-processing block, and is processed according to the following steps: In step 2.3.1, the preprocessing block in the m-th octave convolutional layer consists of two branches, one of which consists of a convolutional layer and an instance normalization layer in sequence to decompose high-frequency features; the other branch consists of an average pooling layer, a convolutional layer, and an instance normalization layer in sequence to decompose low-frequency features; The convolution kernels of the two convolutional layers have the same size but different numbers of output channels; m∈M; The mth feature map passes through the two branches of the preprocessing block in the mth octave convolution layer, and outputs the mth high-frequency feature map accordingly and the mth low-frequency feature map In step 2.3.2, the octave convolution block in the m-th octave convolution layer consists of four convolution layers, two instance normalization layers, one average pooling layer, and one upsampling layer; The mth high-frequency feature map After being processed by the first convolution layer of the octave convolution block in the mth octave convolution layer, the mth high-frequency to high-frequency feature map is obtained At the same time, the mth high-frequency feature map After being processed by an average pooling layer, it is input into the second convolutional layer for processing to obtain the mth high-frequency to low-frequency feature map The mth low-frequency feature map After processing by the third convolutional layer, the mth low-frequency to low-frequency feature map is obtained At the same time, the mth low-frequency feature map After being input into the fourth convolution layer and the upsampling layer for processing, the mth low-frequency to high-frequency feature map is obtained. Will and After fusion, it is input into the first instance normalization layer and outputs the mth high-frequency feature map Will and After fusion, it is input into the second instance normalization layer and outputs the mth low-frequency feature map Step 2.3.3, the post-processing block in the m-th octave convolutional layer consists of two convolutional layers, an upsampling layer, and an instance normalization layer; The mth high-frequency feature map After a convolutional layer in the post-processing block, the mth high-frequency to high-frequency feature map is obtained The mth low-frequency feature map After passing through another convolutional layer in the post-processing block, it is input into an upsampling layer to obtain the mth low-frequency to high-frequency feature map and After fusion, the mth feature map is obtained after processing through an instance normalization layer. The mth feature map Input the maximum pooling corresponding to the mth octave convolution layer for processing to obtain the mth effective feature map F m Thus, after the feature maps of M different channel numbers, polarization angles, heights and widths are processed by the spatial redundancy reduction module SSR, M effective feature maps F1, F2, ..., F are obtained. m ,…,F M .

3. The polarization image defogging method based on 3D convolution according to claim 2, characterized in that: The POL decoder in step 2.4 is composed of M deconvolution layers; When m=1,2,…,M-1, each deconvolution layer includes a 2D convolution layer and an instance normalization layer; when m=M, the deconvolution layer only includes a 2D convolution layer; When m=M, the Mth feature map F M After being processed by the bilinear interpolation function, it is input into the first deconvolution layer and the first deconvolution layer feature map F'1 is output; When m=M-1,M-2,…,2, the mth feature map F m And the Mm-th deconvolution layer feature map F' M-m After fusion, the fused feature map is processed using the bilinear interpolation function and then input into the M-m+1th deconvolution layer, and the M-m+1th deconvolution layer feature map F' is output. M-m+1 ; When m=1, the first feature map is processed by the Mth deconvolution layer to obtain the Mth deconvolution layer feature map F' M This is the final dehazing prediction map.

4. An electronic device comprising a memory and a processor, characterized in that: The memory is used to store a program that supports the processor to execute the polarization image defogging method according to any one of claims 1 to 3, and the processor is configured to execute the program stored in the memory.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the polarization image defogging method according to any one of claims 1 to 3 are executed.

Images