A polarization de-reflecting method based on pseudo 3D convolution

Abstract

The application discloses a polarization de-reflection method based on pseudo 3D convolution, comprising the following steps: 1, a real-world polarization de-reflection dataset of real shooting is constructed, and a synthetic polarization de-reflection dataset is obtained by using a polarization imaging principle and optical theory; 2, a deep convolutional neural network based on pseudo 3D convolution and using polarization images is constructed, a plurality of polarization images with different polarization angles are taken as input, the deep convolutional neural network is trained, and a multi-image de-reflection model is obtained; 3, the trained model is used for de-reflection processing on polarization images to be mixed, and a restored transmission image is obtained. The application can realize polarization de-reflection based on pseudo 3D convolution, effectively improve the de-reflection effect in a complex and changeable scene, and thus can provide clearer images for many advanced vision tasks.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of computer vision, image processing and analysis, in particular to a polarization de-reflecting method based on pseudo 3D convolution. BACKGROUND

[0002] Nowadays, with the rapid update iteration of various digital products and shooting devices, in photography, science and computer vision imaging systems, there are many cases that inevitably need to capture images through glass windows or transparent shells, such as shooting the objects behind the glass in the museum or shooting the goods in the showcase. The scenes of glass reflection are often contained in these images, and the undesirable reflections in these images will reduce the visual effect and quality of the images and also affect the performance of subsequent computer vision algorithms. Since the clear transmission scene can be restored by eliminating these reflection scenes, image de-reflection and separation of reflection and transmission image are popular research contents in computer photography and computer vision field.

[0003] Image de-reflection algorithm is mainly used in computer vision and image processing to improve image quality and accuracy. For example, when shooting indoors, due to the reflection and refraction of light, there will be obvious reflection images and highlight areas in the image, which will affect the clarity and color restoration of the image. Image de-reflection algorithm can effectively eliminate these reflections and highlights, making the image more realistic and clear, which helps to improve the accuracy of image recognition and the effectiveness of image analysis.

[0004] In recent years, image de-reflection has attracted more and more attention from researchers, and many good performance image de-reflection models have been proposed. At present, the existing frameworks can be roughly divided into two categories: traditional de-reflection methods based on artificial prior knowledge and de-reflection methods based on deep learning algorithm. Traditional de-reflection methods rely on prior knowledge such as gradient sparsity and gradient independence, and use mathematical models to separate reflection and transmission images. The famous one is the image de-reflection algorithm based on independent component analysis proposed by Farid H. Although traditional de-reflection methods have made some progress, these assumptions and prior knowledge are specific to certain scenarios, so the generalization ability is limited, that is, once the environment changes significantly, the de-reflection ability of the model will be significantly reduced. The method based on deep learning trains a de-reflection model through a large amount of training data, and tests the trained model on test data. Deep learning-based algorithms have shown strong modeling and understanding capabilities, and can detect various reflection image features. Deep learning-based methods can also be divided into two categories: one is single-image de-reflection that only uses one mixed image to separate the transmission image and the reflection image, and the other is multi-image de-reflection that uses multiple mixed images to separate the transmission image and the reflection image.

[0005] However, these learning-based methods still have some shortcomings:

[0006] 1. Single-image deflection methods cannot handle complex specular reflections because the input is only one image. In addition, the effect of single-image deflection methods can be poor for strong reflections;

[0007] 2. Most multi-image deflection methods rely on specific conditions (such as near-infrared light, depth maps, etc.), and capturing multiple images under these specific constraints is often complex and time-consuming;

[0008] 3. Many current deep learning-based algorithms instead use synthetic datasets for training. However, existing synthetic methods mainly rely on simple linear synthesis to generate mixed images, which significantly differs from the real imaging process, resulting in poor performance of network models when processing real-world reflection data, poor generalization ability, and unsatisfactory deflection effects. SUMMARY

[0009] To address the shortcomings of the prior art, the present application provides a polarization deflection method based on pseudo-3D convolution, which aims to improve the quality of captured reflection images, transmitted images, and mixed images, and to improve the deflection effect in complex and variable scenarios, thereby meeting the requirements of high-level vision tasks and providing clearer images for many high-level vision tasks.

[0010] To solve the above technical problems, the technical solution adopted by the present application is:

[0011] The polarization image deflection method based on pseudo-3D convolution of the present application is characterized by the following steps:

[0012] Step 1, obtain a comprehensive polarization dataset containing real mixed polarization images and synthetic mixed polarization images;

[0013] Step 1.1, obtain a real mixed polarization image set under K1 static scenes , a real transmitted polarization image set , and a real reflection polarization image set ; wherein, represents the real mixed polarization image with a polarization angle of under the k1th static scene; represents the real transmitted polarization image with a polarization angle of under the k1th static scene; represents the real reflection polarization image with a polarization angle of under the k1th static scene;

[0014] Step 1.2, obtain a synthetic mixed polarization image set under another K2 static scenes , a set of synthetic transmission polarization images , and a set of synthetic reflection polarization images ; wherein, represents a synthetic mixed polarization image with polarization angle under the k2th static scene; represents a synthetic transmission polarization image with polarization angle under the k2th static scene; represents a synthetic reflection polarization image with polarization angle under the k2th static scene;

[0015] Step 1.3, merging and into a mixed polarization dataset ; merging and into a transmission polarization dataset ; merging and into a reflection polarization dataset ; wherein, represents a mixed polarization image with polarization angle under the kth static scene; represents a transmission polarization image with polarization angle under the kth static scene; represents a transmission polarization image with polarization angle under the kth static scene;

[0016] After pixel-wise averaging of for all polarization angles under the same scene, a mixed non-polarization image under the kth static scene is obtained ; after pixel-wise averaging of for all polarization angles under the same scene, a transmission non-polarization image under the kth static scene is obtained ; after pixel-wise averaging of for all polarization angles under the same scene, a reflection non-polarization image under the kth static scene is obtained ;

[0017] Step 2, based on the U-Net architecture, a polarization de-reflection network based on pseudo 3D convolution is constructed, including: a feature dimension increasing module, an encoder, a transmission image decoder, and a reflection image decoder;

[0018] Step 2.1, the feature dimension increasing module performs dimension increasing and fusion operations on , thereby obtaining a 4-dimensional feature map under the kth static scene ;

[0019] Step 2.2, the encoder is composed of M basic convolution modules, and is sequentially processed to obtain M high-frequency feature maps , M low-frequency feature maps and M high-low frequency superimposed feature maps under the kth static scene; wherein, represents the mth high-frequency feature map under the kth static scene, represents the mth low-frequency feature map under the kth static scene, represents the mth high-low frequency superimposed feature map under the kth static scene. Step 2.3, the transmission image decoder processes and to obtain the recovered transmission image

[0020] under the kth static scene. Step 2.4, the reflection image decoder processes and to obtain the recovered reflection image

[0021] under the kth static scene. Step 3, training the polarization image de-reflection model based on pseudo 3D convolution. Step 3.1, constructing the polarization image de-reflection model based on

[0022] , and

[0023] , and using formula (1) to construct the total loss function L: (1)

[0024] (1)

[0025] In formula (1), represents the reconstruction loss, represents the perceptual loss, represents the repulsion loss, and represents the adversarial loss, λ1, λ2, λ3 are the weights of the perceptual loss , the adversarial loss and the repulsion loss respectively.

[0026] Step 3.2, using the ADAM optimizer to train the polarization image de-reflection network based on pseudo 3D convolution, and in the training process, first fixing the generator, updating the discriminator to minimize the discriminator loss , and then fixing the discriminator, updating the generator to minimize the generator loss ​​, so as to alternately update the discriminator and the generator until the total loss function L converges, thereby obtaining an optimal polarization image de-reflection model based on pseudo 3D convolution for de-reflection processing of the synthesized polarization mixed image and the actually shot polarization mixed image.

[0027] The polarization image de-reflection method based on pseudo 3D convolution provided by the application is also characterized in that each basic convolution module in the step 2.2 is composed of a nonlinear transformation module, a polarization feature attention module PFAM, a spatial feature attention module SFAM and an octave convolution residual module OCRM;

[0028] In step 2.2.1, when m = 1, the nonlinear transformation module in the mth basic convolution module sequentially uses a first pseudo 3D convolution layer, a ReLU activation function operation and a second pseudo 3D convolution layer to perform nonlinear transformation on , and obtains the mth nonlinear 4D feature map under the kth static scene with a shape of C x P x H x W.

[0029] In step 2.2.2, the polarization feature attention module in the mth basic convolution module sequentially uses a 3D average pooling layer, a third pseudo 3D convolution layer, a ReLU activation function operation, a fourth pseudo 3D convolution layer and a Sigmoid activation function operation to convert into the mth polarization attention feature map with a shape of C x P x 1 x 1. After pixel-by-pixel multiplication of and , the mth polarization attention weighted feature map under the kth static scene is obtained.

[0030] In step 2.2.3, the spatial feature attention module in the mth basic convolution module sequentially uses a fifth pseudo 3D convolution layer, a ReLU activation function operation, a sixth pseudo 3D convolution layer and a Sigmoid activation function operation to convert into the mth spatial attention feature map . After pixel-by-pixel multiplication of and , the mth comprehensive attention weighted feature map under the kth static scene is obtained.

[0031] In step 2.2.4, the octave convolution residual module in the mth basic convolution module first uses a seventh pseudo 3D convolution layer and an InstanceNorm3D normalization layer to obtain ​​Extract the first High and low frequency mixed feature map ;

[0032] Then, along the polarization channel dimension Decomposed into the first Original high-frequency feature map With the Original low-frequency feature map ;

[0033] right After performing average pooling and pseudo-3D convolution operations sequentially, we obtain the... High-frequency to low-frequency feature map ;

[0034] right After performing pseudo-3D convolution, we obtain the first... High-frequency to high-frequency feature map ;

[0035] right After performing pseudo-3D convolution and upsampling operations sequentially, we obtain the... Low-frequency to high-frequency feature map ;

[0036] right After performing pseudo-3D convolution, we obtain the first... Low-frequency to low-frequency feature map ;

[0037] Will and After adding pixel by pixel, and with Perform pixel-by-pixel multiplication, followed by ReLU activation function operation, to obtain the [then]. High-frequency feature map ;

[0038] Will and After adding pixel by pixel, and with Perform pixel-by-pixel multiplication, followed by ReLU activation, to obtain the [then]. Low-frequency feature map ;

[0039] right After performing pseudo-3D convolution and InstanceNorm3D normalization operations, we obtain the first... Normalized high-frequency feature map ;

[0040] right After performing pseudo-3D convolution and InstanceNorm3D normalization operations, we obtain the first... one normalized low-frequency feature map ;

[0041] and After superimposing in the polarization channel dimension, the first high-low frequency superimposed feature map ;

[0042] Step 2.2.5, when m = 2, 3, …, M, the first high-low frequency superimposed feature map is input into the mth basic convolution module for processing to obtain the first high-low frequency superimposed feature map , so that the Mth high-low frequency superimposed feature map is output from the Mth basic convolution module.

[0043] Further, the transmission image decoder and the reflection image decoder in the step 2.3 are each composed of M up-sampling skip connection modules and M 2D convolution modules;

[0044] Step 2.3.1, when m = 1, the mth up-sampling skip connection module of the transmission image decoder enlarges in the spatial dimension, and then, after being spliced with in the polarization channel dimension, the mth 2D high-frequency transmission feature map is obtained through the processing of the mth 2D convolution module of the transmission image decoder.

[0045] When m = 2, 3, …, M-1, the (m-1)th 2D high-frequency transmission feature map is input into the mth up-sampling skip connection module of the transmission image decoder, and after being enlarged in the spatial dimension, is spliced with in the polarization channel dimension, the mth 2D high-frequency transmission feature map is obtained through the processing of the mth 2D convolution module of the transmission image decoder; and the (M-1)th 2D high-frequency transmission feature map is output from the (M-1)th 2D convolution module of the transmission image decoder.

[0046] When m = M, the (M-1)th 2D high-frequency transmission feature map is input into the Mth up-sampling skip connection module of the transmission image decoder, and after being enlarged in the spatial dimension, is spliced with ​After the splicing in the polarization channel dimension, the mth 2D high-frequency transmission feature map output by the transmission image decoder is obtained after the processing of the mth 2D convolution module of the transmission image decoder , and used as the recovered transmission image.

[0047] Step 2.3.2, when m = 1, the mth upsampling skip connection module in the reflection image decoder is enlarged in the spatial dimension after the processing of , and then the mth 2D low-frequency reflection feature map is obtained after the splicing in the polarization channel dimension and the processing of the mth 2D convolution module in the reflection image decoder .

[0048] When m = 2, 3, …, M-1, the (m-1)th 2D low-frequency reflection feature map is input into the mth upsampling skip connection module of the reflection image decoder, and then enlarged in the spatial dimension after the processing of , and then the mth 2D low-frequency reflection feature map is obtained after the splicing in the polarization channel dimension and the processing of the mth 2D convolution module in the reflection image decoder . ; and the (M-1)th 2D low-frequency reflection feature map is output by the (M-1)th 2D convolution module of the reflection image decoder .

[0049] When m = M, the (M-1)th 2D low-frequency reflection feature map is input into the Mth upsampling skip connection module of the reflection image decoder, and then enlarged in the spatial dimension after the processing of , and then the Mth 2D low-frequency reflection feature map output by the reflection image decoder is obtained after the splicing in the polarization channel dimension and the processing of the mth 2D convolution module in the reflection image decoder , and used as the recovered reflection image.

[0050] Further, the reconstruction loss in step 3.1 is constructed by using formula (2) :

[0051] (2)

[0052] In formula (2), is the absolute value distance, is the gradient with respect to the image horizontal coordinate x, is the gradient with respect to the image vertical coordinate y.

[0053] ​​​​The polarized reflection removal network based on pseudo 3D convolution is taken as the generator G, and the loss of the generator G in the adversarial loss is constructed by using formula (5) and formula (6) respectively and the loss of the discriminator D :

[0054] (5)

[0055] (6)

[0056] In formula (5) and formula (6), is the mapping value of the discriminator D, is the mapping value of the discriminator D.

[0057] The electronic device comprises a memory and a processor, and the memory is used for storing a program supporting the processor to execute the polarized reflection removal method, and the processor is configured to execute the program stored in the memory.

[0058] The computer readable storage medium stores a computer program, and when the computer program is run by a processor, the steps of the polarized reflection removal method are executed.

[0059] Compared with the prior art, the present application has the following advantages:

[0060] 1. The polarized image deep neural network based on pseudo 3D convolution is constructed, and the polarized images with different polarization angles are used as input, combined with the pseudo 3D convolution encoder, which solves the problem that only 2D convolution is used for model design in the traditional multi-image reflection removal network, and the correlation between grouped images is ignored, thereby improving the quality of the recovered reflection and transmission images.

[0061] 2. The polarized image deep neural network based on pseudo 3D convolution constructed in the present application introduces polarization information, uses multiple polarized images with different polarization angles in the same scene, can obtain more rich scene information, solves the problem that the traditional single-image reflection removal network uses a single image as input and relies on the image features extracted in the training data, resulting in poor generalization ability, thereby improving the robustness of the reflection removal network and adapting to complex and variable reflection environments.

[0062] ​​3、The polarized image deep neural network based on pseudo 3D convolution constructed by the application introduces an octuple convolution residual module based on local residual operation and octuple convolution, enhances the association of context information, so that the two prediction streams can pay more attention to the feature information that the current prediction stream needs to pay attention to, and at the same time, the feature maps output by the convolution layer are decomposed into features of different spatial frequencies, the memory and calculation cost are reduced, the model can more efficiently learn important information in different frequencies in the image, and the generalization ability of image de-reflection is improved.

[0063] 4、The polarized image deep neural network based on pseudo 3D convolution constructed by the application introduces a spatial feature attention module and a polarized feature attention module based on an attention mechanism. The attention is focused on the strong reflection area. In this way, the change of the reflection scene can be more effectively processed, the accuracy and robustness of image processing are improved; at the same time, effective feature information is extracted from redundant polarized information to improve the quality of the recovered image.

[0064] 5、The polarized image deep neural network based on pseudo 3D convolution constructed by the application aggregates the feature maps output by each layer of the encoder and the feature maps of the corresponding decoder through a skip connection to obtain more refined prediction results, thereby improving the accuracy of image de-reflection. BRIEF DESCRIPTION OF DRAWINGS

[0065] Figure 1 It is a deep neural network structure diagram for image de-reflection based on pseudo 3D convolution of the application;

[0066] Figure 2 It is a basic convolution module structure diagram of the deep neural network for image de-reflection based on pseudo 3D convolution of the application;

[0067] Figure 3 It is a de-reflection result diagram of the application and other de-reflection methods on a real non-polarized data set;

[0068] Figure 4 It is a de-reflection result diagram of the application and other de-reflection methods on a synthetic polarized data set. DETAILED DESCRIPTION

[0069] In this embodiment, a polarized image de-reflection method based on pseudo 3D convolution aims to solve the problems that the existing network lacks a polarized data set and extracts useful information from grouped polarized images (multiple images taken at the same viewing angle and different polarized angles), a polarized image deep neural network based on pseudo 3D convolution is constructed, a de-reflection model capable of effectively de-reflection without specific clues is obtained, so as to improve the quality of images taken in a reflection environment and meet the requirements of pictures required for advanced visual tasks. Specifically, the following steps are performed:

[0070] Step 1, obtaining a comprehensive polarimetric dataset containing real mixed polarimetric images and synthetic mixed polarimetric images;

[0071] Step 1.1, obtaining a real mixed polarimetric image set under K1 static scenes , a real transmission polarimetric image set , and a real reflection polarimetric image set ; wherein, denotes a real mixed polarimetric image with a polarimetric angle of under the k1th static scene; denotes a real transmission polarimetric image with a polarimetric angle of under the k1th static scene; denotes a real reflection polarimetric image with a polarimetric angle of under the k1th static scene;

[0072] Step 1.2, obtaining a synthetic mixed polarimetric image set under another K2 static scenes , a synthetic transmission polarimetric image set , and a synthetic reflection polarimetric image set ; wherein, denotes a synthetic mixed polarimetric image with a polarimetric angle of under the k2th static scene; denotes a synthetic transmission polarimetric image with a polarimetric angle of under the k2th static scene; denotes a synthetic reflection polarimetric image with a polarimetric angle of under the k2th static scene;

[0073] Step 1.3, merging and into a mixed polarimetric dataset ; merging and into a transmission polarimetric dataset ; and merging and into a reflection polarimetric dataset ; wherein, denotes a mixed polarimetric image with a polarimetric angle of under the kth static scene; denotes a transmission polarimetric image with a polarimetric angle of under the kth static scene; denotes a transmission polarimetric image with a polarimetric angle of under the kth static scene;

[0074] for all polarimetric angles under the same scene After averaging pixel by pixel, the mixed unpolarized image of the k-th static scene is obtained. For all polarization angles in the same scene of The average value is calculated pixel by pixel to obtain the transmitted unpolarized image of the k-th static scene. For all polarization angles in the same scene of The average value is calculated pixel by pixel to obtain the unpolarized reflection image of the k-th static scene. In this embodiment, .

[0075] Step 2: Based on the U-Net architecture, construct a polarization dereflection network based on pseudo-3D convolution, including: a feature upscaling module, an encoder, a transport image decoder, and a reflection image decoder. The overall network structure is as follows: Figure 1 As shown;

[0076] Step 2.1, the feature dimensionality enhancement module performs the following steps: } Perform dimensionality upscaling and fusion operations to obtain the 4D feature map of the k-th static scene. In this embodiment, the four dimensions C×P×H×W of the high-dimensional feature map include the number of color channels, polarization angle, image height, and width;

[0077] Step 2.2: The encoder is constructed from M basic convolutional modules, which are then sequentially processed... The process yields M high-frequency feature maps for the k-th static scene. M low-frequency feature maps and M high and low frequency superimposed feature maps ;in, Indicates the first A high-frequency feature map, Indicates the first A low-frequency feature map, This represents the m-th high- and low-frequency superimposed feature map under the k-th static scene. In this embodiment, When m = 1, 2, 3, 4, the number of input and output channels of the m-th basic convolutional module are (3, 32), (32, 64), (64, 128), and (128, 256), respectively. The basic convolutional module is as follows: Figure 2 As shown.

[0078] Step 2.2.1: When m=1, the nonlinear transformation module in the m-th basic convolutional module sequentially uses the first pseudo-3D convolutional layer, the ReLU activation function operation, and the second pseudo-3D convolutional layer to... After performing a nonlinear transformation, the m-th nonlinear 4D feature map with shape C×P×H×W is obtained in the k-th static scene. ; wherein C represents a color channel, P represents a polarization channel, H represents a height, and W represents a width. In the present embodiment, when m = 1, the input and output channel numbers of the first and second pseudo 3D convolution layers of the mth basic convolution module are (3, 32) and (32, 32) respectively, and the convolution kernel size is (1, 3, 3);

[0079] Step 2.2.2, the polarization feature attention module PFAM in the mth basic convolution module sequentially uses a 3D average pooling layer, a third pseudo 3D convolution layer, a ReLU activation function operation, a fourth pseudo 3D convolution layer, and a Sigmoid activation function operation to convert into the mth polarization attention feature map with a shape of C x P x 1 x 1; After pixel-by-pixel multiplication of and , the mth polarization attention weighted feature map under the kth static scene is obtained. In the present embodiment, when m = 1, the input and output channel numbers of the third and fourth pseudo 3D convolution layers of the mth basic convolution module are (32, 4) and (4, 1) respectively, and the convolution kernel size is (3, 1, 1); when m = 2, the input and output channel numbers of the third and fourth pseudo 3D convolution layers of the mth basic convolution module are (64, 8) and (8, 1) respectively, and the convolution kernel size is (3, 1, 1); when m = 3, the input and output channel numbers of the third and fourth pseudo 3D convolution layers of the mth basic convolution module are (128, 16) and (16, 1) respectively, and the convolution kernel size is (3, 1, 1); when m = 4, the input and output channel numbers of the third and fourth pseudo 3D convolution layers of the mth basic convolution module are (256, 32) and (32, 256) respectively, and the convolution kernel size is (3, 1, 1).

[0080] Step 2.2.3, the spatial feature attention module SFAM in the mth basic convolution module sequentially uses a fifth pseudo 3D convolution layer, a ReLU activation function operation, a sixth pseudo 3D convolution layer, and a Sigmoid activation function operation to convert into the mth spatial attention feature map ; After pixel-by-pixel multiplication of and , the mth comprehensive attention weighted feature map under the kth static scene is obtained. In this embodiment, when m=1, the number of input and output channels of the fifth and sixth pseudo-3D convolutional layers of the m-th basic convolutional module are (32,4) and (4,32) respectively, and the kernel size is (1,3,3) for both; when m=2, the number of input and output channels of the fifth and sixth pseudo-3D convolutional layers of the m-th basic convolutional module are (64,8) and (8,64) respectively, and the kernel size is (1,3,3) for both; when m=3, the number of input and output channels of the fifth and sixth pseudo-3D convolutional layers of the m-th basic convolutional module are (128,16) and (16,128) respectively, and the kernel size is (1,3,3) for both; when m=4, the number of input and output channels of the fifth and sixth pseudo-3D convolutional layers of the m-th basic convolutional module are (256,32) and (32,256) respectively, and the kernel size is (1,3,3) for both.

[0081] Step 2.2.4: The octave convolutional residual module OCRM in the m-th basic convolutional module first uses the seventh pseudo-3D convolutional layer and the InstanceNorm3D normalization layer from... Extract the first High and low frequency mixed feature map In this embodiment, when m=1, the number of input / output channels of the seventh pseudo-3D convolutional layer of the m-th basic convolutional module is (8,8), and the kernel size is 1; when m=2, the number of input / output channels of the seventh pseudo-3D convolutional layer of the m-th basic convolutional module is (64,32), and the kernel size is 1; when m=3, the number of input / output channels of the seventh pseudo-3D convolutional layer of the m-th basic convolutional module is (128,64), and the kernel size is 1; when m=4, the number of input / output channels of the seventh pseudo-3D convolutional layer of the m-th basic convolutional module is (256,128), and the kernel size is 1.

[0082] Then, along the polarization channel dimension Decomposed into the first Original high-frequency feature map With the Original low-frequency feature map ;

[0083] right After performing average pooling and pseudo-3D convolution operations sequentially, we obtain the... High-frequency to low-frequency feature map In this embodiment, the size of the pooling window is... The pooling window step size is When m=1,2,3,4, the kernel of the pseudo-3D convolution operation of the m-th basic convolution module is 1, and the number of input and output channels are (8,8), (16,16), (32,32), and (64,64), respectively.

[0084] right After performing pseudo-3D convolution, we obtain the first... High-frequency to high-frequency feature map In this embodiment, when m=1,2,3,4, the kernel of the pseudo-3D convolution operation of the m-th basic convolution module is 1, and the number of input and output channels are (8,8), (16,16), (32,32), and (64,64), respectively.

[0085] right After performing pseudo-3D convolution and upsampling operations sequentially, we obtain the... Low-frequency to high-frequency feature map In this embodiment, when m=1,2,3,4, the kernel of the pseudo-3D convolution operation of the m-th basic convolution module is 1, and the number of input and output channels are (8,8), (16,16), (32,32), and (64,64), respectively; the upsampling magnification factor is (1,2,2), and the algorithm uses the nearest interpolation algorithm;

[0086] right After performing pseudo-3D convolution, we obtain the first... Low-frequency to low-frequency feature map In this embodiment, when m=1,2,3,4, the kernel of the pseudo-3D convolution operation of the m-th basic convolution module is 1, and the number of input and output channels are (8,8), (16,16), (32,32), and (64,64), respectively.

[0087] Will and After adding pixel by pixel, and with Perform pixel-by-pixel multiplication, followed by ReLU activation function operation, to obtain the [then]. High-frequency feature map ;

[0088] Will and After adding pixel by pixel, and with Perform pixel-by-pixel multiplication, followed by ReLU activation, to obtain the [then]. Low-frequency feature map ;

[0089] right After performing pseudo-3D convolution and InstanceNorm3D normalization operations, we obtain the first... Normalized high-frequency feature map In this embodiment, when m=1,2,3,4, the convolution kernel of the pseudo-3D convolutional layer of the m-th basic convolutional module is 1, and the number of input and output channels are (8,32), (16,64), (32,128), and (64,256), respectively.

[0090] After the pseudo 3D convolution operation and the InstanceNorm3D normalization operation are performed on the first low-frequency feature map, the m-th normalized low-frequency feature map is obtained. In the embodiment, when m=1, 2, 3, 4, the convolution kernel of the pseudo 3D convolution layer of the m-th basic convolution module is 1, and the input and output channel numbers are (8, 32), (16, 64), (32, 128), and (64, 256) respectively. After the first low-frequency feature map and the second low-frequency feature map are superimposed in the polarization channel dimension, the m-th high-low frequency superimposed feature map is obtained.

[0091]

[0092] Step 2.2.5, when m=2, 3, …, M, the m-th high-low frequency superimposed feature map is input into the m-th basic convolution module for processing, to obtain the (m+1)-th high-low frequency superimposed feature map, so that the M-th high-low frequency superimposed feature map is output from the M-th basic convolution module.

[0093] In the embodiment, when m=2, the input and output channel numbers of the first and second pseudo 3D convolution layers of the m-th basic convolution module are (32, 64) and (64, 64) respectively, and the convolution kernel size is (1, 3, 3).

[0094] When m=3, the input and output channel numbers of the first and second pseudo 3D convolution layers of the m-th basic convolution module are (64, 128) and (128, 128) respectively, and the convolution kernel size is (1, 3, 3).

[0095] When m=4, the input and output channel numbers of the first and second pseudo 3D convolution layers of the m-th basic convolution module are (128, 256) and (256, 256) respectively, and the convolution kernel size is (1, 3, 3).

[0096] Step 2.3, each decoder is composed of M up-sampling skip connection modules and M 2D convolution modules; wherein each 2D convolution module includes a plurality of 2D units, each 2D unit includes a 2D convolution layer and an activation function; the transmission image decoder processes and the reflection image decoder processes to obtain the restored transmission image under the k-th static scene. ​​​​​​​​​​​​​​​The processing is performed to obtain a restored reflection image under the kth static scene In this embodiment, each 2D convolution module includes 2 2D units, the activation function of the last 2D unit of each decoder is a Sigmoid activation function, and the activation functions of the other 2D units are Sigmoid activation functions and ReLU;

[0097] Step 2.3.1, transmitting image decoder pair and The processing is performed to obtain M 2D high-frequency transmission feature maps ;

[0098] When m = 1, the mth up-sampling skip connection module in the transmitting image decoder is used to process the first 2D high-frequency transmission feature map After 2*2 magnification in the spatial dimension by using the nearest interpolation algorithm, the first 2D high-frequency transmission feature map is connected to the first 2D high-frequency transmission feature map in the transmitting image decoder through the skip connection After splicing in the polarization channel dimension, the mth 2D convolution module of the transmitting image decoder is used for processing, so that the mth 2D high-frequency transmission feature map output by the transmitting image decoder is obtained In this embodiment, when m = 1, the convolution kernel sizes of the two 2D convolution layers of the mth 2D convolution module in the transmitting image decoder are both 3, and the input and output channel numbers are (2048, 128) and (128, 128) respectively.

[0099] When m = 2, 3, …, M-1, the (m-1)th 2D high-frequency transmission feature map is input into the mth up-sampling skip connection module of the transmitting image decoder, and the (m-1)th 2D high-frequency transmission feature map is magnified by 2*2 in the spatial dimension by using the nearest interpolation algorithm , and then connected to the (m-1)th 2D high-frequency transmission feature map in the transmitting image decoder through the skip connection After splicing in the polarization channel dimension, the mth 2D convolution module is used for processing, so that the mth 2D high-frequency transmission feature map output by the transmitting image decoder is obtained ; and the (M-1)th 2D high-frequency transmission feature map is output by the (M-1)th 2D convolution module of the transmitting image decoder In this embodiment, when m = 2, the convolution kernel sizes of the two 2D convolution layers of the mth 2D convolution module in the transmitting image decoder are both 3, and the input and output channel numbers are (640, 64) and (64, 64) respectively; when m = 3, the convolution kernel sizes of the two 2D convolution layers of the mth 2D convolution module in the transmitting image decoder are both 3, and the input and output channel numbers are (320, 32) and (32, 32) respectively.

[0100] When m = M, the (M-1)th 2D high-frequency transmission feature map In the mth 2D convolution upsampling module of the transmission image decoder, the nearest interpolation algorithm is adopted in the spatial dimension, and the 2×2 magnification is performed on After the jump connection with After splicing in the polarization channel dimension, the mth 2D high-frequency transmission feature map output by the transmission image decoder is obtained through the processing of the mth 2D convolution module of the transmission image decoder , that is, the transmission image is restored. In this embodiment, when m=4, the kernel size of the two 2D convolution layers in the mth 2D convolution module in the transmission image decoder is 3, and the input and output channel numbers are (160, 3) and (3, 3) respectively.

[0101] Step 2.3.2, the reflection image decoder processes and to obtain M 2D low-frequency reflection feature maps ;

[0102] When m=1, the mth upsampling jump connection module in the reflection image decoder performs 2×2 magnification on in the spatial dimension by using the nearest interpolation algorithm, and then performs the jump connection with After splicing in the polarization channel dimension, the mth 2D low-frequency reflection feature map output by the reflection image decoder is obtained through the processing of the mth 2D convolution module of the reflection image decoder . In this embodiment, when m=1, the kernel size of the two 2D convolution layers in the mth 2D convolution module in the transmission image decoder is 3, and the input and output channel numbers are (2048, 128) and (128, 128) respectively.

[0103] When m=2, 3, …, M-1, the m-1th 2D low-frequency reflection feature map is input into the mth upsampling jump connection module of the reflection image decoder, and the nearest interpolation algorithm is adopted in the spatial dimension to perform 2×2 magnification on , and then perform the jump connection with After splicing in the polarization channel dimension, the mth 2D low-frequency reflection feature map output by the reflection image decoder is obtained through the processing of the mth 2D convolution module of the reflection image decoder ; and the M-1th 2D low-frequency reflection feature map is output from the M-1th 2D convolution module of the reflection image decoder In this embodiment, when m=2, the convolution kernel size of the two 2D convolution layers in the mth 2D convolution module in the transmission image decoder is 3, and the input and output channel numbers are (640, 64) and (64, 64) respectively; when m=3, the convolution kernel size of the two 2D convolution layers in the mth 2D convolution module in the transmission image decoder is 3, and the input and output channel numbers are (320, 32) and (32, 32) respectively.

[0104] When m=M, the (M-1)th 2D low-frequency reflection feature map is input into the Mth upsampling skip connection of the reflection image decoder, and the nearest interpolation algorithm is used in the spatial dimension to perform 2x2 upscaling, and then the (M-1)th 2D low-frequency reflection feature map is obtained through the jump connection with After splicing in the polarization channel dimension, the (M-1)th 2D low-frequency reflection feature map is processed by the mth 2D convolution module of the reflection image decoder, and the Mth 2D low-frequency reflection feature map output by the reflection image decoder is obtained , that is, the reflection image is restored. In this embodiment, when m=4, the convolution kernel size of the two 2D convolution layers in the mth 2D convolution module in the transmission image decoder is 3, and the input and output channel numbers are (160, 3) and (3, 3) respectively.

[0105] Step 3, training a polarized image anti-reflection model based on pseudo 3D convolution;

[0106] Step 3.1, constructing a pseudo 3D convolution based on , and , and constructing a total loss function L using formula (1);

[0107] (1)

[0108] In formula (1), the reconstruction loss is represented by , the perceptual loss is represented by , the repulsion loss is represented by , and the adversarial loss is represented by ; λ1, λ2, and λ3 are the weights of the perceptual loss , the adversarial loss , and the repulsion loss , respectively.

[0109] The reconstruction loss is constructed using formula (2):

[0110] (2)

[0111] In formula (2), is the absolute value distance, is the gradient with respect to the image horizontal coordinate x, and for the gradient with respect to the image ordinate y;

[0112] The polarized image de-reflectance network based on pseudo 3D convolution is used as the generator, and a discriminator is constructed. The image x is input into the discriminator for processing in the middle to obtain the discriminator mapping value of the image x between 0 and 1 ;

[0113] The generator loss in the adversarial loss is constructed by using formula (5) and formula (6) respectively and the discriminator loss :

[0114] (5)

[0115] (6)

[0116] In formula (5) and formula (6), is the discriminator mapping value of , and is the discriminator mapping value of .

[0117] Step 3.2, the polarized image de-reflectance network based on pseudo 3D convolution is trained by using the ADAM optimizer, and the total loss function L is calculated to update the network parameters. In the training process of the network, the discriminator and the generator are updated alternately. First, the generator is fixed, the discriminator is updated to minimize the discriminator loss ; then, the discriminator is fixed, the generator is updated to minimize the generator loss , and the training is performed until the total loss function converges, so as to obtain the optimal de-reflectance model based on pseudo 3D convolution using polarized images, which is used for de-reflectance processing of the synthesized polarized mixed image and the actually shot polarized mixed image. In this embodiment, in the training stage, the network is trained for 120 epochs, the initial learning rate is set to ; after the 20th epoch, the adversarial loss is added, after the 60th epoch, the learning rate is decayed to ; after the 80th epoch, the learning rate is decayed to ; after the 100th epoch, the learning rate is decayed to .

[0118] In this embodiment, 8925 synthesized images of 4 different polarization angles (0°, 45°, 90° and 135°) are used polarization mixed images of 8925 real captured polarization mixed images of 4 different polarization angles (0°, 45°, 90°, 135°) and their corresponding transmission images, reflection images constitute a polarization dataset for training, in the training process, the input polarization image size is randomly cropped to 352x352, the transmission image output by the pseudo 3D convolution based defogging model is used as the ground truth to calculate the loss value and the reflection image The total loss is calculated, and the calculated loss is used to guide the network training with the ADAM optimizer to obtain the pseudo 3D convolution based defogging model using polarization images.

[0119] In this embodiment, an electronic device includes a memory for storing a program supporting the processor to execute the above method, and a processor configured to execute the program stored in the memory.

[0120] In this embodiment, a computer readable storage medium has a computer program stored thereon, and the computer program performs the steps of the above method when executed by a processor.

[0121] Table 1 is a comparison of the pseudo 3D convolution based polarization image defogging method of the present application with other defogging methods in terms of peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) as evaluation indicators. In order to compare fairly, all defogging methods are retrained on the dataset, and in the test process, the present application and other defogging methods use polarization images and ordinary RGB images respectively. PSNR is the ratio of the maximum power of a signal to the power of noise that may affect its representation accuracy, and the larger the value, the smaller the distortion of the image defogging. SSIM is the similarity of an image in brightness, contrast and structure, and its value range is [0, 1], and the larger the value, the more similar the transmission image is to the real transmission image. According to the quantitative analysis of Table 1, it can be seen that the method of the present application achieves the best or second best effect in both indicators.

[0122] Table 1

[0123]

[0124] Figure 3Results of the present pseudo-3D convolution based polarized image reflection removal method and other current reflection removal methods on real non-polarized dataset. Among them, Ours represents the present pseudo-3D convolution based polarized image reflection removal method; the IBCLN method in Single image reflection removal through cascaded refinement proposes an iterative boosting convolution LSTM network, which refines the estimation of the transmission layer and the reflection layer through an iterative manner, uses the LSTM in the cascaded step to pass information, and solves the gradient vanishing problem; the ERRNet proposed in Single image reflection removal exploiting misaligned training data and network enhancements enhances the baseline network architecture by embedding context encoding modules, which can reduce uncertainty in strong reflection regions by using advanced context clues; the BDN method in Seeing deeply and bidirectionally: A deep learning approach for single image reflection removal proposes a cascaded deep neural network to estimate the background image and the reflection layer simultaneously, which significantly improves the quality of reflection removal; Wen et al. propose a deep synthesis network (SynNet) in Single image reflection removal beyond linearity to predict a nonlinear Alpha blending mask for two input images to simulate different types of reflections; Dong et al. propose a location-aware single image reflection removal method in Location-aware single image reflection removal, which uses a network with a reflection detection module that can regress a probability reflection confidence map (RCMap) and use it as a clue to control the feature flow when predicting the reflection and transmission layers.

[0125] Figure 4 Results of the present pseudo-3D convolution based polarized image reflection removal method and other current reflection removal methods on synthetic polarized dataset.

Claims

1. A method for polarization image dereflection based on pseudo-3D convolution, characterized in that, Follow these steps: Step 1: Obtain a comprehensive polarization dataset containing real mixed polarization images and synthetic mixed polarization images; Step 1.1: Obtain a set of real mixed polarization images for K1 static scenes. Real transmission polarization image set and real reflection polarization image set ;in, The polarization angle in the k1th static scene is . A true mixed polarization image; The polarization angle in the k1th static scene is . The true transmission polarization image; The polarization angle in the k1th static scene is . The true reflection polarization image; Step 1.2: Obtain a set of composite mixed polarization images from another K2 static scenes. Synthetic transmission polarization image set and synthesized reflection polarization image set ;in, The polarization angle in the k2th static scene is . Synthetic mixed polarization images; The polarization angle in the k2th static scene is . Synthesized transmission polarization image; The polarization angle in the k2th static scene is . Synthetic reflection polarization image; Step 1.3, and Merged into a hybrid polarization dataset ;Will and Merged into a transmission polarization dataset ;Will and Merged into a reflection polarization dataset ;in, The polarization angle in the k-th static scene is . Mixed polarization images; The polarization angle in the k-th static scene is . Transmission polarization image; The polarization angle in the k-th static scene is . The reflected polarization image; For all polarization angles in the same scene of After averaging pixel by pixel, the mixed unpolarized image of the k-th static scene is obtained. For all polarization angles in the same scene of The average value is calculated pixel by pixel to obtain the transmitted unpolarized image of the k-th static scene. For all polarization angles in the same scene of The average value is calculated pixel by pixel to obtain the unpolarized reflection image of the k-th static scene. ; Step 2: Based on the U-Net architecture, construct a polarization dereflection network based on pseudo-3D convolution, which includes: a feature upscaling module, an encoder, a transmission image decoder, and a reflection image decoder; Step 2.1, the feature dimensionality enhancement module performs the following steps: } Perform dimensionality upscaling and fusion operations to obtain the 4D feature map of the k-th static scene. ; Step 2.2: The encoder consists of M basic convolutional modules, which are sequentially processed... The process yields M high-frequency feature maps for the k-th static scene. M low-frequency feature maps and M high and low frequency superimposed feature maps ;in, Indicates the first A high-frequency feature map, Indicates the first A low-frequency feature map This represents the m-th high- and low-frequency superimposed feature map in the k-th static scene; Step 2.3: Transmit the image decoder pair and Processing is performed to obtain the restored transmitted image of the k-th static scene. ; Reflection image decoder pair and Processing is performed to obtain the restored reflection image of the k-th static scene. ; Step 3: Train a polarization image dereflection model based on pseudo-3D convolution; Step 3.1, based on , and The total loss function L is constructed using equation (1); (1) In equation (1), Indicates the losses incurred during reconstruction. Indicates perceived loss, Indicates rejection of loss, Let λ1, λ2, and λ3 represent the adversarial loss, and λ3 represent the perceptual loss, respectively. Rejection loss and combat losses The weights; Step 3.2: Train the pseudo-3D convolution-based polarization image dereflection network using the ADAM optimizer. During training, first fix the generator and then update the discriminator to minimize the discriminator loss. Then, fix the discriminator and update the generator to minimize the generator's adversarial loss. The discriminator and generator are updated alternately until the total loss function L converges, thus obtaining the optimal pseudo-3D convolution-based dereflection model using polarized images, which is used to dereflect synthesized polarized mixed images and real-world polarized mixed images.

2. The polarization image dereflection method based on pseudo-3D convolution according to claim 1, characterized in that, Each basic convolutional module in step 2.2 consists of a nonlinear transformation module, a polarization feature attention module (PFAM), a spatial feature attention module (SFAM), and an octave convolutional residual module (OCRM). Step 2.2.1: When m=1, the nonlinear transformation module in the m-th basic convolutional module sequentially uses the first pseudo-3D convolutional layer, the ReLU activation function operation, and the second pseudo-3D convolutional layer to... After performing a nonlinear transformation, the m-th nonlinear 4D feature map with shape C×P×H×W is obtained in the k-th static scene. Where C represents the color channel, P represents the polarization channel, H represents the height, and W represents the width; Step 2.2.2: The polarization feature attention module in the m-th basic convolutional module sequentially uses a 3D average pooling layer, a third pseudo-3D convolutional layer, a ReLU activation function operation, a fourth pseudo-3D convolutional layer, and a Sigmoid activation function operation to... Transform into the shape of C×P×1×1. Polarization attention feature map Then and After performing pixel-by-pixel multiplication, we obtain the k-th static scene. Polarization attention-weighted feature map ; Step 2.2.3: The spatial feature attention module in the m-th basic convolutional module sequentially uses the fifth pseudo-3D convolutional layer, ReLU activation function operation, sixth pseudo-3D convolutional layer, and Sigmoid activation function operation to... Convert to the first Spatial attention feature map Then and After performing pixel-by-pixel multiplication, we obtain the k-th static scene. A comprehensive attention-weighted feature map ; Step 2.2.4: The octave convolutional residual module in the m-th basic convolutional module first uses the seventh pseudo-3D convolutional layer and the InstanceNorm3D normalization layer from... Extract the first High and low frequency mixed feature map ; Then, along the polarization channel dimension Decomposed into the first Original high-frequency feature map With the Original low-frequency feature map ; right After performing average pooling and pseudo-3D convolution operations sequentially, we obtain the... High-frequency to low-frequency feature map ; right After performing pseudo-3D convolution, we obtain the first... High-frequency to high-frequency feature map ; right After performing pseudo-3D convolution and upsampling operations sequentially, we obtain the... Low-frequency to high-frequency feature map ; right After performing pseudo-3D convolution, we obtain the first... Low-frequency to low-frequency feature map ; Will and After adding pixel by pixel, and with Perform pixel-by-pixel multiplication, followed by ReLU activation function operation, to obtain the [then]. High-frequency feature map ; Will and After adding pixel by pixel, and with Perform pixel-by-pixel multiplication, followed by ReLU activation, to obtain the [then]. Low-frequency feature map ; right After performing pseudo-3D convolution and InstanceNorm3D normalization operations, we obtain the first... Normalized high-frequency feature map ; right After performing pseudo-3D convolution and InstanceNorm3D normalization operations, we obtain the first... Normalized low-frequency feature map ; Will and After superimposing along the polarization channel dimension, we obtain the first... High and low frequency superimposed feature map ; Step 2.2.5: When m = 2, 3, ..., M, the first... High and low frequency superimposed feature map The m-th basic convolutional module is input and processed to obtain the m-th... High and low frequency superimposed feature map Thus, the Mth basic convolutional module outputs the Mth high- and low-frequency superimposed feature map. .

3. The polarization image dereflection method based on pseudo-3D convolution according to claim 2, characterized in that, The transmission image decoder and reflection image decoder in step 2.3 are each composed of M upsampling skip connection modules and M 2D convolution modules; Step 2.3.1: When m=1, the m-th upsampling skip connection module of the transmission image decoder performs spatial dimension operations. After magnification, jump connections are made with... After being stitched together along the polarization channel dimension, the m-th 2D convolutional module of the transmission image decoder processes the data to obtain the m-th 2D high-frequency transmission feature map. ; When m=2,3,…,M-1, the (m-1)th 2D high-frequency transmission feature map will be... In the m-th upsampling skip connection module of the input transmission image decoder, and in the spatial dimension... After magnification, jump connections are made with... After being stitched together along the polarization channel dimension, the m-th 2D convolutional module of the transmission image decoder is used to obtain the m-th 2D high-frequency transmission feature map. Thus, the (M-1)th 2D convolutional module of the transmission image decoder outputs the (M-1)th 2D high-frequency transmission feature map. ; When m=M, the (M-1)th 2D high-frequency transmission feature map In the Mth upsampling skip connection module of the input transmission image decoder, and in the spatial dimension... After magnification, jump connections are made with... After being stitched along the polarization channel dimension, the image is processed by the m-th 2D convolutional module of the transmission image decoder to obtain the M-th 2D high-frequency transmission feature map output by the transmission image decoder. And as a way to restore the transmitted image; Step 2.3.2: When m=1, the m-th upsampling skip connection module in the reflection image decoder performs spatial dimension... After magnification, jump connections are made with... After being stitched along the polarization channel dimension, the m-th 2D convolutional module in the reflection image decoder is used to obtain the m-th 2D low-frequency reflection feature map. ; When m=2,3,…,M-1, the (m-1)th 2D low-frequency reflection feature map In the m-th upsampled skip connection module of the input reflection image decoder, and in the spatial dimension... After magnification, jump connections are made with... After stitching along the polarization channel dimension, the m-th 2D convolutional module in the reflection image decoder is used to obtain the m-th 2D low-frequency reflection feature map. Thus, the (M-1)th 2D convolutional module of the reflection image decoder outputs the (M-1)th 2D low-frequency reflection feature map. ; When m=M, the (M-1)th 2D low-frequency reflection feature map In the Mth upsampled skip connection module of the input reflection image decoder, and in the spatial dimension... After magnification, jump connections are made with... After stitching along the polarization channel dimension, the image is processed by the m-th 2D convolutional module in the reflection image decoder to obtain the M-th 2D low-frequency reflection feature map output by the reflection image decoder. And as a way to recover the reflected image.

4. An electronic device, comprising a memory and a processor, characterized in that, The memory is used to store a program that supports the processor in executing any of the polarization image dereflection methods of claims 1-3, and the processor is configured to execute the program stored in the memory.

5. A computer-readable storage medium storing a computer program thereon, characterized in that, The computer program, when run by a processor, performs the steps of any of the polarization image dereflection methods described in claims 1-3.

Images