Polarization de-reflection method based on pseudo 3D convolution

Abstract

The invention discloses a polarization dereflection method based on pseudo 3D convolution, and the method comprises the steps: 1, constructing a real-world polarization dereflection data set of real shooting, and obtaining a synthesized polarization dereflection data set through employing a polarization imaging principle and an optical theory; 2, constructing a deep convolutional neural network using polarization images based on pseudo 3D convolution, and training the deep convolutional neural network by taking a plurality of polarization images with different polarization angles as input to obtain a multi-image de-reflection model; and 3, carrying out de-reflection processing on a polarization image to be mixed by utilizing the trained model to obtain a recovered transmission image. According to the invention, polarization de-reflection based on pseudo 3D convolution can be realized, so that the de-reflection effect in a complex and changeable scene can be effectively improved, and clearer images can be provided for many advanced visual tasks.

Description

Technical Field

[0001] The invention belongs to the fields of computer vision, image processing and analysis, and in particular is a polarization de-reflection method based on pseudo 3D convolution. Background Art

[0002] Nowadays, with the rapid update and iteration of various digital products and shooting equipment, in many cases in photography, science and computer vision imaging systems, it is inevitable to capture images through glass windows or transparent shells, such as photographing objects behind glass in museums or photographing goods in windows. These images often contain scenes reflected by glass. The bad reflections in these images will reduce the visual effect and quality of the images, and will also affect the performance of subsequent computer vision algorithms. Since clear transmission scenes can be restored by eliminating these reflection scenes, image de-reflection and separation of reflection and transmission images are hot research topics in the fields of computer photography and computer vision.

[0003] Image de-reflection algorithms are mainly used in computer vision and image processing to improve image quality and accuracy. For example, when shooting indoors, due to the reflection and refraction of light, obvious reflected images and highlight areas will appear in the image, affecting the clarity and color reproduction of the image. Image de-reflection algorithms can effectively eliminate these reflections and highlights, making the image more realistic and clear, and helping to improve the accuracy of image recognition and the effectiveness of image analysis.

[0004] In recent years, image dereflection has attracted more and more attention from researchers, and many image dereflection models with good performance have been proposed. At present, the existing frameworks can be roughly divided into two categories: traditional dereflection methods based on artificial prior knowledge and dereflection methods based on deep learning algorithms. Traditional dereflection methods rely on prior knowledge such as gradient sparsity and gradient independence, and use mathematical models to separate reflection and transmission images. The famous one is the image dereflection algorithm based on independent component analysis proposed by Farid H. et al. Although traditional dereflection methods have made some progress, these assumptions and prior knowledge are all for specific scenes, so the generalization ability is limited, that is, once the environment changes significantly, the dereflection ability of the model will be significantly reduced. The deep learning-based method trains a dereflection model through a large amount of training data, and uses the trained model to test on the test data. The deep learning-based algorithm shows strong modeling and understanding capabilities, and can detect various reflection image features. The deep learning-based method can also be divided into two categories, one is single-image dereflection that uses only one mixed image to separate the transmission image and the reflection image, and the other is multi-image dereflection that uses multiple mixed images to separate the transmission image and the reflection image.

[0005] However, these learning-based methods still have shortcomings:

[0006] 1. The single image de-reflection method cannot handle complex specular reflections because it only has one image as input. In addition, for strong reflections, the effect of the single image de-reflection method may be poor;

[0007] 2. Most multi-image dereflection methods rely on specific conditions (such as near-infrared light, depth map, etc.), and capturing multiple images under these specific constraints is often complex and time-consuming;

[0008] 3. Currently, many deep learning-based algorithms have turned to synthetic datasets for training. However, existing synthesis methods mainly rely on simple linear synthesis to generate mixed images, which is significantly different from the real imaging process. This leads to poor performance of network models when processing real-world reflection data, poor generalization ability, and unsatisfactory de-reflection effects. Summary of the invention

[0009] In order to address the deficiencies in the prior art, the present invention provides a polarization de-reflection method based on pseudo 3D convolution, in order to improve the quality of captured reflected images, transmitted images, and mixed images, and to improve the de-reflection effect in complex and changing scenes, thereby meeting the requirements of images required for advanced visual tasks and providing clearer images for many advanced visual tasks.

[0010] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0011] The polarization image dereflection method based on pseudo 3D convolution of the present invention is characterized in that it is performed according to the following steps:

[0012] Step 1, obtaining a comprehensive polarization dataset including a real mixed polarization image and a synthetic mixed polarization image;

[0013] Step 1.1: Obtain a set of real mixed polarization images in K1 static scenes , real transmission polarization image set and the real reflection polarization image set ;in, Indicates that the polarization angle in the k1th static scene is True mixed polarization image of Indicates that the polarization angle in the k1th static scene is The real transmission polarization image of Indicates that the polarization angle in the k1th static scene is The real reflected polarization image of

[0014] Step 1.2: Obtain another K2 sets of synthetic mixed polarization images under static scenes , Synthetic transmission polarization image set and synthetic reflected polarization image sets ;in, Indicates that the polarization angle in the k2th static scene is The synthetic mixed polarization image of Indicates that the polarization angle in the k2th static scene is The synthetic transmission polarization image of Indicates that the polarization angle in the k2th static scene is The synthetic reflected polarization image of

[0015] Step 1.3: and Merge into a hybrid polarization dataset ;Will and Merge into transmission polarization dataset ;Will and Merge into reflectance polarization dataset ;in, Indicates that the polarization angle in the kth static scene is Mixed polarization images of Indicates that the polarization angle in the kth static scene is Transmission polarization image of Indicates that the polarization angle in the kth static scene is Transmission polarization image of

[0016] For all polarization angles in the same scene of After averaging pixel by pixel, the mixed non-polarized image in the kth static scene is obtained. ; For all polarization angles in the same scene of Calculate the average pixel by pixel to obtain the transmitted non-polarized image in the kth static scene ; For all polarization angles in the same scene of Calculate the average pixel by pixel to obtain the reflected non-polarized image in the kth static scene ;

[0017] Step 2: Based on the U-Net architecture, a polarization de-reflection network based on pseudo 3D convolution is constructed, which includes: a feature dimension-enhancing module, an encoder, a transmission image decoder, and a reflection image decoder;

[0018] Step 2.1: The feature dimension-upgrading module } Perform dimension upgrading and fusion operations to obtain the 4-dimensional feature map of the kth static scene ;

[0019] Step 2.2: Constructing the encoder is composed of M basic convolutional modules, and sequentially Processing is performed to obtain M high-frequency feature maps in the kth static scene , M low-frequency feature maps And M high and low frequency superposition feature maps ;in, Indicates A high-frequency feature map, Indicates A low-frequency feature map, Represents the mth high- and low-frequency superimposed feature map in the kth static scene;

[0020] Step 2.3, transmit image decoder pair and Processing is performed to obtain the restored transmission image in the kth static scene ;

[0021] Reflection Image Decoder and Processing is performed to obtain the restored reflection image in the kth static scene ;

[0022] Step 3: training a polarization image de-reflection model based on pseudo 3D convolution;

[0023] Step 3.1: Build based on , and , use formula (1) to construct the total loss function L:

[0024] (1)

[0025] In formula (1), represents the reconstruction loss, Represents perceived loss, represents the rejection loss and represents adversarial loss, λ1, λ2, and λ3 are perceptual losses respectively. , Fighting Losses and rejection loss The weight of

[0026] Step 3.2: Use the ADAM optimizer to train the pseudo 3D convolution-based polarization image dereflection network, and during the training process, first fix the generator and update the discriminator to minimize the discriminator loss. , then fix the discriminator and update the generator to minimize the generator loss , thereby alternately updating the discriminator and the generator until the total loss function L converges, thereby obtaining the optimal pseudo-3D convolution-based de-reflection model using polarization images, which is used to de-reflect the synthesized polarization mixed images and the real polarization mixed images.

[0027] The characteristic of the polarization image dereflection method based on pseudo 3D convolution described in the present invention is that each basic convolution module in step 2.2 is composed of a nonlinear transformation module, a polarization feature attention module PFAM, a spatial feature attention module SFAM and an octave convolution residual module OCRM;

[0028] Step 2.2.1, when m=1, the nonlinear transformation module in the mth basic convolution module uses the first pseudo 3D convolution layer, the ReLU activation function operation and the second pseudo 3D convolution layer to After nonlinear transformation, the mth nonlinear 4-dimensional feature map with the shape of C×P×H×W is obtained in the kth static scene. ; Wherein, C represents the color channel, P represents the polarization channel, H represents the height, and W represents the width;

[0029] Step 2.2.2, the polarization feature attention module in the mth basic convolution module uses the 3D average pooling layer, the third pseudo 3D convolution layer, the ReLU activation function operation, the fourth pseudo 3D convolution layer and the Sigmoid activation function operation in sequence to Transformed into a shape of C×P×1×1 Polarized attention feature map ; then and After pixel-by-pixel multiplication, we get the kth static scene Polarized attention weighted feature map ;

[0030] Step 2.2.3, the spatial feature attention module in the mth basic convolution module uses the fifth pseudo 3D convolution layer, ReLU activation function operation, the sixth pseudo 3D convolution layer and Sigmoid activation function operation in turn to Convert to Spatial attention feature map , and then and After pixel-by-pixel multiplication, we get the kth static scene Comprehensive attention weighted feature map ;

[0031] Step 2.2.4, the octave convolution residual module in the mth basic convolution module first uses the seventh pseudo 3D convolution layer and the InstanceNorm3D normalization layer from Extract the A high- and low-frequency mixed feature map ;

[0032] Then, along the polarization channel dimension, Decompose into Original high-frequency feature map With Original low-frequency feature map ;

[0033] right After performing average pooling and pseudo 3D convolution operations in sequence, we get High-frequency to low-frequency feature map ;

[0034] right After performing the pseudo 3D convolution operation, we get High-frequency to high-frequency feature map ;

[0035] right After performing pseudo 3D convolution and upsampling operations in sequence, we get Low-frequency to high-frequency feature map ;

[0036] right After pseudo 3D convolution, we get Low-frequency to low-frequency feature map ;

[0037] Will and After adding pixel by pixel, Perform pixel-by-pixel multiplication and then perform the ReLU activation function operation to obtain the High frequency feature map ;

[0038] Will and After adding pixel by pixel, Perform pixel-by-pixel multiplication and then perform ReLU function activation operation to obtain the first Low frequency feature map ;

[0039] right After performing pseudo 3D convolution and InstanceNorm3D normalization, we get Normalized high-frequency feature map ;

[0040] right After performing pseudo 3D convolution and InstanceNorm3D normalization, we get Normalized low-frequency feature map ;

[0041] Will and After superposition in the polarization channel dimension, we get High and low frequency superposition feature map ;

[0042] Step 2.2.5: When m=2,3,…,M, High and low frequency superposition feature map Input the mth basic convolution module for processing and get the mth High and low frequency superposition feature map , so that the Mth basic convolution module outputs the Mth high- and low-frequency superimposed feature map .

[0043] Furthermore, the transmission image decoder and the reflection image decoder in step 2.3 are both composed of M upsampling skip connection modules and M 2D convolution modules;

[0044] Step 2.3.1, when m=1, the mth upsampling skip connection module of the transmission image decoder is After amplification, the jump connection After splicing in the polarization channel dimension, it is processed by the mth 2D convolution module of the transmission image decoder to obtain the mth 2D high-frequency transmission feature map ;

[0045] When m=2,3,…,M-1, the m-1th 2D high frequency transmission feature map The input is sent to the mth upsampling skip connection module of the image decoder and is processed in the spatial dimension. After amplification, the jump connection After splicing in the polarization channel dimension, the mth 2D high-frequency transmission feature map is obtained by processing the mth 2D convolution module of the transmission image decoder. ; Thus, the M-1th 2D convolution module of the transmission image decoder outputs the M-1th 2D high-frequency transmission feature map ;

[0046] When m=M, the M-1th 2D high-frequency transmission feature map The input is sent to the Mth upsampling skip connection module of the image decoder and is processed in the spatial dimension. After amplification, the jump connection After splicing in the polarization channel dimension, the Mth 2D high-frequency transmission feature map output by the transmission image decoder is obtained after being processed by the mth 2D convolution module of the transmission image decoder. , and used as the restored transmission image;

[0047] Step 2.3.2, when m = 1, the mth upsampling skip connection module in the reflected image decoder is After amplification, the jump connection After splicing in the polarization channel dimension, it is processed by the mth 2D convolution module in the reflection image decoder, so that the mth 2D low-frequency reflection feature map ;

[0048] When m=2,3,…,M-1, the m-1th 2D low-frequency reflection feature map Input the mth upsampling skip connection module of the reflected image decoder and perform After amplification, the jump connection After splicing in the polarization channel dimension, the mth 2D low-frequency reflection feature map is obtained by processing the mth 2D convolution module in the reflection image decoder. ; Thus, the M-1th 2D convolution module of the reflection image decoder outputs the M-1th 2D low-frequency reflection feature map ;

[0049] When m=M, the M-1th 2D low-frequency reflection feature map The input is fed into the Mth upsampling skip connection module of the reflected image decoder and is processed in the spatial dimension. After amplification, the jump connection After splicing in the polarization channel dimension, it is processed by the mth 2D convolution module in the reflection image decoder to obtain the Mth 2D low-frequency reflection feature map output by the reflection image decoder. , and used as the restored reflection image.

[0050] Furthermore, in step 3.1, the reconstruction loss is constructed using formula (2): :

[0051] (2)

[0052] In formula (2), is the absolute value distance, is the gradient about the image horizontal coordinate x, is the gradient about the image ordinate y;

[0053] Use formula (3) to construct the perceptual loss :

[0054] (3)

[0055] Using formula (4) to construct the exclusion loss :

[0056] (4)

[0057] In formula (4), and Respectively express and Carry on 2 n The feature map obtained after downsampling, tanh is the hyperbolic tangent activation function, λ T and λ R are 2 normalization factors,  represents element-by-element multiplication;

[0058] The polarization de-reflection network based on pseudo 3D convolution is used as the generator G, and the loss of the generator G in the adversarial loss is constructed using equations (5) and (6) respectively. and the loss of the discriminator D :

[0059] (5)

[0060] (6)

[0061] In formula (5) and formula (6), for The mapping value of the discriminator D, for The mapping value of the discriminator D.

[0062] An electronic device of the present invention includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the polarization de-reflection method, and the processor is configured to execute the program stored in the memory.

[0063] The present invention provides a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and the computer program executes the steps of the polarization dereflection method when the computer program is executed by a processor.

[0064] Compared with the prior art, the present invention has the following beneficial effects:

[0065] 1. The present invention constructs a polarization image deep neural network based on pseudo 3D convolution, uses polarization images with different polarization angles as input, and combines it with a pseudo 3D convolution encoder to solve the problem that the traditional multi-image de-reflection network only uses 2D convolution for model design and ignores the correlation between grouped images, thereby improving the quality of the restored reflection and transmission images.

[0066] 2. The polarization image deep neural network based on pseudo 3D convolution constructed by the present invention introduces polarization information and uses multiple polarization images with different polarization angles in the same scene to obtain richer scene information. It solves the problem that the traditional single image de-reflection network uses a single image as input and relies on the image features extracted from the training data, resulting in poor generalization ability. This improves the robustness of the de-reflection network and can adapt to complex and changeable reflection environments.

[0067] 3. The polarization image deep neural network based on pseudo 3D convolution constructed by the present invention introduces an octave convolution residual module based on local residual operation and octave convolution to enhance the association of contextual information, so that the two prediction streams can pay more attention to the feature information that the current prediction stream needs to pay attention to. At the same time, the feature map output by the convolution layer is decomposed into features of different spatial frequencies, which reduces memory and computing costs, and enables the model to more efficiently learn important information at different frequencies in the image, thereby improving the generalization ability of image de-reflection.

[0068] 4. The polarization image deep neural network based on pseudo 3D convolution constructed by the present invention introduces a spatial feature attention module and a polarization feature attention module based on the attention mechanism. The attention is focused on the strong reflection area in a targeted manner. In this way, the changes in the reflection scene can be handled more effectively, and the accuracy and robustness of image processing can be improved; at the same time, effective feature information can be extracted from the redundant polarization information to improve the quality of the restored image.

[0069] 5. The polarization image deep neural network based on pseudo 3D convolution constructed by the present invention aggregates the feature map output by each layer of the encoder with the feature map of the corresponding decoder through jump connections to obtain a more refined prediction result, thereby improving the accuracy of image de-reflection. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 This is a schematic diagram of the structure of the image de-reflection deep neural network based on pseudo 3D convolution of the present invention;

[0071] Figure 2 This is a schematic diagram of the basic convolution module structure of the image de-reflection deep neural network based on pseudo 3D convolution of the present invention;

[0072] Figure 3It is a de-reflection result diagram of the present invention and other de-reflection methods on a real non-polarized data set;

[0073] Figure 4 The figure is a diagram of the dereflection results of the present invention and other dereflection methods on a synthetic polarization data set. DETAILED DESCRIPTION

[0074] In this embodiment, a polarization image dereflection method based on pseudo 3D convolution aims to solve the problem that the existing network lacks polarization data sets and extracts useful information from grouped polarization images (multiple images taken at different polarization angles of the same viewing angle). By constructing a polarization image deep neural network based on pseudo 3D convolution, a dereflection model that can effectively dereflect without specific clues is obtained, thereby improving the quality of images taken in a reflective environment and meeting the requirements of images required for advanced visual tasks. Specifically, the following steps are followed:

[0075] Step 1, obtaining a comprehensive polarization dataset including a real mixed polarization image and a synthetic mixed polarization image;

[0076] Step 1.1: Obtain a set of real mixed polarization images in K1 static scenes , real transmission polarization image set and the real reflection polarization image set ;in, Indicates that the polarization angle in the k1th static scene is True mixed polarization image of Indicates that the polarization angle in the k1th static scene is The real transmission polarization image of Indicates that the polarization angle in the k1th static scene is The real reflected polarization image of

[0077] Step 1.2: Obtain another K2 sets of synthetic mixed polarization images under static scenes , Synthetic transmission polarization image set and synthetic reflected polarization image sets ;in, Indicates that the polarization angle in the k2th static scene is The synthetic mixed polarization image of Indicates that the polarization angle in the k2th static scene is The synthetic transmission polarization image of Indicates that the polarization angle in the k2th static scene is The synthetic reflected polarization image of

[0078] Step 1.3: and Merge into a hybrid polarization dataset ;Will and Merge into transmission polarization dataset ;Will and Merge into reflectance polarization dataset ;in, Indicates that the polarization angle in the kth static scene is Mixed polarization images of Indicates that the polarization angle in the kth static scene is Transmission polarization image of Indicates that the polarization angle in the kth static scene is Transmission polarization image of

[0079] For all polarization angles in the same scene of After averaging pixel by pixel, the mixed non-polarized image in the kth static scene is obtained. ; For all polarization angles in the same scene of Calculate the average pixel by pixel to obtain the transmitted non-polarized image in the kth static scene ; For all polarization angles in the same scene of Calculate the average pixel by pixel to obtain the reflected non-polarized image in the kth static scene In this embodiment, .

[0080] Step 2: Based on the U-Net architecture, a polarization de-reflection network based on pseudo 3D convolution is constructed, which includes: a feature dimension-enhancing module, an encoder, a transmission image decoder, and a reflection image decoder. The overall network structure is as follows: Figure 1 As shown;

[0081] Step 2.1: Feature Dimension Upgrading Module } Perform dimension upgrading and fusion operations to obtain the 4-dimensional feature map of the kth static scene In this embodiment, the four dimensions C×P×H×W of the high-dimensional feature map include the number of color channels, polarization angle, image height and width;

[0082] Step 2.2: Constructing the encoder is composed of M basic convolutional modules, and sequentially Processing is performed to obtain M high-frequency feature maps in the kth static scene , M low-frequency feature maps And M high and low frequency superposition feature maps ;in, Indicates A high-frequency feature map, Indicates A low-frequency feature map, represents the mth high- and low-frequency superimposed feature map in the kth static scene. In this embodiment, , when m=1,2,3,4, the number of input and output channels of the mth basic convolution module are (3,32), (32,64) (64,128), (128,256) respectively. The basic convolution module is as follows Figure 2 shown.

[0083] Step 2.2.1, when m=1, the nonlinear transformation module in the mth basic convolution module uses the first pseudo 3D convolution layer, the ReLU activation function operation and the second pseudo 3D convolution layer to After nonlinear transformation, the mth nonlinear 4-dimensional feature map with the shape of C×P×H×W is obtained in the kth static scene. ; Wherein, C represents the color channel, P represents the polarization channel, H represents the height, and W represents the width. In this embodiment, when m=1, the number of input and output channels of the first and second pseudo 3D convolution layers of the mth basic convolution module are (3,32) and (32,32) respectively, and the convolution kernel sizes are all (1,3,3);

[0084] Step 2.2.2, the polarization feature attention module PFAM in the mth basic convolution module uses the 3D average pooling layer, the third pseudo 3D convolution layer, the ReLU activation function operation, the fourth pseudo 3D convolution layer and the Sigmoid activation function operation in sequence to Transformed into a shape of C×P×1×1 Polarized attention feature map ; then and After pixel-by-pixel multiplication, we get the kth static scene Polarized attention weighted feature map In this embodiment, when m=1, the number of input and output channels of the third and fourth pseudo 3D convolution layers of the mth basic convolution module are (32, 4) and (4, 1), respectively, and the convolution kernel sizes are (3, 1, 1); when m=2, the number of input and output channels of the third and fourth pseudo 3D convolution layers of the mth basic convolution module are (64, 8) and (8, 1), respectively, and the convolution kernel sizes are (3, 1, 1); when m=3, the number of input and output channels of the third and fourth pseudo 3D convolution layers of the mth basic convolution module are (128, 16) and (16, 1), respectively, and the convolution kernel sizes are (3, 1, 1); when m=4, the number of input and output channels of the third and fourth pseudo 3D convolution layers of the mth basic convolution module are (256, 32) and (32, 256), respectively, and the convolution kernel sizes are (3, 1, 1).

[0085] Step 2.2.3, the spatial feature attention module SFAM in the mth basic convolution module sequentially uses the fifth pseudo 3D convolution layer, ReLU activation function operation, the sixth pseudo 3D convolution layer and Sigmoid activation function operation to convert Convert to Spatial attention feature map , and then and After pixel-by-pixel multiplication, we get the kth static scene Comprehensive attention weighted feature map . In this embodiment, when m=1, the number of input and output channels of the fifth and sixth pseudo 3D convolution layers of the mth basic convolution module are (32, 4) and (4, 32), respectively, and the convolution kernel sizes are (1, 3, 3); when m=2, the number of input and output channels of the fifth and sixth pseudo 3D convolution layers of the mth basic convolution module are (64, 8) and (8, 64), respectively, and the convolution kernel sizes are (1, 3, 3); when m=3, the number of input and output channels of the fifth and sixth pseudo 3D convolution layers of the mth basic convolution module are (128, 16) and (16, 128), respectively, and the convolution kernel sizes are (1, 3, 3); when m=4, the number of input and output channels of the fifth and sixth pseudo 3D convolution layers of the mth basic convolution module are (256, 32) and (32, 256), respectively, and the convolution kernel sizes are (1, 3, 3);

[0086] Step 2.2.4, the octave convolution residual module OCRM in the mth basic convolution module first uses the seventh pseudo 3D convolution layer and the InstanceNorm3D normalization layer from Extract the A high- and low-frequency mixed feature map . In this embodiment, when m=1, the number of input and output channels of the seventh pseudo 3D convolution layer of the mth basic convolution module is (8,8), and the convolution kernel size is 1; when m=2, the number of input and output channels of the seventh pseudo 3D convolution layer of the mth basic convolution module is (64,32), and the convolution kernel size is 1; when m=3, the number of input and output channels of the seventh pseudo 3D convolution layer of the mth basic convolution module is (128,64), and the convolution kernel size is 1; when m=4, the number of input and output channels of the seventh pseudo 3D convolution layer of the mth basic convolution module is (256,128), and the convolution kernel size is 1.

[0087] Then, along the polarization channel dimension, Decompose into Original high-frequency feature map With Original low-frequency feature map ;

[0088] right After performing average pooling and pseudo 3D convolution operations in sequence, we get High-frequency to low-frequency feature map In this embodiment, the size of the pooling window is , the step size of the pooling window is , when m=1,2,3,4, the convolution kernel of the pseudo 3D convolution operation of the mth basic convolution module is 1, and the number of input and output channels are (8,8), (16,16), (32,32), (64,64) respectively;

[0089] right After performing the pseudo 3D convolution operation, we get High-frequency to high-frequency feature map In this embodiment, when m=1, 2, 3, 4, the convolution kernel of the pseudo 3D convolution operation of the mth basic convolution module is 1, and the number of input and output channels is (8, 8), (16, 16), (32, 32), (64, 64) respectively;

[0090] right After performing pseudo 3D convolution and upsampling operations in sequence, we get Low-frequency to high-frequency feature map In this embodiment, when m=1,2,3,4, the convolution kernel of the pseudo 3D convolution operation of the mth basic convolution module is 1, and the number of input and output channels is (8,8), (16,16), (32,32), (64,64) respectively; the upsampling magnification factor is (1,2,2), and the algorithm adopts the nearest interpolation algorithm;

[0091] right After pseudo 3D convolution, we get Low-frequency to low-frequency feature map In this embodiment, when m=1, 2, 3, 4, the convolution kernel of the pseudo 3D convolution operation of the mth basic convolution module is 1, and the number of input and output channels are (8, 8), (16, 16), (32, 32), and (64, 64), respectively.

[0092] Will and After adding pixel by pixel, Perform pixel-by-pixel multiplication and then perform the ReLU activation function operation to obtain the High frequency feature map ;

[0093] Will and After adding pixel by pixel, Perform pixel-by-pixel multiplication and then perform ReLU function activation operation to obtain the first Low frequency feature map ;

[0094] right After performing pseudo 3D convolution and InstanceNorm3D normalization, we get Normalized high-frequency feature map In this embodiment, when m=1, 2, 3, 4, the convolution kernel of the pseudo 3D convolution layer of the mth basic convolution module is 1, and the number of input and output channels is (8, 32), (16, 64), (32, 128), (64, 256) respectively;

[0095] right After performing pseudo 3D convolution and InstanceNorm3D normalization, we get Normalized low-frequency feature map In this embodiment, when m=1, 2, 3, 4, the convolution kernel of the pseudo 3D convolution layer of the mth basic convolution module is 1, and the number of input and output channels is (8, 32), (16, 64), (32, 128), (64, 256) respectively;

[0096] Will and After superposition in the polarization channel dimension, we get High and low frequency superposition feature map .

[0097] Step 2.2.5: When m=2,3,…,M, High and low frequency superposition feature map Input the mth basic convolution module for processing and get the mth High and low frequency superposition feature map , so that the Mth basic convolution module outputs the Mth high- and low-frequency superimposed feature map ;

[0098] In this embodiment, when m=2, the numbers of input and output channels of the first and second pseudo 3D convolution layers of the mth basic convolution module are (32, 64) and (64, 64), respectively, and the convolution kernel sizes are both (1, 3, 3);

[0099] When m=3, the input and output channel numbers of the first and second pseudo 3D convolution layers of the mth basic convolution module are (64, 128) and (128, 128), respectively, and the convolution kernel sizes are (1, 3, 3);

[0100] When m=4, the number of input and output channels of the first and second pseudo 3D convolution layers of the mth basic convolution module are (128, 256) and (256, 256), respectively, and the convolution kernel sizes are all (1, 3, 3).

[0101] Step 2.3, each decoder is composed of M upsampling jump connection modules and M 2D convolution modules; each 2D convolution module includes several 2D units, each 2D unit includes a 2D convolution layer and an activation function; the transmission image decoder is and Processing is performed to obtain the restored transmission image in the kth static scene ; Reflection image decoder pair and Processing is performed to obtain the restored reflection image in the kth static scene In this embodiment, each 2D convolution module includes 2 2D units, the activation function of the last 2D unit of each decoder is the Sigmoid activation function, and the activation functions of the other 2D units are the Sigmoid activation function and the ReLU activation function;

[0102] Step 2.3.1, Transmit image decoder pair and Processing is performed to obtain M 2D high-frequency transmission feature maps ;

[0103] When m=1, the mth upsampling skip connection module in the transmitted image decoder is The nearest interpolation algorithm is used in the spatial dimension to enlarge the image by 2×2 times, and then the image is connected with After splicing in the polarization channel dimension, it is processed by the mth 2D convolution module of the transmission image decoder to obtain the mth 2D high-frequency transmission feature map output by the transmission image decoder. In this embodiment, when m=1, the convolution kernel sizes of the two 2D convolution layers of the mth 2D convolution module in the transmission image decoder are both 3, and the numbers of input and output channels are (2048, 128) and (128, 128) respectively;

[0104] When m=2,3,…,M-1, the m-1th 2D high frequency transmission feature map In the mth upsampling jump connection module of the input transmission image decoder, the nearest interpolation algorithm is used in the spatial dimension. After 2×2 magnification, the After splicing in the polarization channel dimension, the mth 2D high-frequency transmission feature map output by the transmission image decoder is obtained after being processed by the mth 2D convolution module. ; Thus, the M-1th 2D convolution module of the transmission image decoder outputs the M-1th 2D high-frequency transmission feature map In this embodiment, when m=2, the convolution kernel sizes of the two 2D convolution layers of the m-th 2D convolution module in the transmission image decoder are both 3, and the numbers of input and output channels are (640, 64) and (64, 64), respectively; when m=3, the convolution kernel sizes of the two 2D convolution layers of the m-th 2D convolution module in the transmission image decoder are both 3, and the numbers of input and output channels are (320, 32) and (32, 32), respectively.

[0105] When m=M, the M-1th 2D high-frequency transmission feature map In the Mth 2D convolution upsampling module of the input transmission image decoder, the nearest interpolation algorithm is used in the spatial dimension. After 2×2 magnification, the After splicing in the polarization channel dimension, the Mth 2D high-frequency transmission feature map output by the transmission image decoder is obtained after being processed by the mth 2D convolution module of the transmission image decoder. , that is, the transmission image is restored. In this embodiment, when m=4, the convolution kernel sizes of the two 2D convolution layers of the mth 2D convolution module in the transmission image decoder are both 3, and the numbers of input and output channels are (160, 3) and (3, 3) respectively.

[0106] Step 2.3.2, reflected image decoder pair and Processing is performed to obtain M 2D low-frequency reflection feature maps ;

[0107] When m=1, the mth upsampling skip connection module in the reflected image decoder will After the nearest interpolation algorithm is used to magnify the space dimension by 2×2, the skip connection is connected with After splicing in the polarization channel dimension, it is processed by the mth 2D convolution module of the reflection image decoder to obtain the mth 2D low-frequency reflection feature map output by the reflection image decoder. In this embodiment, when m=1, the convolution kernel sizes of the two 2D convolution layers of the mth 2D convolution module in the transmission image decoder are both 3, and the numbers of input and output channels are (2048, 128) and (128, 128) respectively;

[0108] When m=2,3,…,M-1, the m-1th 2D low-frequency reflection feature map In the mth upsampling jump connection module of the input reflection image decoder, the nearest interpolation algorithm is used in the spatial dimension. After 2×2 magnification, the After splicing in the polarization channel dimension, the mth 2D low-frequency reflection feature map output by the reflection image decoder is obtained through processing by the mth 2D convolution module of the reflection image decoder. ; Thus, the M-1th 2D convolution module of the reflection image decoder outputs the M-1th 2D low-frequency reflection feature map In this embodiment, when m=2, the convolution kernel sizes of the two 2D convolution layers of the m-th 2D convolution module in the transmission image decoder are both 3, and the numbers of input and output channels are (640, 64) and (64, 64), respectively; when m=3, the convolution kernel sizes of the two 2D convolution layers of the m-th 2D convolution module in the transmission image decoder are both 3, and the numbers of input and output channels are (320, 32) and (32, 32), respectively.

[0109] When m=M, the M-1th 2D low-frequency reflection feature map Input the Mth upsampling jump connection of the reflected image decoder and use the nearest interpolation algorithm in the spatial dimension to After 2×2 magnification, the After splicing in the polarization channel dimension, the Mth 2D low-frequency reflection feature map output by the reflection image decoder is obtained after being processed by the mth 2D convolution module of the reflection image decoder. , that is, the reflected image is restored. In this embodiment, when m=4, the convolution kernel sizes of the two 2D convolution layers of the mth 2D convolution module in the transmission image decoder are both 3, and the numbers of input and output channels are (160, 3) and (3, 3) respectively.

[0110] Step 3: training a polarization image de-reflection model based on pseudo 3D convolution;

[0111] Step 3.1: Build based on , and , use formula (1) to construct the total loss function L;

[0112] (1)

[0113] In formula (1), represents the reconstruction loss, Represents perceived loss, represents the rejection loss and represents adversarial loss, λ1, λ2, and λ3 are perceptual losses respectively. , Fighting Losses and rejection loss The weight of

[0114] Use formula (2) to construct the reconstruction loss :

[0115] (2)

[0116] In formula (2), is the absolute value distance, is the gradient about the image horizontal coordinate x, is the gradient about the image ordinate y;

[0117] Use formula (3) to construct the perceptual loss :

[0118] (3)

[0119] Using formula (4) to construct the exclusion loss :

[0120] (4)

[0121] In formula (4), and Respectively express and Carry on 2 n The feature map obtained after downsampling, tanh is the hyperbolic tangent activation function, λ T and λ R are 2 normalization factors, and  represents element-by-element multiplication.

[0122] The polarization de-reflection network based on pseudo 3D convolution is used as the generator, and the discriminator is constructed at the same time. The image x is input into the discriminator for processing, and the discriminator mapping value of the image x between 0 and 1 is obtained. ;

[0123] The generator loss in the adversarial loss is constructed using equations (5) and (6) respectively. and the discriminator loss :

[0124] (5)

[0125] (6)

[0126] In formula (5) and formula (6), for The discriminator mapping value of for The discriminator mapping value of .

[0127] Step 3.2: Use the ADAM optimizer to train the pseudo 3D convolution-based polarization image dereflection network and calculate the total loss function L to update the network parameters. During the training of the network, the discriminator and the generator are updated alternately. First, the generator is fixed and the discriminator is updated to minimize the discriminator loss. ; Then, fix the discriminator and update the generator to minimize the generator loss , training until the total loss function converges, thereby obtaining the optimal pseudo 3D convolution-based de-reflection model using polarization images, which is used to de-reflect the synthesized polarization mixed image and the real polarization mixed image. In this embodiment, during the training phase, the network is trained for 120 epochs, and the initial learning rate is set to ; After the 20th epoch, adversarial loss is added, and after the 60th epoch, the learning rate decays to ; After the 80th epoch, the learning rate decays to ; After the 100th epoch, the learning rate decays to .

[0128] In this embodiment, 8925 images of 4 different polarization angles ( ) polarization mixed images, 8925 polarization mixed images with four different polarization angles (0°, 45°, 90°, 135°) and their corresponding transmission images and reflection images are used to form a polarization dataset for training. During the training process, the input polarization image size is randomly cropped to 352×352, and the transmission image output by the de-reflection model based on pseudo 3D convolution is used. and reflected image The total loss is calculated, and the calculated loss is combined with the ADAM optimizer to guide the network training to obtain a de-reflection model based on pseudo 3D convolution using polarization images.

[0129] In this embodiment, an electronic device includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the above method, and the processor is configured to execute the program stored in the memory.

[0130] In this embodiment, a computer-readable storage medium stores a computer program on the computer-readable storage medium, and the computer program executes the steps of the above method when executed by a processor.

[0131] Table 1 shows the comparison between the polarization image dereflection method based on pseudo 3D convolution of the present invention and other dereflection methods using peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) as evaluation indicators. For fair comparison, all dereflection methods were retrained on the data set. During the test, the present invention and other dereflection methods used polarization images and ordinary RGB images, respectively. PSNR is the ratio of the maximum power of a signal to the noise power that may affect its representation accuracy. The larger the value, the smaller the distortion of image dereflection. SSIM is the similarity of an image in terms of brightness, contrast, and structure. Its value range is [0, 1]. The larger the value, the more similar the transmitted image is to the real transmitted image. According to the quantitative analysis of Table 1, it can be seen that the method of the present invention has achieved the best or second best results in both indicators.

[0132] Table 1

[0133]

[0134] Figure 3The figures are the results of the polarization image de-reflection method based on pseudo 3D convolution of the present invention and other current de-reflection methods on a real non-polarization data set. Among them, Ours represents the polarization image dereflection method based on pseudo 3D convolution of the present invention; the IBCLN method in "Single image reflection removal through cascaded refinement" proposes an iterative lifting convolutional LSTM network, which refines the estimation of the transmission layer and the reflection layer in an iterative manner, uses the LSTM in the cascade step to transfer information, and solves the gradient vanishing problem; the ERRNet proposed in "Single image reflection removal exploiting misaligned training data and network enhancements" enhances the baseline network architecture by embedding context encoding modules, which can reduce the uncertainty in the strong reflection area by using high-level context clues; the BDN method in "Seeing deeply and bidirectionally: A deeplearning approach for single image reflection removal" proposes a cascaded deep neural network to estimate the background image and the reflection layer at the same time, which significantly improves the quality of reflection removal; Wen et al. proposed a deep synthesis network (SynNet) in "Single image reflection removal beyond linearity" to predict nonlinear alpha blending masks of two input images to simulate different types of reflections; Dong et al. proposed a deep synthesis network (SynNet) in "Location-aware single image reflection removal beyond linearity" to predict nonlinear alpha blending masks of two input images to simulate different types of reflections; In this paper, we propose a position-aware single-image reflection removal method that uses a network with a reflection detection module that can regress a probabilistic reflection confidence map (RCMap) and use it as a cue to control the feature flow when the network predicts the reflection and transmission layers.

[0135] Figure 4 The figures are the results of the polarization image dereflection method based on pseudo 3D convolution of the present invention and the other current dereflection methods mentioned above on the synthetic polarization dataset.

Claims

1. A polarization image dereflection method based on pseudo 3D convolution, characterized in that: Follow these steps: Step 1, obtaining a comprehensive polarization dataset including a real mixed polarization image and a synthetic mixed polarization image; Step 1.1: Obtain a set of real mixed polarization images in K1 static scenes , real transmission polarization image set and the real reflection polarization image set ;in, Indicates that the polarization angle in the k1th static scene is True mixed polarization image of Indicates that the polarization angle in the k1th static scene is The real transmission polarization image of Indicates that the polarization angle in the k1th static scene is The real reflected polarization image of Step 1.2: Obtain another K2 sets of synthetic mixed polarization images under static scenes , Synthetic transmission polarization image set and synthetic reflected polarization image sets ;in, Indicates that the polarization angle in the k2th static scene is The synthetic mixed polarization image of Indicates that the polarization angle in the k2th static scene is The synthetic transmission polarization image of Indicates that the polarization angle in the k2th static scene is The synthetic reflected polarization image of Step 1.3: and Merge into a hybrid polarization dataset ;Will and Merge into transmission polarization dataset ;Will and Merge into reflectance polarization dataset ;in, Indicates that the polarization angle in the kth static scene is Mixed polarization images of Indicates that the polarization angle in the kth static scene is Transmission polarization image of Indicates that the polarization angle in the kth static scene is Transmission polarization image of For all polarization angles in the same scene of After averaging pixel by pixel, the mixed non-polarized image in the kth static scene is obtained. ; For all polarization angles in the same scene of Calculate the average pixel by pixel to obtain the transmitted non-polarized image in the kth static scene ; For all polarization angles in the same scene of Calculate the average pixel by pixel to obtain the reflected non-polarized image in the kth static scene ; Step 2: Based on the U-Net architecture, a polarization de-reflection network based on pseudo 3D convolution is constructed, which includes: a feature dimension-enhancing module, an encoder, a transmission image decoder, and a reflection image decoder; Step 2.1: The feature dimension-upgrading module } Perform dimension upgrading and fusion operations to obtain the 4-dimensional feature map of the kth static scene ; Step 2.2: Constructing the encoder is composed of M basic convolutional modules, and sequentially Processing is performed to obtain M high-frequency feature maps in the kth static scene , M low-frequency feature maps And M high and low frequency superposition feature maps ;in, Indicates A high-frequency feature map, Indicates A low-frequency feature map, Represents the mth high- and low-frequency superimposed feature map in the kth static scene; Step 2.3, transmit image decoder pair and Processing is performed to obtain the restored transmission image in the kth static scene ; Reflection Image Decoder and Processing is performed to obtain the restored reflection image in the kth static scene ; Step 3: training a polarization image de-reflection model based on pseudo 3D convolution; Step 3.1: Build based on , and , use the formula to construct the total loss function L; (1) In formula (1), represents the reconstruction loss, Represents perceived loss, represents the rejection loss and represents adversarial loss, λ1, λ2, and λ3 are perceptual losses respectively. , Fighting Losses and rejection loss The weight of Step 3.2: Use the ADAM optimizer to train the pseudo 3D convolution-based polarization image dereflection network, and during the training process, first fix the generator and update the discriminator to minimize the discriminator loss. , then fix the discriminator and update the generator to minimize the generator loss , thereby alternately updating the discriminator and the generator until the total loss function L converges, thereby obtaining the optimal pseudo-3D convolution-based de-reflection model using polarization images, which is used to de-reflect the synthesized polarization mixed images and the real polarization mixed images.

2. The polarization image dereflection method based on pseudo 3D convolution according to claim 1, characterized in that: Each basic convolution module in step 2.2 is composed of a nonlinear transformation module, a polarization feature attention module PFAM, a spatial feature attention module SFAM and an octave convolution residual module OCRM; Step 2.2.1, when m=1, the nonlinear transformation module in the mth basic convolution module uses the first pseudo 3D convolution layer, the ReLU activation function operation and the second pseudo 3D convolution layer to After nonlinear transformation, the mth nonlinear 4-dimensional feature map with the shape of C×P×H×W is obtained in the kth static scene. ; Wherein, C represents the color channel, P represents the polarization channel, H represents the height, and W represents the width; Step 2.2.2, the polarization feature attention module in the mth basic convolution module uses the 3D average pooling layer, the third pseudo 3D convolution layer, the ReLU activation function operation, the fourth pseudo 3D convolution layer and the Sigmoid activation function operation in turn. Transformed into a shape of C×P×1×1 Polarized attention feature map ; then and After pixel-by-pixel multiplication, we get the kth static scene Polarized attention weighted feature map ; Step 2.2.3, the spatial feature attention module in the mth basic convolution module uses the fifth pseudo 3D convolution layer, ReLU activation function operation, the sixth pseudo 3D convolution layer and Sigmoid activation function operation in turn to Convert to Spatial attention feature map , and then and After pixel-by-pixel multiplication, we get the kth static scene Comprehensive attention weighted feature map ; Step 2.2.4, the octave convolution residual module in the mth basic convolution module first uses the seventh pseudo 3D convolution layer and the InstanceNorm3D normalization layer from Extract the High and low frequency mixed feature map ; Then, along the polarization channel dimension, Decompose into Original high-frequency feature map With Original low-frequency feature map ; right After performing average pooling and pseudo 3D convolution operations in sequence, we get High-frequency to low-frequency feature map ; right After performing the pseudo 3D convolution operation, we get High-frequency to high-frequency feature map ; right After performing pseudo 3D convolution and upsampling operations in sequence, we get Low-frequency to high-frequency feature map ; right After pseudo 3D convolution, we get Low-frequency to low-frequency feature map ; Will and After adding pixel by pixel, Perform pixel-by-pixel multiplication and then perform the ReLU activation function operation to obtain the first High frequency feature map ; Will and After adding pixel by pixel, Perform pixel-by-pixel multiplication and then perform ReLU function activation operation to obtain the first Low frequency feature map ; right After performing pseudo 3D convolution and InstanceNorm3D normalization, we get Normalized high-frequency feature map ; right After performing pseudo 3D convolution and InstanceNorm3D normalization, we get Normalized low-frequency feature map ; Will and After superposition in the polarization channel dimension, we get High and low frequency superposition feature map ; Step 2.2.5: When m=2,3,…,M, High and low frequency superposition feature map Input the mth basic convolution module for processing and get the mth High and low frequency superposition feature map , so that the Mth basic convolution module outputs the Mth high- and low-frequency superimposed feature map .

3. The polarization image dereflection method based on pseudo 3D convolution according to claim 2, characterized in that: The transmission image decoder and the reflection image decoder in step 2.3 are both composed of M upsampling skip connection modules and M 2D convolution modules; Step 2.3.1, when m=1, the mth upsampling skip connection module of the transmission image decoder is After amplification, the jump connection After splicing in the polarization channel dimension, it is processed by the mth 2D convolution module of the transmission image decoder to obtain the mth 2D high-frequency transmission feature map ; When m=2,3,…,M-1, the m-1th 2D high frequency transmission feature map The input is sent to the mth upsampling skip connection module of the image decoder and is processed in the spatial dimension. After amplification, the jump connection After splicing in the polarization channel dimension, the mth 2D high-frequency transmission feature map is obtained by processing the mth 2D convolution module of the transmission image decoder. ; Thus, the M-1th 2D convolution module of the transmission image decoder outputs the M-1th 2D high-frequency transmission feature map ; When m=M, the M-1th 2D high-frequency transmission feature map The input is sent to the Mth upsampling skip connection module of the image decoder and is processed in the spatial dimension. After amplification, the jump connection After splicing in the polarization channel dimension, the Mth 2D high-frequency transmission feature map output by the transmission image decoder is obtained after being processed by the mth 2D convolution module of the transmission image decoder. , and used as the restored transmission image; Step 2.3.2, when m = 1, the mth upsampling skip connection module in the reflected image decoder is After amplification, the jump connection After splicing in the polarization channel dimension, it is processed by the mth 2D convolution module in the reflection image decoder, so that the mth 2D low-frequency reflection feature map ; When m=2,3,…,M-1, the m-1th 2D low-frequency reflection feature map Input the mth upsampling skip connection module of the reflected image decoder and perform After amplification, the jump connection After splicing in the polarization channel dimension, the mth 2D low-frequency reflection feature map is obtained by processing the mth 2D convolution module in the reflection image decoder. ; Thus, the M-1th 2D convolution module of the reflection image decoder outputs the M-1th 2D low-frequency reflection feature map ; When m=M, the M-1th 2D low-frequency reflection feature map Input the Mth upsampling skip connection module of the reflected image decoder and perform After amplification, the jump connection After splicing in the polarization channel dimension, it is processed by the mth 2D convolution module in the reflection image decoder to obtain the Mth 2D low-frequency reflection feature map output by the reflection image decoder. , and used as the restored reflection image.

4. The polarization image dereflection method based on pseudo 3D convolution according to claim 3, characterized in that: In step 3.1, the reconstruction loss is constructed using formula (2): : (2) In formula (2), is the absolute value distance, is the gradient about the image horizontal coordinate x, is the gradient about the image ordinate y; Use formula (3) to construct the perceptual loss : (3) Using formula (4) to construct the exclusion loss : (4) In formula (4), and Respectively express and Carry on 2 n The feature map obtained after downsampling, tanh is the hyperbolic tangent activation function, λ T and λ R are 2 normalization factors,  represents element-by-element multiplication; The polarization de-reflection network based on pseudo 3D convolution is used as the generator G, and the loss of the generator G in the adversarial loss is constructed using equations (5) and (6) respectively. and the loss of the discriminator D : (5) (6) In formula (5) and formula (6), for The mapping value of the discriminator D, for The mapping value of the discriminator D.

5. An electronic device, comprising a memory and a processor, characterized in that: The memory is used to store a program that supports a processor to execute the polarization de-reflection method according to any one of claims 1 to 4, and the processor is configured to execute the program stored in the memory.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the polarization dereflection method according to any one of claims 1 to 4 are executed.

Images