A single image reflection removal method based on feature difference enhancement

Abstract

The present invention discloses a single-image reflection removal method based on feature differential enhancement. The method involves inputting a reflection-contaminated image into a VGG9 network to extract hypercolumn features. The hypercolumn features are then connected to the reflection-contaminated image channel by channel to form the network's input features. The input features are fed into a shared encoder with shared parameters and downsampled four times to generate feature maps of four different scales and resolutions. Each downsampling step is normalized using a 1 / 4 instance normalization strategy. A single encoder and a dual decoder are used to obtain transmission enhancement features and reflection enhancement features of a reflectance map R. The transmission enhancement features and reflection enhancement features are then input into an adaptive information exchange module for adaptive information exchange. Subsequently, upsampling and convolution are performed sequentially to obtain the final predicted transmission and reflection maps. This method utilizes the feature differences and information exchange between the transmission and reflection layers of the reflection-contaminated image, fully considering the complementary mechanism between the transmission and reflection layers.

Description

Technical Field

[0001] The present invention relates to computer underlying vision and image restoration technology, and in particular to a single image reflection removal method based on feature difference enhancement. Background Art

[0002] Reflective contamination often occurs in scenes containing glass (or materials with both transmissive and reflective properties), such as museums, aquariums, window displays, and other settings. Complex reflections in images can cause visual discomfort, and these unpleasant reflections can severely impact advanced computer vision tasks, including scene understanding, object detection, and autonomous driving. Traditional methods rely primarily on additional prior information, such as using user annotations and gradient sparsity priors to guide image separation, assuming that reflections are blurry, introducing specific priors, penalizing large gradients, and using edge classification. While these methods can achieve good results when these assumptions hold, real-world imaging conditions and scene content are difficult to predict in advance, resulting in a lack of generalization. Deep learning methods have garnered significant attention in recent years, driven by increases in computing power. Based on whether or not the reflected image is used to predict the transmitted image, existing deep learning-based techniques are categorized into two trends: direct prediction of the transmitted image and reflection-aware prediction of the transmitted image.

[0003] For direct prediction of transmission maps, Wan, Sun, Shi, Levin, and others use deep learning to estimate transmission images directly from reflection-contaminated images. Among them, Levin et al. use a sparse prior-guided method to decompose images in layers with fewer gradient and angle operators; Fan et al. use structural edge information to construct a cascade network for image separation, and they also proposed a method for synthesizing image reflection layers; Wan et al. proposed a feature collaborative sharing network that focuses on improving the efficiency of gradient information fusion to enhance the ability to predict transmission layers. ERRNet addresses the problem of misaligned training data by designing an alignment-invariant loss function, combined with targeted network structure design to solve the problem of single-image reflection removal; Song et al. proposed a multi-level curvature-guided network that integrates multiple network architectures to gradually reconstruct the transmission layer. Prasad B et al. proposed a new scale-space method for removing reflections, which combines a low-scale subnetwork (LSSNet) to handle the lowest scale and a progressive inference (PI) stage to handle all higher scales.

[0004] However, these methods usually require deeper mining of structural information or additional priors, which greatly reduces the generality of these methods.

[0005] In addition, methods have been studied to directly estimate the transmission image by taking multiple images from different viewpoints. Lei et al. used the flash image alone from a pair of flash and non-flash images as a clue to predict the transmission image. Chang et al. proposed a dense Siamese network to solve the problem of removing reflections from flash and non-flash image pairs. However, these methods require multi-view images, which greatly limits their application scenarios.

[0006] For reflection-aware transmission map prediction, more useful guidance information can be embedded. Some methods attempt to simultaneously predict both transmission and reflection images, leveraging the communication between the two streams to improve the reflection removal task. Levin et al. manually annotated transmission and reflection images using edge information to separate the images by their differences, but this is often a labor-intensive task. Li et al. proposed a two-layer gradient regularization scheme and considered using the difference in blur and reflection to decompose mixed images. Jie et al. proposed a cascaded deep neural network that first predicts reflectivity based on the predicted transmittance and then uses the predicted reflection to estimate transmission. Kim et al. used a physics-based approach to synthesize training data and proposed a two-stage reflection removal network. The decomposition network aims to separate transmission and reflection, while the backpropagation network aims to enhance reflection prediction, thereby providing more prior information for predicting the transmission map. Li et al. proposed an iteratively boosted convolutional LSTM network for reflection removal. They used the LSTM to iteratively improve the estimation of the transmission and reflection layers. Hu et al. proposed a two-stream decomposition network that cleverly uses information discarded by the activation function as a carrier to enable information exchange between the transmission and reflection maps, achieving satisfactory performance. Rahmanikhezri used semantic features to optimize neural network parameters, generating two dedicated transmission and reflection layers, thereby achieving an unsupervised method for single-image reflection removal. However, most of these methods are two-stage or iterative, relying heavily on the prediction results of the previous stage. Errors in the previous stage are then used in the next stage, leading to error accumulation and affecting the image de-reflection effect. Furthermore, this multi-stage training strategy consumes a large amount of computing resources and has a relatively slow inference speed. Summary of the Invention

[0007] Purpose of the invention: Most of the existing technologies adopt a reflection removal method, which focuses too much on the transmission layer and ignores the complementary mechanism between the transmission layer and the reflection layer. The purpose of the present invention is to address the shortcomings of the existing technology and provide a single image reflection removal method based on feature difference enhancement. It utilizes the feature differences and information exchange between the transmission layer and the reflection layer of the reflection contaminated image, and fully considers the complementary mechanism between the transmission layer and the reflection layer.

[0008] Technical solution: A single image reflection removal method based on feature difference enhancement of the present invention comprises the following steps:

[0009] Step 1: Obtain the reflection-contaminated image and supervision image required for model training. The supervision image includes a transmission image and a reflection image. Perform preprocessing operations such as random cropping and normalization on the reflection-contaminated image and the supervision image to obtain the preprocessed reflection-contaminated image and supervision image.

[0010] Step 2: Input the pre-processed reflection-contaminated image into the pre-trained VGG-19 network to extract the hypercolumn features with 1473 channels; then, concatenate the hypercolumn features with the reflection-contaminated image by channel to form the input features of the network;

[0011] Step 3: The input features obtained in step 2 are fed into a shared encoder with shared parameters, and four downsampling operations are performed in sequence to generate feature maps of four different scales and resolutions, thereby extracting coupled features.

[0012] Each downsampling is performed through the 1 / 4 instance normalization strategy to perform feature normalization, using the mean and variance of the features on a single channel to recalibrate the data distribution;

[0013] Step 4: In the decoders that predict the transmission image and the reflection image respectively, the features output by the previous decoder layer and the coupled features of the encoder at the corresponding scale are input into the feature difference enhancement module (the features of the first decoder layer come from the last encoder layer). The feature difference enhancement module calculates the corresponding transmission enhancement features of the predicted transmission image T and the reflection enhancement features of the reflection image R.

[0014] Step 5: The transmission enhancement features and reflection enhancement features obtained in step 4 are then input into the adaptive information exchange module, so that the two branches of the predicted transmission map T and the reflection map R perform adaptive information exchange, and then perform an upsampling operation;

[0015] Step 6: After the four-stage upsampling in step 5, the dual-branch decoder outputs the predicted transmission map and reflection map through a convolutional layer respectively.

[0016] This method leverages the feature differences and information exchange between the transmission and reflection layers of a reflection-contaminated image to gradually achieve feature decoupling, thereby eliminating reflections from the contaminated image. The method utilizes the coupled features in a single encoder as middleware, rationally using differential features to enhance the prediction branches of the transmitted and reflected images T and R, thereby achieving decoupling. Furthermore, the method employs a U-Net network architecture as the backbone network, with a shared encoder for extracting the original coupled features and two independent decoders for predicting the transmitted and reflected images T and R.

[0017] Furthermore, the step 2 extracts the hypercolumn features using a pre-trained VGG network model.

[0018] Furthermore, the structure of the 1 / 4 instance normalization strategy is as follows:

[0019] For the input feature F in each layer Encoder I ,First, use the channel attention mechanism CAM to weight its channels and select the input feature F I The characteristic F of the first 1 / 4 of the channel 1 / 4 Send it to the instance normalization layer to obtain the instance normalized feature F IN ;

[0020] Then, the instance normalized feature F IN With input feature F I The remaining features F 3 / 4 Connect and fuse to get feature F con ;

[0021] Finally, use one convolution to transform the feature F con and the remaining features F 3 / 4 Perform convolution, using input features F I Perform jump connections on the convolution results to obtain the normalized output feature F. out , to improve the robustness of feature encoding.

[0022] Furthermore, the specific method of obtaining the enhanced features of the predicted transmission image and reflection image in step 4 is:

[0023] For the coupled features extracted from the i-th layer of the encoder obtained in step 3 The feature difference enhancement module starts with the coupled features The transmission features and reflection features of the previous stage of the corresponding decoder are subtracted from each other; then, the result of the feature difference is sent to the attention module to obtain a favorable differential feature and

[0024] In addition, the strategy of calculating feature differences may bring about the problem of feature incompatibility and information loss. Therefore, we then calculate the feature differences in the following order: and Add features Finally, add the features The differential characteristics of and Make jump connections by channel and obtain corresponding enhanced features and To improve the robustness of the algorithm, the expression is as follows:

[0025]

[0026] The encoder has 4 layers, i represents the i-th layer of the encoder, and j represents the number of layers of the decoder corresponding to the encoder. ATT indicates that a convolutional attention module is applied in the feature difference enhancement module to capture the potential connection between channels and enhance the representation ability of features. and is the input of the j+1th layer in the decoder, which are the feature difference enhancement module pairs and The feature enhancement results can significantly improve the performance of predicting transmission features T and reflection features R respectively.

[0027] Furthermore, the specific processing method of the adaptive information exchange module in step 5 is:

[0028] For processing transmission enhancement feature F ET , reflection enhancement feature F ER , firstly, a 3×3 convolution is performed to extract the corresponding features and the number of channels is halved;

[0029] Then apply the activation function ReLU to get and and It is the information discarded after the ReLU activation function, and the information contained in it is helpful to the opposite branch; and It is the information retained after the ReLU activation function;

[0030] The calculation formula is defined as follows:

[0031]

[0032]

[0033] Next, to avoid the error accumulation problem that may be caused by incorrect information exchange, and The 3×3 convolution layer will be used to adaptively retain the beneficial information in the discarded information, and the redundant neurons will be discarded again through ReLU;

[0034] Finally, the processed information will be added to the opposite branch, that is, the processed Add to the branch of prediction T, processed Add it to the branch of prediction R to get the corresponding information exchange result.

[0035] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0036] (1) The present invention provides a novel single-stage single-image reflection elimination network with a shared encoder and two decoders. The coupling features in the shared encoder are used as middleware to reasonably calculate the differential features to gain the prediction of the T branch and the R branch, thereby achieving the purpose of decoupling and giving full play to the differential and complementary advantages between the reflection layer R and the transmission layer T. The present invention adopts a single-stage training strategy, and the inference speed is faster than that of multi-level or multi-stage training methods.

[0037] (2) The present invention proposes a feature difference enhancement module to effectively calculate the enhanced feature difference, which has the advantages of being lightweight and portable.

[0038] (3) The present invention proposes an adaptive information exchange block to further achieve robust and efficient complementary information exchange.

[0039] (4) The present invention introduces a 1 / 4 selective instance normalization strategy for the features in the encoder to calibrate the distribution of features, thereby synergistically enhancing the feature difference enhancement module and the adaptive information exchange block. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 Schematic diagram of the process of the present invention

[0041] Figure 2 This is the principle framework diagram of the single image reflection removal method based on feature difference enhancement of the present invention

[0042] Figure 3 Schematic diagram of the 1 / 4 instance normalization strategy of the present invention

[0043] Figure 4 This is the structural diagram of the feature difference enhancement module of the present invention

[0044] Figure 5 This is the structural diagram of the adaptive information exchange module of the present invention

[0045] Figure 6 This is a diagram showing the effect of reflection removal in an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The technical solution of the present invention is described in detail below, but the protection scope of the present invention is not limited to the embodiments.

[0047] like Figure 1 and Figure 2 As shown, the single image reflection removal method based on feature difference enhancement of the present invention includes the following steps:

[0048] Step 1: Obtain the reflection-contaminated image and supervision image required for model training. The supervision image includes a transmission image and a reflection image. Perform preprocessing operations such as random cropping and normalization on the reflection-contaminated image and the supervision image to obtain the preprocessed reflection-contaminated image and supervision image.

[0049] Step 2: Input the pre-processed reflection-contaminated image into the pre-trained VGG-19 network to extract the hypercolumn features with 1473 channels; then, concatenate the hypercolumn features with the reflection-contaminated image by channel to form the input features of the network;

[0050] Step 3: The input features obtained in step 2 are fed into a shared encoder with shared parameters, and four downsampling operations are performed in sequence to generate feature maps of four different scales and resolutions, thereby extracting coupled features.

[0051] Each downsampling is performed through the 1 / 4 instance normalization strategy to perform feature normalization, using the mean and variance of the features on a single channel to recalibrate the data distribution;

[0052] Step 4: In the decoder that predicts the transmission image and the reflection image respectively, the features of the previous stage and the coupled features of the encoder at the corresponding scale are input into the feature difference enhancement module, and the transmission enhancement features of the corresponding predicted transmission image T and the reflection enhancement features of the reflection image R are calculated by the feature difference enhancement module;

[0053] Step 5: The transmission enhancement features and reflection enhancement features obtained in step 4 are then input into the adaptive information exchange module, so that the two branches of the predicted transmission map T and the reflection map R perform adaptive information exchange, and then perform an upsampling operation;

[0054] Step 6: After the four-stage upsampling in step 5, the dual-branch decoder outputs the predicted transmission map and reflection map through a convolutional layer respectively.

[0055] This embodiment first uses a feature difference enhancement module to distinguish the features of the reflective and transmissive layers, effectively improving the image separation task. Secondly, an adaptive information exchange module between the transmissive and reflective layers is proposed in the decoder to capture more complementary information. Finally, an instance normalization strategy is introduced to improve the model's convergence speed and prediction accuracy, ultimately achieving the goal of optimizing the reflection removal method.

[0056] like Figure 3 As shown, the structure of the 1 / 4 instance normalization strategy for the original coupling features in this embodiment is as follows:

[0057] For the input feature F I ,First, use the channel attention mechanism CAM to weight its channels and select the enhanced input feature F IThe characteristic F of the first 1 / 4 of the channel 1 / 4 Send it to the instance normalization layer to obtain the instance normalized feature F IN ; Then, normalize the instance feature F IN With input feature F I The remaining features F 3 / 4 Connect and fuse to get feature F con ; Finally, use 1 convolution to feature F con and the remaining features F 3 / 4 Perform convolution and use the original input feature F I Perform jump connection on the convolution result to finally obtain the coupling feature F out .

[0058] Moreover, the instance normalization weight is optional, for example, 1 / 4, 1 / 2, and full instance normalization are possible, and 1 / 4 instance normalization achieves the best balance in terms of effect and performance. Since the biggest obstacle to image separation tasks (such as de-reflection, de-raining, and de-fogging) is to ensure that the network can effectively learn highly differentiated features from the feature layers of different images. In addition, the features in the encoder often contain significant heterogeneous coupling information, and truly effective feature decoupling tends to be performed at the decoder stage. Therefore, the present invention uses feature differentials to enhance the prediction of the transmission map T and the reflection map R. Figure 4 As shown, the specific method for obtaining the enhanced features of the predicted transmission image and reflection image in step 4 of this embodiment is:

[0059] For the coupled features Feature difference enhancement module first couples features The transmission feature and reflection feature of the previous stage of the decoder are subtracted from each other; then, the result of the feature difference is sent to the attention module to obtain a favorable differential feature and

[0060] Then, the differential features and Add coupling features

[0061] Finally, there is a coupling feature for adding The differential characteristics of and Make jump connections by channel and obtain corresponding enhanced features and The expression is as follows:

[0062]

[0063] As can be seen above, the difference between the coupled and reflective features completely suppresses features related to predicted reflection, while retaining features related to predicted transmission, and vice versa. Furthermore, the feature difference enhancement module connects the coupled and differential features channel by channel using skip connections, rather than directly using the original skip connections in UNet. This feature difference enhancement module only introduces a small number of parameters and can be easily applied to other decomposition tasks after simple modification.

[0064] Because the features processed by the feature difference enhancement module are received and forward-propagated in the decoder, including differential information as well as the original coupled information from the encoder, these features need to be further selected and exchanged in the decoder using appropriate strategies. Existing technical solutions mainly focus on the information discarded by the activation function and directly add the discarded information from one branch to the other without any processing or selection. This significantly reduces the robustness after multiple exchanges. In addition, the two independent encoders may exacerbate feature incompatibility.

[0065] This embodiment uses a shared encoder to reduce this feature ambiguity, so that differential information can be effectively exchanged in the forward propagation of the dual-branch decoder, and it is used as a carrier for information exchange. In addition, the present invention focuses more on useful information rather than all discarded information of the decoder. Based on this analysis, the present invention gradually separates the mixed reflection image through an adaptive information exchange block, and gradually restores the transmission map T and the reflection map R through the interaction of complementary information. In order to ensure that both branches can obtain more support information from each other, this information exchange strategy is adopted in each upsampling layer of the decoder. The above process is as follows Figure 5 As shown, the specific processing method of the adaptive information exchange module is:

[0066] For processing transmission enhancement feature F ET , reflection enhancement feature F ER , firstly, a 3×3 convolution is performed to extract the corresponding features and the number of channels is halved;

[0067] Then apply the activation function ReLU to get and and It is the information discarded after the ReLU activation function;

[0068] The calculation formula is defined as follows:

[0069]

[0070]

[0071] Next, to avoid the error accumulation problem that may be caused by incorrect information exchange, and The 3×3 convolution layer will adaptively retain favorable information and the ReLU will discard redundant neurons;

[0072] Finally, the processed information will be added to the opposite branch, that is, the processed Add to the branch of prediction T, processed Add to the branch of prediction R, and we get in, and This is the result of the first information exchange. and This is the second information exchange, which is equivalent to the present invention using the adaptive information exchange module to perform two information exchanges; Represents the output of the current module, that is, the features of T and R output by each layer of decoder.

[0073] like Figure 6 As shown, the first column from top to bottom is 4 embodiments, namely photos contaminated by reflection (which can be understood as pictures with reflection), the second column is the predicted effect of the transmission layer of these 4 embodiments, and the third column is the predicted effect of the reflection layer of these 4 embodiments. Step 3 extracts the features of the 4 embodiments from top to bottom in the first column. The transmission features and reflection features are coupled, so they are called coupled features. After processing in step 456, the final transmission and reflection prediction maps are obtained. The entire reflection removal process is to calculate the super-column features for the reflection-contaminated images in the first column respectively, and then use the encoder to extract the coupled features 4 times (the 1 / 4 instance normalization strategy is used in the extraction process), and calculate the differential features 4 times, and then use the decoder (adaptive information exchange module) to decode 4 times, and finally obtain the predicted transmission map and reflection map.

Claims

1. A single image reflection removal method based on feature difference enhancement, characterized by: The following steps are involved: Step 1: Acquire a reflection-contaminated image and a supervisory image, where the supervisory image includes a transmission image and a reflection image, and preprocess the reflection-contaminated image and the supervisory image to obtain preprocessed reflection-contaminated image and supervisory image; Step 2: Input the pre-processed reflection-contaminated image into the pre-trained VGG-19 network to extract the hypercolumn features with 1473 channels; Then, the hypercolumn features are connected with the reflection-contaminated image by channel to form the input features of the network; Step 3: The input features obtained in step 2 are fed into a shared encoder with shared parameters, and four downsampling operations are performed in sequence to generate feature maps of four different scales and resolutions, thereby extracting coupled features. Each downsampling operation is normalized using the 1 / 4 instance normalization strategy. Step 4: In the decoders that predict the transmission map and the reflection map respectively, the features output by the previous decoder and the coupled features of the encoder at the corresponding scale are input into the feature difference enhancement module. The feature difference enhancement module calculates the corresponding transmission enhancement features of the predicted transmission map T and the reflection enhancement features of the reflection map R. The specific method for obtaining the enhanced features of the predicted transmission map and the reflection map is as follows: For the coupled features extracted from the i-th layer of the encoder obtained in step 3 The feature difference enhancement module starts with the coupled features The transmission features and reflection features of the previous stage of the corresponding decoder are subtracted from each other; then, the result of the feature difference is sent to the attention module to obtain a favorable differential feature and Then, in the differential features and Add Finally, to join The differential characteristics of and Make jump connections by channel and obtain corresponding enhanced features and The expression is as follows: Step 5: The transmission enhancement features and reflection enhancement features obtained in step 4 are then input into the adaptive information exchange module, so that the two branches of the predicted transmission map T and the reflection map R perform adaptive information exchange, and then perform an upsampling operation; Step 6: After the four-stage upsampling in step 5, the dual-branch decoder outputs the predicted transmission map and reflection map through a convolutional layer respectively.

2. The single image reflection removal method based on feature difference enhancement according to claim 1, characterized in that: In step 2, a pre-trained VGG network is used as the backbone network when extracting hypercolumn features.

3. The single image reflection removal method based on feature difference enhancement according to claim 1, characterized in that: The structure of the 1 / 4 instance normalization strategy is as follows: For the input feature F in each layer Encoder I ,First, use the channel attention mechanism CAM to weight its channels and select the input feature F I The characteristic F of the first 1 / 4 of the channel 1 / 4 Send it to the instance normalization layer to obtain the instance normalized feature F IN ; Then, the instance normalized feature F IN With input feature F I The remaining features F 3 / 4 Connect and fuse to get feature F con ; Finally, use one convolution to transform the feature F con and the remaining features F 3 / 4 Perform convolution, using input features F I Perform jump connections on the convolution results to obtain the normalized output feature F. out .

4. The single image reflection removal method based on feature difference enhancement according to claim 1, characterized in that: The specific processing method of the adaptive information exchange module in step 5 is: For processing transmission enhancement feature F ET , reflection enhancement feature F ER , firstly, a 3×3 convolution is performed to extract the corresponding features and the number of channels is halved; Then apply the ReLU activation function to get and and It is the information retained after the ReLU activation function. and It is the information discarded after the ReLU activation function; The calculation formula is defined as follows: Next, and Adaptively retain the beneficial information in the discarded information through the 3×3 convolution layer, and discard redundant neurons through ReLU again; Finally, the processed information is added to the opposite branch, i.e. the processed Add to the branch of prediction T, processed Add it to the branch of prediction R to get the corresponding information exchange result.

Images