Shadow removal method based on characterization decoupling

Abstract

The application discloses a shadow removal method based on representation disentanglement, which comprises the following steps: pre-processing a sample image and inputting the sample image into a convolutional neural network comprising a shadow encoder, a content encoder and a decoder, so as to obtain a content feature map and a shadow feature map through the encoders respectively; performing average pooling operation on corresponding features in the shadow feature map by using a pre-labeled shadow range, obtaining a shadow feature vector, fusing the shadow feature vector with the content feature map, inputting the fused feature map into the decoder, and training the encoders and the decoder; and finally inputting a to-be-processed image into the trained content encoder and the decoder to realize shadow removal. The application can better remove the shadow covering on the background and achieve a more accurate effect than traditional methods.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to a shadow removal method for an actual shadow image by using a deep learning model and a technology based on representation decoupling. BACKGROUND

[0002] At present, the technologies for removing shadows in images include a method of constructing a physical optical model to use some prior knowledge to restore normal illumination of a shadow area and a shadow removal work and a shadow removal technology based on a deep learning method, wherein the shadow removal work based on the deep learning method mainly learns a high-level context feature to remove the shadow based on the feature. The method can be divided into two categories, a method of using a traditional light model as prior knowledge and a deep learning method without using the traditional light model. The method of using the traditional light model as the prior knowledge generates an overexposed image first based on an existing shadow image, and then uses a deep learning method to fuse the two to obtain a final shadow-removed image. The deep learning method without using the traditional light model hopes to directly remove the shadow by the deep learning model to obtain an image without the shadow. However, both of them cannot obtain a perfect representation of removing the shadow, so that the shadow cannot be completely removed, and the representation decoupling of the shadow and the background content will solve this problem. SUMMARY

[0003] The present application proposes a shadow removal method based on representation decoupling to solve the defects that the prior art cannot obtain a perfect representation of removing the shadow and cannot completely remove the shadow. The method can decouple the intermediate representation of the image into a background content representation and a shadow representation to obtain a purer content representation and better remove the shadow representation, so that the shadow covering the background can be better removed, and a more accurate result than the traditional method can be achieved.

[0004] The present application is implemented by the following technical solutions:

[0005] The present application relates to a shadow removal method based on representation decoupling. A sample image is preprocessed and input into a convolutional neural network containing a shadow encoder, a content encoder and a decoder. A content feature map and a shadow feature map are obtained by the encoders respectively. A pre-labeled shadow range is used to perform an average pooling operation on corresponding features in the shadow feature map to obtain a shadow feature vector, which is fused with the content feature map and input into the decoder to train the encoders and the decoder. Finally, a to-be-processed image is input into the trained content encoder and decoder to remove the shadow.

[0006] The sample image includes a sample image with shadows, a corresponding sample image without shadows, and a sample image without shadows.

[0007] The fusion refers to a region-based adaptive instance normalization operation (Region AdaIN) on the shadow feature vector and the content feature map in a region-based adaptive instance normalization manner.

[0008] The training of the encoder and the decoder specifically includes:

[0009] 1) The sample image with shadows is respectively input into the content encoder and the shadow encoder to generate a corresponding content feature map and a shadow feature map, the shadow feature vector is obtained by performing a region average pooling operation on the shadow feature map, the content feature map is fused with the shadow feature vector and input into the decoder, the output image is reconstructed to obtain a generated image with shadows, the content feature map is input into the decoder, and the output image is reconstructed to obtain a generated image without shadows, thereby constructing a complete reconstruction loss.

[0010] The shadow reconstruction refers to reconstructing the generated image with shadows to the sample image with shadows.

[0011] The shadow reconstruction refers to reconstructing the generated image with shadows to the sample image with shadows.

[0012] The complete reconstruction loss is composed of the losses used in the above two reconstructions.

[0013] 2) The sample image without shadows is input into the content encoder to generate a content feature map, the generated content feature map is fused with the shadow feature vector in step 1) and input into the decoder to obtain a reconstructed image with shadows, and the image is input into the discriminator together with the sample image with shadows in step 1) and the shadow mask of the image to generate a discrimination loss.

[0014] 3) The reconstructed image with shadows generated by the decoder in step 2) is input into the content encoder again to obtain a corresponding content feature map, and the content feature map loss is obtained by making the content feature map similar to the content feature map generated in step 2) through a loss function.

[0015] 4) The reconstruction loss calculated in step 1), the discrimination loss calculated in step 2), and the content feature map loss calculated in step 3) are weighted and summed to realize the training of the encoder and the decoder. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 It is a network model schematic diagram of the present application.

[0017] Figure 2 Example effect schematic diagram. DETAILED DESCRIPTION

[0018] The embodiment relates to a shadow removal method based on characteristic decoupling, a sample image is preprocessed and input into a convolutional neural network containing a shadow encoder, a content encoder and a decoder, content feature maps and shadow feature maps are obtained through the encoders respectively; a corresponding feature in the shadow feature map is subjected to average pooling operation using a pre-labeled shadow range, a shadow feature vector is obtained, and then the shadow feature vector is fused with the content feature map and input into the decoder, so that the encoder and the decoder are trained; finally, a to-be-processed image is input into the trained content encoder and decoder, so that shadow removal is realized.

[0019] The fusion refers to that the content feature map is subjected to region-based adaptive instance normalization operation (Region AdaIN) using region-based adaptive instance normalization operation of the shadow feature vector.

[0020] The training of the encoder and the decoder refers to that a sample image with a shadow is input into the content encoder and the shadow encoder to generate corresponding content feature maps and shadow feature maps, the shadow feature vector is obtained through region average pooling operation of the shadow feature map, the shadow feature vector is fused with the content feature map and input into the decoder, and an output image is obtained, and the output image is reconstructed to the input image with a shadow; and the content feature map is directly input into the decoder, and an output image is obtained, and the output image is reconstructed to a sample image without a shadow corresponding to the input sample.

[0021] In addition, a sample image without a shadow is input into the content encoder to generate a content feature map, the generated content feature map is fused with the shadow feature vector in the above step and input into the decoder to obtain a reconstructed image with a shadow, and the reconstructed image with a shadow is input into a discriminator together with the sample image with a shadow input in the above step and a shadow mask of the image to generate a discrimination loss.

[0022] The reconstructed image with a shadow generated by the decoder in the above step is input into the content encoder again to obtain a corresponding content feature map, and the content feature map is made to be the same as the content feature map generated in the above step through a loss function, so that a content feature map loss is obtained.

[0023] The convolutional neural network comprises a content encoder, a shadow encoder, a decoder and a discriminator, wherein: the content encoder obtains a content feature map through content feature decoupling according to an input sample image, the shadow encoder obtains a shadow feature map through shadow feature decoupling according to an input sample image with a shadow, the decoder obtains a corresponding output image according to an input feature map, and the discriminator discriminates whether an input image with a shadow and a shadow mask are a real shadow image or a generated shadow image.

[0024] The training of the encoder and decoder specifically includes:

[0025] Step 1) Constructing the reconstruction loss: For an input image I′ with shadows... A First, use the content encoder E C Get I′ A Content Feature Map Next The image is input into decoder G to obtain the deshaded image. Because the decoder's input only contains content feature maps and lacks shadow information, the decoder's output should be a shadowless image, which, along with the shadowless label image I, should be the same. A Construct the reconstruction loss, including L1 loss and VGG loss: in: This represents the activation value of the k-th layer in the VGG16 network.

[0026] Step 2) For I′ A Input it into the shadow encoder E S I′ A Shadow feature map The corresponding features in the feature map are subjected to average pooling using a pre-labeled shadow mask to obtain the shadow feature vector. For the obtained shadow feature vector and content feature map The two are merged using the RegionAdaIN operation, which is a region-based adaptive instance normalization. Where z is the activation value of the previous convolutional layer, μ and σ are the channel-level average and channel-level standard deviation operations, and γ and β are parameters obtained through a multilayer perceptron. Applying adaptive instance normalization only to the shadow mask region is called region-based adaptive instance normalization. Residual blocks can be equipped to the adaptive instance normalization layer, whose parameters are dynamically generated from the shadow feature vector by the multilayer perceptron. For regions outside the mask, their feature values ​​remain unchanged. The fused feature map is then passed through a decoder G to obtain the image with shadows. This image uses the input shaded image I′ A Supervision of reconstruction loss is performed. Loss function.

[0027] Step 3) Construct the discriminator loss: except for the image I′ with shadows A In addition, a shadowless image I was used. B For this image, it is first input into the content encoder E. C In the middle, its content feature map is obtained. Next, the shadow feature vector is processed using region-based adaptive instance normalization. Content Feature Map The images are then fused together and then decoded by the decoder G to obtain the image with shadows. expect and I′ A It contains similar shadow information. However, There is no direct supervisory information. Therefore, a discriminator is used to distinguish between real and false shadows in order to improve performance. The realism of the shadows. Specifically, As the generator output, I′ A As the real sample input and I′ A shadow mask m A The input is fed into the discriminator D, and the discriminator loss L is constructed. adv =V(D, G), the desired result is a shadowed image. It is closer to the real image.

[0028] Step 4) Constructing Content Feature Map Loss: To ensure the generated image has shadows Reserve I B Content information, for generated images with shadows Input it back into the content encoder E C In the process, a new content feature map is obtained. for and the first obtained content feature map Constructing an L2 loss makes the values ​​of the two more similar:

[0029] Step 5) Calculate the overall loss function of the model L = L rec +L′ rec +λ adv L adv +λ2L2, where: λ adv λ2 and λ2 are hyperparameters.

[0030] Through specific practical experiments, two datasets, SRD and ISTD, were used. SRD contains 408 pairs of images with and without shadows, but does not include shadow masks. A simple threshold detection algorithm is used to obtain the shadow mask by detecting the differences between the images with and without shadows. The ISTD dataset has 1330 training images and 540 test images. Each set of images contains images with and without shadows, as well as shadow masks.

[0031] During training, the Adam optimizer was used with a batch size of 8 and an initial learning rate of 0.001. λ advThe values of λ1 and λ2 are set to 0.5 and 1 respectively. During the test, STCGAN (CVPR2018), DHAN (AAAI2020), FusionNet+RefineNet (CVPR2021) are selected to compare with the method. ST-CGAN (CVPR2018) analyzes the relationship between shadow detection and removal, and then proposes to use a stacked conditional generative adversarial network model to simultaneously perform shadow detection and removal.

[0032] DHAN (AAAI2020) is proposed to hierarchically aggregate attention and predict the context features of the results, so as to generate shadow-free images. FusionNet+RefineNet (CVPR2021) first generates various overexposed images of different exposure, and then mixes the original image and these overexposed images to generate shadow-removed images.

[0033] On the SRD and ISTD datasets, the RMSE indicators of the method and the above two methods in the shadow area, non-shadow area and the whole image are compared respectively, to measure the difference between the output image and the target image in the three areas. The results are shown in Table 1. This indicator can largely show the comprehensive ability of the method in shadow removal. From the table, it can be seen that the method is better than the above three methods on the two datasets, which proves the advantage of the method in quantitative indicators and embodies the effectiveness of the method.

[0034] Table 1: Quantitative indicator results of various shadow removal algorithms

[0035]

[0036]

[0037] To further verify the effectiveness of the method, 50 users are invited to compare the output of the method with the output of DHAN (AAAI2020) and FusionNet+RefineNet (CVPR2021). The outputs of 540 test images on ISTD are all displayed to the users, and the users select the best shadow removal effect among the three output images. The user needs to evaluate a group of images according to the degree of shadow residue, the smoothness of shadow boundary and the naturalness of shadow area color, and finally give the optimal choice. Here, the number of optimal evaluation is used as the evaluation indicator, as shown in Table 2, which proves the effectiveness of the method.

[0038] Table 2: User survey results of various shadow removal algorithms

[0039] Method Optimal evaluation number DHAN 8064 FusionNet+RefineNet 8123 Ours 10813

[0040] In addition, the effectiveness of the method will be embodied through the visual display of the results, such as Figure 2 As shown in the figure, the method is more thorough than the other two methods in shadow removal, which further proves the effectiveness of the method.

[0041] Compared with the prior art, the RMSE index of the method in the shadow area, the non-shadow area and the whole picture exceeds that of the existing method, the optimal evaluation quantity in the user survey exceeds that of the existing method, and the shadow removal effect in the visual display result is more thorough.

[0042] The present application decouples the representation of shadows and background content, can better remove the shadows overlaid on the background, and achieves more accurate results than previous methods. Compared with other deep learning shadow removal methods, the present application tries to decouple the relationship between shadows and background content and recombine them using related technologies to obtain the corresponding expected shadowed image, thereby more accurately decoupling the representation of shadows and background content and further improving the accuracy of the model in shadow removal.

[0043] The above specific embodiments can be adjusted in different ways by those skilled in the art without departing from the principles and purposes of the present application, the protection scope of the present application is subject to the claims and is not limited by the above specific embodiments, and each implementation scheme within the scope is subject to the constraints of the present application.

Claims

1. A shadow removal method based on characterizing decoupling, characterized in that, The sample image is preprocessed and input into a convolutional neural network comprising a shadow encoder, a content encoder and a decoder, to obtain a content feature map and a shadow feature map through the encoders respectively; the corresponding features in the shadow feature map are subjected to average pooling operation using a pre-labeled shadow range to obtain a shadow feature vector, which is fused with the content feature map and input into the decoder, so as to train the encoders and the decoder; Finally, the image to be processed is input into the trained content encoder and decoder to realize shadow removal; The sample image comprises a sample image with shadow, a sample image with shadow removed and a sample image without shadow. The training of the encoders and the decoder comprises: 1) the sample image with shadow is input into the content encoder and the shadow encoder to generate a corresponding content feature map and a shadow feature map respectively, the shadow feature map is subjected to region average pooling operation to obtain a shadow feature vector, which is fused with the content feature map and input into the decoder, and the output image obtained is subjected to shadow reconstruction to obtain a generated image with shadow; the content feature map is input into the decoder, and the output image obtained is subjected to shadowless reconstruction to obtain a generated image without shadow, so as to construct a complete reconstruction loss; 2) the sample image without shadow is input into the content encoder to generate a content feature map, then the generated content feature map is fused with the shadow feature vector in step 1) and input into the decoder to obtain a reconstructed image with shadow, which is input into the discriminator together with the sample image with shadow in step 1) and the shadow mask of the image to generate a discrimination loss; 3) the reconstructed image with shadow generated by the decoder in step 2) is input into the content encoder again to obtain a corresponding content feature map, which is similar to the content feature map generated in step 2) through a loss function, so as to obtain a content feature map loss; 4) the reconstruction loss obtained in step 1), the discrimination loss obtained in step 2) and the content feature map loss obtained in step 3) are weighted and summed to realize the training of the encoders and the decoder; The shadow reconstruction refers to reconstructing the generated image with shadow to the sample image with shadow; the shadowless reconstruction refers to reconstructing the generated image without shadow to the sample image with shadow removed; and the complete reconstruction loss comprises losses used in the above two reconstructions.

2. The decoupled representation-based shadow removal method of claim 1, wherein, The fusion refers to region-based adaptive instance normalization operation on the shadow feature vector and the content feature map by using region-based adaptive instance normalization.

3. The shadow removal method based on characterizing decoupling according to claim 1 or 2, characterized in that, The training of the encoders and the decoder comprises: Step 1) Constructing the reconstruction loss: For an input image with shadows... First, use the content encoder. get Content Feature Map Next Input to decoder Obtain the image with the shadow removed. Because the decoder's input only contains content feature maps and lacks shadow information, the decoder's output should be a shadowless image, which, along with the shadowless label image,... Constructing the reconstruction loss, including Loss and loss: ,in: This represents the activation value of the k-th layer in the VGG16 network. Step 2) for , input it into the shadow encoder to get the shadow feature map , use the pre-labeled shadow mask to perform average pooling operation on the corresponding features in the feature map to get the shadow feature vector , for the obtained shadow feature vector and the content feature map , use the region-based adaptive instance normalization operation RegionAdaIN to fuse the two, adaptive instance normalization , wherein: is the value after activation of the previous convolution layer, and are the average and standard deviation operations on the channel level, and are parameters obtained by a multi-layer perceptron; the adaptive instance normalization operation is only applied to the region of the shadow mask, which is called region-based adaptive instance normalization; the adaptive instance normalization layer is equipped with a residual block, and the parameters of the residual block are dynamically generated by a multi-layer perceptron from the shadow feature vector; for the region outside the mask, the feature value is kept unchanged; then the fused feature map is input into the decoder to get the shadow image , which is supervised by the reconstruction loss using the input shadow image ; the loss function ;​ Step 3) Building discriminator loss: In addition to the shaded image , the unshaded image is also used; for this image, first input it into the content encoder to get its content feature map , then use the operation of region-based adaptive instance normalization to fuse the shadow feature vector with the content feature map , and then pass through the decoder to get the shaded image ; it is expected and to contain similar shadow information; however, there is no direct supervision information; therefore, use the discriminator to distinguish true and false shadows to improve the authenticity of the shadows in ; specifically, input as the generator output, as the real sample input, and the shadow mask of into the discriminator , and build the discriminator loss , hoping to make the generated shaded image closer to the real image; Step 4) Constructing the content feature map loss: to make the generated shaded image preserve the content information, for the generated shaded image , it is input again to the content encoder , to get a new content feature map , for and the content feature map obtained for the first time , construct the loss to make the values of the two more close: ; Step 5) Compute the total loss function of the model where: and are hyperparameters.

Images