A method for removing shadows from unpaired images

Abstract

The present invention relates to a method for removing shadows from unpaired images. A TC-ShadowGAN network model designs a generative network STN, namely a Shadow Transformation Network (STN). Two sets of encoder-decoder based generative sub-networks are used to generate two shadow-free networks from a shadow image. A target consistency constraint is used to restrict the consistency of the two generated shadow-free images. In addition, in the absence of paired shadow-free data supervision, TC-ShadowGAN generates a true shadow-free image by learning common illumination and brightness features from various shadow-free natural images, and specifies two discriminators using different real shadow-free images to constrain the consistency of illumination, color and texture between the generated shadow-free area and the real shadow-free area.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method for removing shadows from unpaired images. Background Art

[0002] Identifying and removing shadows in natural images is very important and critical for many computer vision tasks, such as object detection and tracking. Shadow removal is actually an image inpainting problem, where a shadow-free image is restored by adding the input shadow image with the shadow residual elements, or by multiplying the input shadow image with the shadow matrix (or shadow factor).

[0003] In practical applications, due to environmental uncertainty, on the one hand, the shadows cast by the same object in different environments will present different shapes and depth patterns. On the other hand, the textures of some objects or backgrounds may be similar to some shadow areas in the image, resulting in unexpected visual confusion. Therefore, shadow removal has always been a difficult research topic.

[0004] At present, shadow removal methods based on deep learning have achieved remarkable results. In existing shadow removal methods, shadow matrices or masks and removal operations are all learned from paired shadow removal datasets under supervision. However, paired supervised learning still has great limitations in practical applications. First, high-quality paired shadow and non-shadow images are difficult to obtain and the acquisition cost is high. Second, the current largest paired shadow removal dataset contains only 100 unique scenes. Training based on this dataset can only learn limited scenes and shadow patterns. Third, due to the changes in camera position and lighting in the process of artificially generating data, noise is induced between paired shadow and non-shadow labels. In reality, unpaired natural scene images containing shadow and non-shadow images are relatively easy to obtain, so we are able to collect old shadow images and mismatched non-shadow images of various scenes to promote the learning of unpaired shadow removal. However, existing shadow removal methods are difficult to achieve satisfactory results. Summary of the invention

[0005] The purpose of the present invention is to provide a non-paired image shadow removal method to address the deficiencies in the prior art, which can use non-paired images to achieve shadow removal of shadow images, and the processed shadow-free image has good quality.

[0006] The technical solution of the present invention is: a method for removing shadows from unpaired images, the steps are as follows:

[0007] S1. Establish a data set, which includes: images with shadows and images without shadows;

[0008] S2. Construct the TC-ShadowGAN network model and use two shadow residual generators G RES1 and G RES2 A generating network STN is formed, and two discriminators D1 and D2 are used to form an identification network USDN. A generator G in the generating network STN RES1 Corresponding to a discriminator D1 in the identification network USDN, another generator G RES2 Corresponding to another discriminator D2 in the identification network USDN, a TC-ShadowGAN network model with two sets of unidirectional GAN ​​networks is formed;

[0009] S3. Using the shadow images and non-shadow images in the dataset, the final loss function of TC-ShadowGAN is obtained by the weighted sum of the adversarial loss function, the target consistency loss function and the identity loss function. The entire TC-ShadowGAN network model is trained through the final loss function, where:

[0010] Adopt adversarial constraints to force two generators and two discriminators to jointly optimize the adversarial loss function To train the GAN network;

[0011] Adopt target consistency constraints and use target consistency loss function Enable the two groups of GAN networks to learn as a whole;

[0012] Using the generator G RESi Identity loss function for shadow-free images Train the generator by making it learn from real shadow-free images;

[0013] S4. After the TC-ShadowGAN network model is trained, image conversion is performed. Given a shadow image I X , and finally the TC-ShadowGAN network model outputs a shadow-free image.

[0014] I Y =MSM(I X +G RES1 (I X ), I X +G RES2 (I X ))

[0015] In the formula, I Y is a shadow-free image, G RES1 (I X ) and G RES2 (I X ) are the feature representations of the shadow residual image by the two generators respectively.

[0016] Furthermore, for S2, the generator consists of an encoder G STEi and a decoder G SRDi In the network training process, the encoder learns to transform the shadow image I Z Transformed from the image domain into a feature combination, i.e., F i =G STEi (I z ), (i∈{1,2}), where F i is a feature combination, and the decoder is used to decode the feature representation from the feature domain to the shadow residual image, that is, G RESi (I Z )=G SRDi (F i ), (i∈{1,2}), so that the generator G RES1 and G RES2 Different features of the shadow-free image are encoded respectively, and finally two independent shadow-free images I are obtained by element-wise addition of the input shadow image. y1 and I y2 .

[0017] Furthermore, the adversarial loss function is:

[0018]

[0019] Where P Y Represents the data distribution of shadow-free images in the dataset;

[0020] P X Represents the data distribution of shadow images in the dataset;

[0021] I rni represents a sample of the shadow-free domain randomly selected from the dataset;

[0022] x represents the distribution P of the shadow image data X Input samples drawn from ;

[0023] D i (I rni ) represents the probability that the discriminator outputs a shadow-free image sample;

[0024] D i (x+G RESi (x)) represents the probability of the discriminator outputting a shadow image sample.

[0025] Furthermore, the I rni I rn1 and I rn2 A different sample is selected in each iteration.

[0026] Furthermore, the target consistency loss function is:

[0027]

[0028] In the formula, G RES1 (x), G RES2 (x) represents the feature representation of each generator from the feature domain to the shadow residual image.

[0029] Furthermore, the identity loss function is:

[0030]

[0031] In the formula, G RESi (I rni ) is the feature representation of the generator for shadow-free image samples.

[0032] Furthermore, the final loss function of the TC-ShadowGAN network model is:

[0033]

[0034] Where λ1, λ2, and λ3 represent trade-off hyperparameters, which are used to weigh the contributions of adversarial loss, target consistency loss, and identity loss, respectively.

[0035] Furthermore, the trade-off hyperparameters λ1, λ2, and λ3 are set to 1, 40, and 5.

[0036] Furthermore, the encoder has 3 convolutional layers with a stride of 2 and 9 residual blocks, the decoder has 3 deconvolutional layers with a stride of 2, and each layer of the encoder and decoder is followed by an instance normalization layer.

[0037] Furthermore, the TC-ShadowGAN network model is optimized using a minimal optimizer.

[0038] Beneficial effects of adopting the above technical solution:

[0039] 1. This method learns a unilateral and single-step generative adversarial network from unpaired shadow and shadow-free data, generates shadow residuals and restores shadow-free images without using paired or unpaired shadow-free or mask labels for training, so that the color and illumination of the generated shadow-free image are consistent with the shadow-free area in the image, and has the quality of a true shadow-free image, thereby achieving shadow removal of the image.

[0040] 2. Without relying on any labels, our method has better performance than existing unpaired shadow removal methods on the unpaired shadow removal dataset in terms of quantitative and qualitative evaluation.

[0041] 3. Compared with the cycle consistency constraint, the one-sided mapping framework of the target consistency constraint adopted in this method can promote the efficient training and reasoning process of the model and is more suitable for shadow removal.

[0042] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a schematic diagram of the network model structure of the present invention;

[0044] Figure 2 Visual comparison of the USR dataset;

[0045] Figure 3 Visual comparison of the ISTD dataset. DETAILED DESCRIPTION

[0046] See also Figure 1 , a non-paired image shadow removal method,TC-ShadowGAN network model designs a generative network STN, namely Shadow Transformation Network (STN), which generates two shadow-free networks from shadow images through two sets of encoder-decoder based generative sub-networks, and adopts target consistency constraints to restrict the consistency of the two generated shadow-free images. In addition, in the absence of paired shadow-free data supervision, TC-ShadowGAN generates true shadow-free images by learning common illumination and brightness features from various shadow-free natural images, and uses different real shadow-free images to specify two discriminators to constrain the consistency of illumination, color and texture between the generated shadow-free area and the real shadow-free area.

[0047] The specific steps are as follows:

[0048] S1. Establish a dataset, which includes: images with shadows and images without shadows, such as the USR dataset and the ISTD dataset. The USR dataset is a unique unpaired shadow removal dataset, which contains 2445 shadow images and 1770 unmatched non-shadow images, while the ISTD dataset is widely used for supervised shadow removal, which contains 1330 paired training images and 540 paired test images.

[0049] S2. Construct the TC-ShadowGAN network model, namely the target consistency generation adversarial network, using two shadow residual generators G RES1 and G RES2 A generating network STN is formed, and two discriminators D1 and D2 are used to form an identification network USDN. A generator G in the generating network STNRES1 Corresponding to a discriminator D1 in the identification network USDN, another generator G RES2 Corresponding to another discriminator D2 in the identification network USDN, a TC-ShadowGAN network model with two sets of unidirectional GAN ​​networks is formed;

[0050] The generator consists of an encoder G STEi and a decoder G SRDi In the network training process, the encoder is used to learn to transform the shadow image I Z Transformed from the image domain into a feature combination, i.e., F i =G STEi (I z ), (i∈{1,2}), where F i is a feature combination, and the decoder is used to decode the feature representation from the feature domain to the shadow residual image, that is, G RESi (I Z )=G SRDi (F i ), (i∈{1,2}), so that the generator G RES1 and G RES2 Different features of the shadow-free image are encoded respectively, and finally two independent shadow-free images I are obtained by element-wise addition of the input shadow image. y1 and I y2 The encoder has 3 convolutional layers with a stride of 2 and 9 residual blocks, the decoder has 3 deconvolutional layers with a stride of 2, and each layer of the encoder and decoder is followed by an instance normalization layer.

[0051] S3. Using the shadow images and non-shadow images in the dataset, the final loss function of TC-ShadowGAN is obtained by the weighted sum of the adversarial loss function, the target consistency loss function and the identity loss function. The entire TC-ShadowGAN network model is trained through the final loss function, and the minimum optimizer is used to optimize the entire TC-ShadowGAN network model.

[0052] The final loss function of the TC-ShadowGAN network model is:

[0053]

[0054] Wherein, λ1, λ2, and λ3 represent trade-off hyperparameters, which are used to weigh the contributions of adversarial loss, target consistency loss, and identity loss, respectively. The trade-off hyperparameters λ1, λ2, and λ3 are set to 1, 40, and 5, respectively, based on experience.

[0055] represents the adversarial loss function;

[0056] represents the target consistency loss function;

[0057] represents the identity loss function.

[0058] Among them, ① the adversarial constraint is adopted to force the two generators and the two discriminators to jointly optimize the adversarial loss function L gan (G RESi , D i )(i∈{1,2}), to train the GAN network, the adversarial loss function is:

[0059] Where P Y Represents the data distribution of shadow-free images in the dataset;

[0060] P X Represents the data distribution of shadow images in the dataset;

[0061] I rni represents a sample of the shadow-free domain randomly selected from the dataset, the I rni I rn1 and I rn2 A different sample is selected in each iteration;

[0062] x represents the distribution P of the shadow image data X Input samples drawn from ;

[0063] D i (I rni ) represents the probability that the discriminator outputs a shadow-free image sample;

[0064] D i (x+G RESi (x)) represents the probability of the discriminator outputting a shadow image sample.

[0065] ②Use target consistency constraints and use target consistency loss function Connect two groups of GAN networks for overall learning;

[0066] The target consistency loss function is:

[0067]

[0068] In the formula, G RES1 (x), G RES2 (x) represents the feature representation of each generator from the feature domain to the shadow residual image.

[0069] Target consistency is used to constrain the shadow-free image I y1 and I y2 The correlation between them makes the output targets of the two generators as close as possible through the target consistency constraint, which can promote the shadow-free image I y1 and I y2 The consistency of G RES1 -D1 and G RES2 The adversarial learning between -D2 can also restrict the output shadow-free targets, thereby avoiding the blurred texture caused by the random interference of the mapping in the cycle consistency constraint.

[0070] ③Use generator G RESi Identity loss function for shadow-free images Train the generator by making it learn from real shadow-free images;

[0071] The identity loss function is:

[0072]

[0073] In the formula, G RESi (I rni ) is the feature representation of the generator for shadow-free image samples.

[0074] Through identity loss, when the input image is a shadow-free image, the consistency between the shadow-free image output by the generator and the input image in the shadow-free area is guaranteed, and the training of the two generators can converge faster.

[0075] In summary, the negative log-likelihood objective of the final loss function of this method is replaced by the minimum square loss of the adversarial loss, so it is more stable and effective than the existing log-likelihood.

[0076] S4. After the TC-ShadowGAN network model is trained, image conversion is performed. Given a shadow image I X , and finally the TC-ShadowGAN network model outputs a shadow-free image.

[0077] I Y =MSM(I X +G RES1 (I X ), I X +G RES2 (I X ))

[0078] In the formula, I Y is a shadow-free image, G RES1 (I X ) and G RES2 (I X) are the feature representations of the shadow residual image by the two generators respectively.

[0079] The experimental results and analysis using the above method are as follows:

[0080] 1. Experimental Setup

[0081] Datasets: USR dataset and ISTD dataset are used. In order to qualitatively evaluate the USR dataset, a two-stage user study is used to evaluate the performance of shadow removal. In the first stage, each participant is randomly selected 10 generated shadow images. After that, the participants rate all 10 shadow-free images, ranging from 1 (poor quality) to 10 (good quality); in the second stage, we conduct the same settings as the first stage, except that the participants are provided with the original shadow image relative to the shadow-free image as a reference. That is, the purpose of the first stage is to evaluate the quality of the shadow-free image, and the second stage is to evaluate the visual effect by comparing the input shadowed image.

[0082] Evaluation Metrics: For quantitative evaluation, the FID and KID scores are calculated, which aim to measure the similarity between the real image and the converted image. In addition, in order to evaluate the correlation between the input shadow image and the generated shadow-free target, the BARAR method (Bidirectional Recursive Attention Residual method) is used to obtain the shadow mask and measure the RMSE (Root Mean Square Error) of the shadow-free area between them.

[0083] Model baseline: The TC-ShadowGAN network model is compared with the Mask-ShadowGAN network model and some GAN network-based unpaired image translation models, including GAN-alone (Pix2Pix without paired supervision), DistanceGAN, CycleGAN, U-GAT-IT, and AttentionGAN on the USR dataset. In addition, on the ISTD dataset, the TC-ShadowGAN network model is compared with some classic paired shadow removal methods, including Gong, Guo, Yang, ST-CGAN, and DSC.

[0084] The encoder of TC-ShadowGAN has 3 convolutional layers with a stride of 2 and 9 residual blocks, and the decoder has 3 deconvolutional layers with a stride of 2. Each layer of the encoder and decoder is followed by an instance normalization layer. During training, the network is trained in a total of 200 rounds, each with a batch size of 1. The network parameters are randomly initialized by a zero-mean Gaussian distribution, and the input image is randomly cropped to a size of 256×256.

[0085] 2. Ablation Study of TC-ShadowGAN Network Model

[0086] Increasing the number of generators: Considering that the strategy of more than two generators can be easily applied to the TC-ShadowGAN network model, we explore the impact of increasing the number of generators. The performance of the TC-ShadowGAN network model was tested with 3-5 generators, and the FID results are shown in Table 1. After comparing the generated images and quantitative results, it was found that increasing the number of generators did not significantly improve the shadow removal effect, so setting two generators was the best.

[0087] Network reasoning with attention mechanism: As shown in Table 1, different attention mechanisms are induced into the TC-ShadowGAN network model. However, its performance does not lead to better results. One reason may be that the attention module only focuses on the shadow mask, but filling the shadow requires more global information. Therefore, simply using the attention mechanism to improve the results will make the model more difficult to learn, which can also explain that the shadow removal effect of this method is better.

[0088] Table 1 FID results of ablation study on USR dataset

[0089]

[0090] 3. Performance Evaluation

[0091] Study on USR Unpaired Dataset: Since the USR dataset does not have labels corresponding to shadow images, a user study is conducted to conduct a qualitative visual comparison of different models. 10 participants are invited and 100 ratings (10 participants × 10 images) are obtained for each method in the first and second phases, as described in the experimental setup section. The mean and standard deviation of these ratings are calculated and the results are summarized in the second and third columns of Table 2. From Table 2, it can be concluded that the TC-ShadowGAN network model of our method achieves the highest rating performance in both the first and second phases. Moreover, although the Mask-ShadowGAN model achieves the best performance among the models based on cycle consistency (third column, sixth row), the effect is still inferior to the TC-ShadowGAN network model based on target consistency constraint. In addition, since the overall framework of the TC-ShadowGAN network model is to maintain the correlation between the input image and the generated target through adversarial learning, it also shows a greater advantage in the rating of the second phase.

[0092] Quantitative evaluation on USR dataset: The last 3 columns of Table 2 show the FID, KID, and RMSE of all compared methods. In addition, the visual comparison of shadow removal results on USR dataset is shown in Figure 2. Figure 2As shown in the figure, the input image (a) and the output results after the input image is processed by GAN-alone (b), DistanceGAN (c), CycleGAN (d), U-GAT-IT (e), AttentionGAN (f), Mask-ShadowGAN (g), and TC-ShadowGAN (h) respectively. Figure 2 It can be seen that the TC-ShadowGAN network model obtains the smallest FID and KID scores compared to all other methods. It is particularly important to note that since AttentionGAN directly copies the pixels of the input image, the lower RMSE of AttentionGAN does not mean that it can achieve better performance. In addition, the results of GAN-alone and DistanceGAN show that insufficient constraints will lead to perturbations and artifacts. Moreover, Figure 2 Columns (d)-(g) in the middle are the results of models based on cycle consistency. The outputs of these models all retain some shadow features, resulting in the failure of shadow removal. This is because the cycle consistency constraint assumes that the generated shadow-free image can be reconstructed into the input shadow image without additional information. Although Mask-ShadowGAN facilitates the reverse mapping by using the generated shadow mask, the resulting binary shadow mask is not completely accurate, resulting in a large appearance deviation between the output target and the true shadow-free target. In comparison, as shown in Tables 2 and Figure 2 As shown in Figure 2, the TC-ShadowGAN proposed in this method can perfectly remove the shadow area while maintaining a high correlation between the non-shadow area between the restored shadow-free image and the input image.

[0093] Table 2 Qualitative and quantitative evaluation results of USR dataset

[0094]

[0095] Evaluation on paired ISTD dataset: The TC-ShadowGAN network model is compared with some classic supervised methods on the ISTD dataset. The pairing order of shadow and non-shadow images is shuffled and random crops are used to avoid pixel matching during training. In addition, instead of using the binary shadow mask provided in the dataset, the performance is evaluated by calculating the RMSE between the generated shadow-free images and the annotated ground truth, considering 3 validation regions: (i) shadow region, (ii) non-shadow region, and (iii) the entire image. In addition, to avoid the noise of the ISTD dataset, the RMSE of the generated shadow-free images and the input shadow-free images is measured in the shadow-free region (NI).

[0096] The evaluation results are summarized in Table 3. It can be seen that the TC-ShadowGAN network model achieves comparable performance to the supervised method, especially in the shadow-free area (NI). Figure 3 The visual comparison results of the ISTD dataset are shown in Figure 2. Figure 3 From left to right in the figure are the shadow image (a), the real shadow-free image (b), and the output results after processing by Gong et al. (c), Guo et al. (d), Yang et al. (e), ST-CGAN (f), DSC (g) and TC-ShadowGAN (h), respectively. It can be concluded that due to the noise of the training labels, the supervised method sometimes cannot perfectly remove shadows (such as the first row) or keep the non-shadow area unchanged (such as the second row). In comparison, the TC-ShadowGAN network model of this method can effectively remove shadows while retaining the content and brightness of the non-shadow area.

[0097] Table 3 RMSE results on the ISTD dataset, NI is the RMSE result compared with the shadowless area of ​​the input image

[0098]

[0099] Therefore, the target consistency generative adversarial network TC-ShadowGAN of this method is used for unpaired shadow removal. The entire network is based on the structure of the dual GAN ​​network and strictly limits the correlation between the generated shadow-free target and the input shadow image by using the target consistency constant. Experimental results show that this method has excellent performance and good restoration quality, outperforming the existing unpaired shadow removal methods by 14.9% in FID and 31.5% in KID. In addition, the TC-ShadowGAN network model can also be applied to more shadow scene tasks, such as shadow removal in videos.

Claims

1. A method for removing shadows from unpaired images, characterized in that Here are the steps: S1. Establish a data set, which includes: images with shadows and images without shadows; S2. Construct the TC-ShadowGAN network model and use two shadow residual generators G RES1 and G RES2 A generating network STN is formed, and two discriminators D1 and D2 are used to form an identification network USDN. A generator G in the generating network STN RES1 Corresponding to a discriminator D1 in the identification network USDN, another generator G RES2 Corresponding to another discriminator D2 in the identification network USDN, a TC-ShadowGAN network model with two sets of unidirectional GAN ​​networks is formed; S3. Using the shadow images and non-shadow images in the dataset, the final loss function of TC-ShadowGAN is obtained by the weighted sum of the adversarial loss function, the target consistency loss function and the identity loss function. The entire TC-ShadowGAN network model is trained through the final loss function, where: Adopt adversarial constraints to force two generators and two discriminators to jointly optimize the adversarial loss function To train the GAN network; Adopt target consistency constraints and use target consistency loss function Enable the two groups of GAN networks to learn as a whole; Using the generator G RESi Identity loss function for shadow-free images Train the generator by making it learn from real shadow-free images; S4. After the TC-ShadowGAN network model is trained, image conversion is performed. Given a shadow image I X , and finally the TC-ShadowGAN network model outputs a shadow-free image. I Y =MSM(I X +G RES1 (I X ),I X +G RES2 (I X )) In the formula, I Y is a shadow-free image, G RES1 (I X ) and G RES2 (I X ) are the feature representations of the shadow residual image by the two generators respectively.

2. The method for removing shadows from unpaired images according to claim 1, characterized in that: For S2, the generator consists of an encoder G STEi and a decoder G SRDi In the network training process, the encoder learns to transform the shadow image I Z Transformed from the image domain into a feature combination, i.e., F i =G STEi (I z ), (i∈{1,2}), where F i is a feature combination, and the decoder is used to decode the feature representation from the feature domain to the shadow residual image, that is, G RESi (I Z )=G SRDi (F i ), (i∈{1,2}), so that the generator G RES1 and G RES2 Different features of the shadow-free image are encoded respectively, and finally two independent shadow-free images I are obtained by element-wise addition of the input shadow image. y1 and I y2 .

3. The method for removing shadows from unpaired images according to claim 1, characterized in that: The adversarial loss function is: Where P Y Represents the data distribution of shadow-free images in the dataset; P X Represents the data distribution of shadow images in the dataset; I rni represents a sample of the shadow-free domain randomly selected from the dataset; x represents the distribution P of the shadow image data X Input samples drawn from ; D i (I rni ) represents the probability that the discriminator outputs a shadow-free image sample; D i (x+G RESi (x)) represents the probability of the discriminator outputting a shadow image sample.

4. The method for removing shadows from unpaired images according to claim 3, characterized in that: I rni I rn1 and I rn2 A different sample is selected in each iteration.

5. The method for removing shadows from unpaired images according to claim 1, characterized in that: The target consistency loss function is: In the formula, G RES1 (x), G RES2 (x) represents the feature representation of each generator from the feature domain to the shadow residual image.

6. The method for removing shadows from unpaired images according to claim 1, characterized in that: The identity loss function is: In the formula, G RESi (I rni ) is the feature representation of the generator for shadow-free image samples.

7. The method for removing shadows from unpaired images according to claim 1, characterized in that: The final loss function of the TC-ShadowGAN network model is: Where λ1, λ2, and λ3 represent trade-off hyperparameters, which are used to weigh the contributions of adversarial loss, target consistency loss, and identity loss, respectively.

8. The method for removing shadows from unpaired images according to claim 7, characterized in that: The trade-off hyperparameters λ1, λ2, and λ3 are set to 1, 40, and 5.

9. The method for removing shadows from unpaired images according to claim 2, characterized in that: The encoder has 3 convolutional layers with a stride of 2 and 9 residual blocks, the decoder has 3 deconvolutional layers with a stride of 2, and each layer of the encoder and decoder is followed by an instance normalization layer.

10. The method for removing shadows from unpaired images according to claim 1, characterized in that: The TC-ShadowGAN network model is optimized using a minimal optimizer.

Images