A deep learning-based high-resolution remote sensing image shadow removal method

Abstract

The application discloses a high-resolution remote sensing image shadow removing method based on deep learning, and comprises the following steps: constructing a high-resolution remote sensing image shadow removing network model; the remote sensing image shadow removing network model is based on a GAN architecture, and comprises a generation network and a discrimination network net_D; the generation network generates a false image for cheating the discrimination network; the generation network comprises a shadow eliminator net_Gf, a shadow generator net_Gs and a shadow optimizer net_Gr; the generation network and the discrimination network are alternately trained; the high-resolution remote sensing image shadow removing network model is used to complete shadow removing, and a remote sensing image after shadow removing is obtained. The application can effectively remove shadows for a remote sensing area with complex structures.

Description

Technical Field

[0001] This invention relates to the field of image shadow removal, and more particularly to a method for removing shadows from high-resolution remote sensing images based on deep learning. Background Technology

[0002] Satellite remote sensing imagery can rapidly provide information about the Earth's surface. The development of high-resolution satellite remote sensing imagery has made in-depth applications of remote sensing imagery possible, thus providing favorable conditions for GIS data updates and GIS applications. It is also of great significance for map updates, image matching, and target detection.

[0003] In remote sensing imagery, shadows are primarily formed due to objects partially or completely blocking sunlight, resulting in corresponding areas being less bright than other areas. This situation poses challenges to other remote sensing tasks. Currently, shadow removal from high-resolution remote sensing images faces difficulties in acquisition and is costly. Furthermore, high-resolution remote sensing images often exhibit diverse land cover types. Using traditional methods, the complex texture information leads to significant color differences in the restored shadow areas. While deep learning methods can stably restore color differences, texture information is lost. Moreover, previous deep learning discrimination networks treat the image as a whole during convolution. For shadow removal tasks, shadows and non-shadow areas belong to the same image. This traditional discrimination network is affected by other shadow areas when judging the authenticity of shadowless images, reducing the success rate of image discrimination and thus impacting the generator's efficiency. Summary of the Invention

[0004] Therefore, the technical problem this invention aims to solve is: how to generate realistic high-resolution remote sensing imagery without shadows. To address this, this invention employs an unsupervised deep learning model for high-resolution remote sensing shadow removal. This method only requires the original image and the corresponding mask image to train the model and perform the removal. This greatly solves the problem of the lack of shadow-free images in high-resolution remote sensing imagery.

[0005] This invention provides a high-resolution remote sensing image shadow removal method based on deep learning, comprising the following steps:

[0006] S101. Construct a high-resolution remote sensing image shadow removal network model; the remote sensing image shadow removal network model is based on a GAN architecture, including a generator network and a discriminator network net_D; the generator network generates fake images to deceive the discriminator network; the generator network includes: a shadow remover net_Gf, a shadow generator net_Gs, and a shadow optimizer net_Gr;

[0007] S102: Alternately train the generator network and the discriminator network: use the generator network to generate fake images to deceive the discriminator network, and update the parameters of the generator network and the discriminator network to finally obtain the high-resolution remote sensing image shadow removal network model.

[0008] S103: Use the trained high-resolution remote sensing image shadow removal network model to remove shadows and obtain the shadow-removed remote sensing image.

[0009] The beneficial effects provided by this invention are as follows: The network proposed in this invention can output more realistic shadowless images without requiring a true-matching shadowless image. Simultaneously, the discriminator can quantitatively determine the authenticity of a specified area based on the region covered by the mask, thereby more accurately updating the discriminator's precision and improving the generator's generation quality. This network overcomes the shortcomings of traditional shadow removal methods, such as complex steps and tonal distortion in the generated images, ultimately producing shadowless images with sharpness and texture features closer to real high-resolution remote sensing images. Attached Figure Description

[0010] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0011] Figure 2 This is a schematic diagram of the training process of each network in this invention;

[0012] Figure 3 This is a rendering of the invention. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0014] Please refer to Figure 1 , Figure 1 This is a schematic diagram of the method flow of the present invention; a high-resolution remote sensing image shadow removal method based on deep learning includes the following steps:

[0015] S101. Construct a high-resolution remote sensing image shadow removal network model; the remote sensing image shadow removal network model is based on a GAN architecture, including a generator network and a discriminator network net_D; the generator network generates fake images to deceive the discriminator network; the generator network includes: a shadow remover net_Gf, a shadow generator net_Gs, and a shadow optimizer net_Gr;

[0016] It should be noted that before training the network, two completely pure shadow and non-shadow datasets are constructed based on the original image and the shadow mask map to provide data support for subsequent model training.

[0017] The generative network consists of a shadow canceller (net_Gf), a shadow generator (net_Gs), and a shadow optimizer (net_Gr). Each layer employs 64 7x7 convolutional kernels, 256 3x3 downsampling layers, 256 3x3 residual convolutional layers, 64 3x3 upsampling layers, 3 7x7 output layers, an InstanceNorm regularization layer, and a ReLU activation function.

[0018] The difference lies in the shadow canceller net_Gf, the shadow generator net_Gs takes a 3-channel RGB image as input, and net_Gr takes a four-channel image synthesized from a 3-channel RGB image and a 1-channel shadow region binary image mask as input.

[0019] The specific details of the generated network structure are as follows:

[0020] First layer: First, the image is mirrored by 3 pixels, then followed by a convolutional layer with 64 7x7 kernels, and then InstanceNorm and ReLU instantiation are performed;

[0021] Preferably, the input of the first layer is implemented using the torch.nnSequential() function; the specific process is as follows:

[0022] self.conv1_b=nn.Sequential(nn.ReflectionPad2d(3),nn.Conv2d(3,64,7),nn.InstanceNorm2d(64),nn.ReLU(inplace=True))

[0023] Where 3 represents the number of input channels and 64 represents the number of output channels.

[0024] The second layer consists of a downsampling layer with 256 3x3 convolutional kernels and a stride of 2, followed by an InstanceNorm layer and a LeakyReLU layer.

[0025] Preferably, the function call for the second layer is as follows:

[0026] self.downconv2_b=nn.Sequential(nn.Conv2d(64,128,3,stride=2,padding=1),nn.InstanceNorm2d(128),nn.ReLU(inplace=True))

[0027] self.downconv3_b=nn.Sequential(nn.Conv2d(128,256,3,stride=2,padding=1),nn.InstanceNorm2d(256),nn.ReLU(inplace=True))

[0028] Where 64 is the number of input channels, 128 is the number of output channels, stride represents the step size, and padding represents the padding size.

[0029] The third layer consists of 256 residual convolutional layers with 3×3 kernels each. One of these residual convolutional layers comprises two regular convolutional kernels of size 3×3, followed by an InstanceNorm instantiation layer. This process of repeating the residual convolutional layer with the same number of kernels nine times extracts sufficiently detailed information.

[0030] Preferably, the function call for the third layer is as follows:

[0031] self.conv4_b=nn.Sequential(ResidualBlock(256))

[0032] Preferably, the ResidualBlock function is represented as follows:

[0033] conv_block=[nn.ReflectionPad2d(1),nn.Conv2d(in_features,in_features,3),nn.InstanceNorm2d(in_features),nn.ReLU(inplace=True),nn.ReflectionPad2d(1),nn.Conv2d(in_features,in_features,3),nn.InstanceNorm2d(in_features)] where ReflectionPad2d means mirror padding by 1 pixel, and in_features represents the input channels.

[0034] x = x + self.conv_block(x)

[0035] The output of ResidualBlock is in the form of x + self.conv_block(x), and is used as input features to the next layer of ResidualBlock for further feature extraction.

[0036] The fourth layer consists of 64 deconvolutional layers with 3x3 kernels and a stride of 2, followed by InstanceNorm and ReLU layers.

[0037] Preferably, the function call for the fourth layer is as follows:

[0038] self.upconv13_b=nn.Sequential(nn.ConvTranspose2d(256,128,3,stride=2,padding=1,output_padding=1),nn.InstanceNorm2d(128),

[0039] 256 represents the number of input feature channels after 9 residual convolutions, 128 represents the number of output feature channels, 3 represents the kernel size, 2 represents the kernel stride, padding represents the edge padding coefficient, and finally InstanceNorm2d represents the regularization method.

[0040] The fifth layer is an output layer with three 7x7 convolutional kernels and a stride of 1. The specific code is as follows.

[0041] self.conv15_b=nn.Sequential(nn.ReflectionPad2d(3),nn.Conv2d(64,3,7))

[0042] Sixth layer: The processed features are restored to the same size as the input image using bicubic interpolation. The specific code is as follows:

[0043] x=F.interpolate(x, size=xin.size()[2:], mode='bilinear', align_corners=True)

[0044] Where x represents the input feature, xin.size is the size of the input feature, mode represents the interpolation method, and align_corners represents the alignment method of the interpolation. When set to True, the values ​​of each boundary point of the original convolution kernel pixel are preserved, and then the interpolation operation is performed based on the value of each boundary point.

[0045] Please refer to Figure 2 , Figure 2 This is a schematic diagram of the training process of each network in this invention.

[0046] The training process for the generative network is as follows:

[0047] S201: Obtain the original remote sensing image real_A, the shadow mask map corresponding to the shadow area of ​​the original remote sensing image mask, the pure shadowless image real_B, and the pure shadow image real_C; based on the histogram of real_B, use the histogram matching method to process real_C into a pseudo shadowless image real_D;

[0048] S202: Input real_A into the shadow remover net_Gf to perform simple shadow removal and obtain the initial removal result fake_A1; input fake_A1 into the shadow generator net_Gs to perform shadow generation and obtain the initial generation result fake_A2; constrain the training of shadow remover net_Gf and shadow generator net_Gs according to the characteristic that real_A and fake_A2 should be the same or have consistency.

[0049] S203: Compare fake_A1 and mask into the shadow optimizer net_Gr with real_B and real_D respectively, and optimize the tone and texture to obtain the final image fake_A after removing the shadow; constrain the training of shadow optimizer net_Gr according to the characteristic that fake_A and real_B should be the same or consistent.

[0050] The discrimination network consists of 5 layers, all of which use a combination of 4×4 PartialConv convolution and LeakyReLU to extract and discriminate information. Finally, the extracted information is averaged to complete the discrimination.

[0051] The first layer input of the discrimination network is: the image region to be discriminated and the image after channel merging of the shadow mask of the region to be discriminated;

[0052] S102: Alternately train the generator network and the discriminator network: use the generator network to generate fake images to deceive the discriminator network, and update the parameters of the generator network and the discriminator network to finally obtain the high-resolution remote sensing image shadow removal network model.

[0053] It should be noted that the loss functions for the shadow generator net_Gs and the shadow remover net_Gf during training are as follows:

[0054] L GAN =L Cycle ×10+L Dis (1)

[0055] L = L Iden (Gs)×5+L Iden (Gf)×5+L GAN (2)

[0056] In equation (1), LCycle It is the L1 Loss of the pixel values ​​of the shadowless image fake_A1 generated by the shadow remover and the original pure shadowless image real_B; L Dis It is the MSEloss of the discrimination network for identifying the generated shadowless image fake_A1; in equation (2), LI den (Gs) is the L1 loss of the two pixels when the pure shadow image real_C is input into a shadow generator with the same structure as the shadow canceller; LI den (Gf) is the L1 loss of the pixels of the fake non-shadowed image fake_A1 generated by inputting the non-shadowed image real_B into the shadow remover; the parameters of the discrimination network remain unchanged when calculating the loss of the generation network and updating the parameters; the final fused L is the objective function of the initial shadow removal part.

[0057] The loss function during the training of the shadow optimizer net_Gr is as follows:

[0058]

[0059] L ref =0.7×L his -(0.15)×L var +L feature +1.5×L rec (4)

[0060] In equation (3), L var It is the variance loss function, w is the weight factor for each pixel; σ is the standard deviation of its input calculated using a Gaussian blur kernel with a kernel size of 15, G is the Gaussian blur kernel, and I... nsdc For real_D, I sdc For fake_B; in equation (4), L his It is the L1 loss of the histogram of the original shadowed area after removing the shadow and the histogram of the real non-shadowed area in the same image. feature It is the L1 loss of the pixel values ​​of the two feature images after inputting the image with the shadow removed and the image with the true shadow into the VGG16 network before activation. rec L1 loss is the edge similarity of two feature images after processing with the Laplacian operator; L ref This is the final overall loss function.

[0061] As an example, the present invention provides some source code, which is briefly excerpted below:

[0062] loss_hist=(1 / np.sqrt(2.0))*(torch.sqrt(torch.sum(torch.pow(torch.sqrt(input_hist)-torch.sqrt(output_hist),2)))) / input_hist.shape[0]

[0063] loss_var=-1*(1.5 / 10)*torch.sum(torch.abs(input_hist-output_hist))*torch.mean(torch.abs(torch.std(torch.std(input_gauss,dim=2),dim=2)-torch.std(torch.std(generated_gauss,dim=2),dim=2)))

[0064] Wherein, input_hist represents the histogram of the original shadowed region after removing the shadow, output_hist represents the histogram of the true non-shadowed region in the same image, input_guass represents the Gaussian distribution of the original shadowed region image after removing the shadow, generated_guass represents the Gaussian distribution of the true non-shadowed region in the same image, loss_hist represents the loss between input_hist and output_hist, and loss_var represents the var loss between the shadowed region image after removing the shadow and the true non-shadowed region in the same image.

[0065] The preferred code for feature loss is as follows:

[0066] loss_feat=criterion_vgg(vgg,fake_A1,real_D)

[0067] rec_loss=1.5*rec_loss_func.compute_loss(fake_A1,real_D)

[0068] Where vgg represents the network model used, fake_A1 represents the initial result obtained from shadow removal, and real_D represents the pseudo-shadowless image processed by histogram matching method from real_C. Loss_feat represents the feature loss Lfeature, and rec_loss represents the reconstruction loss Lrec.

[0069] The loss function for training the discriminant network net_D is calculated as follows:

[0070] LDis(real)=||D(sd,sd_mask)-1|| (5)

[0071] LDis(fake)=||D(Nsd,Nsd_mask)-0|| (6)

[0072] LDis=(LDis(real)+LDis(fake))×0.5 (7)

[0073] In Equation (5), sd represents the real shadow image and sd_mask is the shadow region mask. The two images are input into the discrimination network D for feature extraction, and the distance between them and 1 is judged, thereby effectively updating the accuracy of the discrimination network. In Equation (6), Nsd represents the real shadowless image and Nsd_mask represents the shadowless region mask. The distance between the extracted features and 0 is judged to measure the authenticity of the generated shadowless image. In Equation (7), the final discrimination network loss function uses the weighted sum of two loss functions to determine the gradient update direction of the discrimination network.

[0074] Specifically, this application uses PartialConv convolution instead of the original convolutional location feature discriminator to adapt to image conditions with irregular shadows. The definition of PartialConv convolution is shown in Equation (8). PartialConv convolution also uses the same linear function combination as standard convolution, consisting of weights W and biases b. X is the pixel value in the currently defined convolution kernel, and M is the corresponding original shadow mask. According to the shadow mask, only its marked position is convolved during the convolution process, avoiding interference to other regions. In addition, as shown in Equation (9), in order to ensure that the feature map after each convolution is not affected by other shadowless regions, the shadow mask m of the current state is also updated after each forward propagation convolution.

[0075]

[0076]

[0077] In the discriminator, shadow region features are extracted using four 4×4 PartialConv convolutions with a span of 2×2, yielding preprocessed shadow features. Then, a 4×4 PartialConv convolution compresses the extracted shadow features into a single channel to extract the most important features. Finally, the compressed values ​​are averaged to further verify the realism of the generated image. Each PartialConv convolution employs an InstanceNorm regularization layer and a LeakyReLU activation function. To ensure the accuracy of the discriminator, at the end of each iteration, the discriminator is updated using the MSELoss loss function based on the real shadow region and the shadowless image. The specific code is as follows:

[0078] self.conv1=PartialConv(input,ouput,kernel_size=kw, stride=2, padding=padw)

[0079] self.norm=norm_layer(output)

[0080] self.relu = nn.LeakyReLU(0.2)

[0081] Where PartialConv represents Partial convolution, input represents input features, output represents output features, kernel_size represents kernel size, stride represents kernel stride, padding represents edge padding coefficient, norm_layer represents InstanceNorm regularization layer, and LeakyReLU represents LeakyReLU activation function.

[0082] S103: Use the trained high-resolution remote sensing image shadow removal network model to remove shadows and obtain the shadow-removed remote sensing image.

[0083] Please refer to Figure 3 , Figure 3 This is a rendering of the invention. Figure (3)a is a remote sensing image with shadows. Figure 3 (b) is the remote sensing image after shadow removal. From Figure 3 As can be seen, the method of the present invention can effectively remove shadows in remote sensing images and restore the texture details of ground features in shadowed areas.

[0084] The beneficial effects of this invention are as follows: The network proposed in this invention can output more realistic shadowless images without requiring a truly matched shadowless image. Simultaneously, the discriminator can quantitatively determine the authenticity of a specified area based on the region covered by the mask, thereby more accurately updating the discriminator's precision and improving the generator's generation quality. This network overcomes the shortcomings of traditional shadow removal methods, such as complex steps and tonal distortion in the generated images, ultimately producing shadowless images with sharpness and texture features closer to real high-resolution remote sensing images.

[0085] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A deep learning based shadow removal method for high resolution remote sensing images, characterized in that: The method comprises the following steps: S101, construct a high-resolution remote sensing image shadow removal network model; the remote sensing image shadow removal network model is based on a GAN architecture, including a generation network and a discrimination network net_D ; A generative network generates fake images for the spoof-discriminating network; the generative network comprises a shadow remover net_Gf , a shadow generator net_Gs , and a shadow optimizer net_Gr ​ S102: alternately training the generation network and the discrimination network: generating false images for cheating the discrimination network by using the generation network, and updating the parameters of the generation network and the discrimination network, finally obtaining a high-resolution remote sensing image shadow removal network model; S103: completing shadow removal by using the high-resolution remote sensing image shadow removal network model trained, and obtaining a remote sensing image after shadow removal; The training process of the generation network is as follows: S201: Acquire raw remote sensing images real_A Shadow mask map corresponding to the shadow area of ​​the original remote sensing image mask Pure image without shadows real_B Pure shadow image real_C ;by real_B Based on the histogram, the histogram matching method is used to... real_C Processed into a pseudo-shadowless image real_D ; S202: Will real_A Input to shadow remover net_Gf A simple shadow removal process is performed to obtain the initial elimination result. fake_A1 ;Will fake_A1 Input to shadow generator net_Gs The initial generation result is obtained by performing shadow generation. fake_ A2 ;according to real_A and fake_A2 The training of the shadow remover should be constrained by the same or consistent characteristics. net_Gf and shadow generator net_Gs ; S203: Will fake_A1 With mask Input shadow optimizer net_Gr In the real_B And real_D respectively, carry on the comparison, carry on the optimization of the color tone, the texture, obtain the last image after removing the shadow fake_A ; According to fake_A And real_B The characteristics of the same or consistent should be constrained to train the shadow optimizer net_Gr .

2. The shadow removal method based on deep learning of high-resolution remote sensing images according to claim 1, characterized in that: The shadow eliminator net_Gf The shadow generator net_Gs And the shadow optimizer net_Gr All adopt 64 convolution kernels with size 7 x 7, 256 3 x 3 down-sampling layers, 256 residual convolution layers with size 3 x 3, 64 up-sampling layers with size 3 x 3, 3 output layers with size 7 x 7, InstanceNorm regularization layers and ReLU activation functions.

3. The shadow removal method based on deep learning of high-resolution remote sensing images according to claim 2, characterized in that: Shadow remover net_Gs , shadow generator net_Gf With 3-channel RGB image input, net_Gr With 4-channel image as input, which is synthesized from 3-channel RGB image and 1-channel shadow region binary mask.

4. The shadow removal method based on deep learning of high-resolution remote sensing images according to claim 1, characterized in that: The discrimination network net_D It includes 5 layers, all of which use PartialConv convolution and LeakyReLU in combination to extract and discriminate information, and finally the extracted information is averaged to complete the discrimination.

5. The shadow removal method based on deep learning of high-resolution remote sensing images according to claim 1, characterized in that: The training process of the discrimination network is as follows: S301: Will fake_A Nearest neighbor interpolation was used to obtain the image that was the same as the original remote sensing image. real_A Image with shadow removal of the same size fake_B ; S302: input to the discrimination network, according to fake_B the authenticity constraint of itself to train the discrimination network. fake_B the authenticity constraint of itself to train the discrimination network.

6. The shadow removal method based on deep learning of high-resolution remote sensing images according to claim 1, characterized in that: Shadow generator net_Gs and shadow remover net_Gf The loss function during training is as follows: In formula (1), L Cycle is the shadowless image generated by the shadow remover fake_A1 and the original pure shadowless image real_B pixel value of both images L1Loss ; L Dis is the shadowless image generated by the discriminator network fake_A1 discriminated MSEloss ; in formula (2), is the pure shadow image real_C input into the shadow generator with the same structure as the shadow remover to generate the false shadow image fake_A2 pixel of both L1Loss ; is the non-shadow image real_B input into the shadow remover to generate the false non-shadow image fake_A1 pixel of both after L1Loss; The generation network loss calculation and parameter update are performed while keeping the parameters of the discriminator network unchanged; and the final fusion is performed L is a target function for the initial shadow removal part.

7. The shadow removal method based on deep learning of high-resolution remote sensing images according to claim 1, characterized in that: Shadow optimizer net_Gr The loss function during training is as follows: In formula (3), is a variance loss function, is a weight factor for each pixel; by using a Gaussian blur kernel with a kernel of 15 to calculate the standard deviation of its input, is a Gaussian blur kernel, is real_D , is fake_B ; in formula (4), is the ratio of the histogram of the original shadow area after shadow removal to the histogram of the true non-shadow area in the same image L1loss , is the L1 loss of the pixel values of the two feature images after inputting the shadow-removed image and the real shadow image into the VGG16 network before activation, is the similarity of the edges of the two feature images after Laplacian operator processing L1loss ; is the overall loss function finally obtained.

8. The shadow removal method based on deep learning of high-resolution remote sensing images according to claim 5, characterized in that: Discrimination network net_D The loss function in the training process is calculated as follows: In formula (5), sd represents a real shadow image, sd_mask is a shadow area mask, two images are input into a discrimination network D to extract features, and the distance between the features and 1 is judged to effectively update the accuracy of the discrimination network; in formula (6), Nsd represents a real non-shadow image, Nsd_mask represents a non-shadow area mask image, the distance between the extracted features and 0 is judged to measure the authenticity of the generated non-shadow image; in formula (7), the final discrimination network loss function adopts a weighted sum of two loss functions to determine the gradient update direction of the discrimination network.

Images