Satellite image shadow removal method based on cycle generative adversarial network

Abstract

This invention discloses a satellite image shadow removal method based on a recurrent generative adversarial network (RGAN). The generator performs both shadow removal and shadow generation, while the discriminator determines whether the generated image belongs to a shadow-containing or shadowless image domain. The generator and discriminator work together to facilitate the shadow removal task. The invention first analyzes the shadow brightness and texture features in shadow-containing and shadowless images of natural scenes. High-resolution satellite images with reduced resolution are used as shadowless satellite images, and shadow-containing satellite images with simulated shadows are constructed based on this. Then, the RGAN learns from paired shadow-containing and shadowless satellite images, thereby generalizing the solution to the shadow removal problem in real satellite images. This invention primarily focuses on simulating datasets of unpaired shadow-containing and shadowless satellite images. It proposes a RGAN to assist shadow removal in the shadow generation task, achieving image transfer between shadow-containing and shadowless satellite image domains. A non-traditional encoder module with a visual self-attention model assists the generator in better feature transfer. Compared to traditional modules, it swaps the multi-head self-attention mechanism and layer normalization, and swaps the multilayer perceptron and layer normalization, giving the network stronger modeling capabilities.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and more specifically to a method for removing shadows from satellite images based on deep learning. Background Technology

[0002] In recent years, with the rapid development of aerospace remote sensing satellite technology, high-resolution satellites have provided high-quality meter-level and sub-meter-level remote sensing images to meet the research and application needs in various fields such as urban road network extraction, navigation map generation, and surface vegetation monitoring. However, in satellite remote sensing images, especially urban satellite images, ground objects often block sunlight due to their own height, creating localized shadow areas. These shadows reduce local illumination in the image, severely affecting the visual appeal and object recognition. Therefore, how to remove shadows from satellite images is a crucial problem that urgently needs to be solved to improve the utilization rate of satellite images.

[0003] Existing research on shadow removal in natural scenes first detects shadow regions across the entire image, then uses shadow-presence / shadow-free data to model shadow removal, and finally restores illumination in shadowed areas. Physically model-based shadow detection and removal methods require prior knowledge of sensor or camera parameters, solar elevation angle, etc.; shadow feature-based detection methods primarily obtain shadow location information and restore illumination intensity by comparing information differences between shadowed and non-shadowed areas, but these methods rely excessively on the shadow region segmentation performance of a single image. Since satellite images containing buildings often produce shadows, it is impossible to obtain shadow-free satellite images as ground truth annotations. Therefore, automatically detecting and removing shadows using deep learning models is particularly challenging. Summary of the Invention

[0004] To address the problems in related technologies, this invention proposes a satellite image shadow removal method based on a recurrent generative adversarial network, which can solve the aforementioned technical problems.

[0005] The technical solution of this invention is implemented as follows:

[0006] A satellite image shadow removal method based on recurrent generative adversarial networks, characterized by comprising:

[0007] (1) A cyclic generative adversarial network that implements the task of shadow removal in satellite images from two perspectives: shadow removal and shadow generation, as shown in the appendix. Figure 1 The generator G_A2B is used to migrate from a shaded region to a shaded region to achieve the purpose of shadow removal, and the generator G_B2A is used to migrate from a shaded region to a shaded region to achieve the purpose of shadow generation.

[0008] (2) As part of feature (1), the present invention creates a set of paired satellite image domain datasets with / without shadows, using high-resolution satellite images with reduced resolution as shadowless satellite images to provide a shadowless satellite image domain. For each shadowless image, a satellite image with shadows is created by random shadowing based on the shadow features and shadow texture information of the paired natural scene shadow dataset to provide a shadowed satellite image domain.

[0009] (3) As a component of feature (1), as shown in the appendix Figure 2 A generator based on a non-traditional encoder module of a visual self-attention model can extract image features from one image domain and fit the image distribution of another image domain. In the invention illustration 1, generator G_A2B implements the shadow removal task, and generator G_B2A in the invention illustration 1 implements the shadow generation task.

[0010] (4) As a component of feature (1), as shown in the appendix Figure 3 A discriminator based on a domain classifier, in Figure 1 of the invention, discriminator D_A determines whether the input image belongs to the shadowed satellite image domain, and in Figure 1 of the invention, discriminator D_B determines whether the input image belongs to the unshadowed satellite image domain;

[0011] (5) As a component of feature (1), as shown in the appendix Figure 4 Co-training between two generators is achieved through the sum of multiple loss functions, including generative adversarial loss based on the generator and discriminator. Based on the cycle consistency loss between the generated image and the original image produced by the two generators. The inherent loss is based on the generated image and the original image produced by this image domain generator. Boundary smoothing loss based on shaded and unshaded images Shadow removal loss based on shadow-removed image and shadow-free image .

[0012] It should be noted that the satellite image domain dataset with / without shadows described in feature (2) first uses high-resolution images with reduced resolution as shadowless satellite images. Then, based on a paired shadow dataset of natural scenes, the Lab space is selected for shadow feature and shadow texture information analysis. Finally, shadows of different sizes and shapes are randomly generated on the shadowless satellite images, thus realizing the creation of the satellite image domain dataset with / without shadows, which includes 2000 training sets and 1450 test sets. This method effectively solves the problem of satellite images not having paired shadowless datasets. Providing shadow images with good generalization from the perspective of generation is more conducive to the network learning more information about shadows.

[0013] Secondly, in the recurrent generative adversarial network, the task of shadow generation helps the shadow removal task to be performed better, so it contains two generators and two discriminators to form the network. The generator mentioned in feature (3) contains a non-traditional encoder module based on a visual self-attention model. Compared with the traditional module, this module swaps the multi-head self-attention layer and layer normalization, multilayer perceptron and layer normalization, which can make the network have stronger modeling ability and allow the features of the shadowed image to be better transferred to the features of the shadowless image, thereby realizing the shadow removal task. The discriminator module mentioned in feature (4) is a Markov discriminator containing a domain classifier. The addition of the domain classifier provides a self-attention map, which can make the discriminator better determine whether the input image belongs to the image domain identified by the discriminator. The loss function part mentioned in feature (5) contains The formal expression is as follows: This allows the generator that removes shadows to remove shadows more accurately, the generator that generates shadows to generate shadows more accurately, the discriminator with a shadowed region to determine whether the input image is a shadowed image more accurately, and the discriminator without a shadowed region to determine whether the image is a shadowless image more accurately. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a schematic diagram of the architecture of a satellite image shadow removal network based on a recurrent generative adversarial network according to an embodiment of the present invention;

[0016] Figure 2 This is a schematic diagram of a generator module for cyclically generating adversarial networks according to an embodiment of the present invention.

[0017] Figure 3 This is a schematic diagram of the discriminator module of the cyclic generative adversarial network according to an embodiment of the present invention;

[0018] Figure 4 This is a schematic diagram of the loss function of a cyclic generative adversarial network according to an embodiment of the present invention. Detailed Implementation

[0019] To achieve the above objectives, the technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0020] To create a dataset of satellite images with and without shadows, this invention describes a shadow generation method based on shadow analysis. By fitting the shadow distribution and studying the texture of natural scenes, the method is transferred to satellite images to generate shadow data. First, shadow correlation analysis was performed on each component of different color spaces. The Lab space, most suitable for shadow detection and removal, was selected, and a shadow coefficient q based on the L component was proposed, as shown in formula (1):

[0021] (1)

[0022] Where i represents the pixel value at a certain point in the image. This represents the pixel value at a point in a shadowless image of a natural scene. This represents the pixel value of a point in an image that has shadows in a natural scene.

[0023] The advantage of this shadow coefficient is that, when the L-component pixel values ​​of the non-shadowed image are known, the pixel values ​​when shadowed can be easily calculated, simplifying the shadow generation process. Next, through the study of shadow coefficient histograms, this invention discovers that shadowed / non-shadowed regions can be divided in the L-space using threshold segmentation. Through distribution fitting studies of shadowed regions and statistical analysis of the a and b components of the Lab space, a set of variation ranges of shadowed regions compared to non-shadowed images were obtained. For the texture information of shadowed regions, this invention selects different thresholds to extract shadow masks, and analyzes the mask changes between thresholds to obtain inferred shadow change features to match distribution patterns. Finally, this invention uses high-resolution satellite images with reduced resolution as shadow-free satellite images, realizing a set of shadow-free satellite image domains. For each shadow-free image, shadowed satellite images are created by randomly generating shadows based on the shadow features and shadow texture information of a paired natural scene shadow dataset, realizing a set of shadowed satellite image domains.

[0024] Having a paired satellite imagery shadow dataset, this invention trains the simulated dataset using a recurrent generative adversarial network (GAN), the network structure diagram of which is attached. Figure 1 To achieve good generalization on real datasets, the network is trained as follows:

[0025] S1. Initialize hyperparameters such as weights and learning rate;

[0026] S2. Build the model, including iterators for the input shaded and unshaded maps, two generators (G_A2B, G_B2A) and two discriminators (D_A, D_B) for the main body of the model, define the loss function, optimizer, etc., and initialize them.

[0027] S3. Training the model includes the following steps—

[0028] 1) Enable all generators and discriminators to begin training;

[0029] 2) If there is a pre-trained model, read the model first and adjust the learning rate;

[0030] 3) Repeat training steps 4-7 until the set number of iterations is reached;

[0031] 4) Take a set of images from the iterator: image a with shadow and image b without shadow;

[0032] 5) Update all discriminators, set the discriminator gradient to zero, generator G_A2B transforms a into b', generator G_B2A transforms b into a', and use the original image a and the generated images a' and b' as inputs to discriminators D_A and D_B respectively to calculate the loss function, perform gradient backpropagation and parameter update;

[0033] 6) Update the generator, set the generator gradient to zero, input images from different domains into the generator according to mapping methods such as cycle consistency loss and self-loss, and then calculate each loss function through the discriminator. Add supervision loss based on removing shadow images and loss based on boundary smoothing. All losses are shown in Figure 4 of the invention. Perform gradient backpropagation and parameter update.

[0034] 7) Determine the number of training iterations and store the parameters after a certain number of steps.

[0035] In the generator section, as shown in the appendix Figure 2 This model incorporates a non-traditional encoder module based on a Vision Transformer (VTM) between the downsampling and upsampling operations of the generator network. This module first flattens the feature maps obtained from downsampling using a flatten layer, adds positional encoding, and then connects 12 non-traditional encoder modules through fully connected layers. Compared to traditional modules, this non-traditional encoder module swaps the multi-head VTM and layer normalization layers, and swaps the multilayer perceptron and layer normalization layers, before performing residual summation. The results are then restored to the dimension of the input Vision Transformer module through fully connected and reshape layers. Finally, upsampling is performed, and residuals are added to the input image to obtain an image from another domain. This part allows the generator to better perform image feature transfer.

[0036] In the discriminator section, as shown in the appendix Figure 3This invention adds a generator module containing a domain classifier and shadow attention based module after downsampling the generative network. This module first uses Global Average Pooling (GAP) to obtain feature map weights from the downsampled feature maps, then multiplies the feature maps with the weights to obtain shadow attention maps. Finally, a fully convolutional network (FCN) classifier outputs the result of a PatchGAN Markov discriminator. The shadow attention maps output the corresponding image domain of the input image through fully connected layers, determining whether it belongs to a shadowed or unshadowed image.

[0037] The calculation of the loss function is the key to this network, as shown in the appendix. Figure 4 It contains three pairs of symmetric loss functions and two separate loss functions.

[0038] The three pairs of loss functions are the generative adversarial loss (GAN), the cycle consistency loss (cycle), and the identity loss (identity). Each pair of loss functions can be calculated using the following formula (2-4):

[0039] (2) (3) (4)

[0040] Of the two separate loss functions, there is a supervised loss based on removing shadows from the image; here, the L1 loss (also known as Mean Absolute Error, MAE) is used. Because the L1 loss is robust and provides supervision to better restore the image to its original state, it can also make the image more blurry. Its loss function is shown in formula (5):

[0041] (5)

[0042] Two separate loss functions are used, one of which is based on boundary smoothing. First, the shaded image and the generated unshaded image are subtracted pixel-by-pixel, and the RGB average is calculated. A threshold of 0.01 is used to divide the region. Pixels with an RGB average difference less than this threshold are set to 0, indicating that these points belong to the unshaded area. Then, the soft shadow mask is obtained using formula (6), where... It is the RGB mean tensor of pixel differences. This is the final obtained shadow mask tensor. Finally, the soft shadow mask and the generated shadowless map are plotted horizontally and vertically. The gradient is calculated using the window, and the loss is calculated as follows (7):

[0043] (6) (7)

[0044] in, Indicates the horizontal direction. Indicates the vertical direction. It is to find the gradient.

[0045] The final sum of losses is given by the following formula (8):

[0046] (8)

[0047] In summary, leveraging the aforementioned technical solutions, the satellite image shadow removal method proposed in this invention utilizes a main architecture consisting of two generators, two discriminators, and connections with different loss functions to implement shadow generation and removal tasks within the same network. Furthermore, it innovatively proposes a design and fabrication method for satellite images with / without shadows, enabling a single deep learning method to remove shadows from realistic satellite images. In the generator design, this invention incorporates a non-traditional encoder module connected to a visual self-attention model, allowing this component to better generate feature information that fits another image domain.

[0048] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A satellite image shadow removal method based on a recurrent generative adversarial network (GAN) is used to address the problem in high-resolution satellite remote sensing images where ground objects occlude sunlight, creating local shadows that severely affect visual perception and object recognition. The method is characterized by... Includes the following steps: S1: Create a set of paired satellite image domain datasets with / without shadows, obtain high-resolution satellite images, generate corresponding shadowless satellite images b by reducing the resolution, forming shadowless satellite image domain B. For each shadowless image, based on the shadows extracted from the natural scene shadow dataset, use statistical methods in the Lab space to analyze and summarize the shadow features, and use the analysis results to randomly generate shadowed satellite images a, forming shadowed satellite image domain A. S2: Construct a recurrent generative adversarial network (GAN) to perform the task of removing shadows from satellite images from two perspectives: shadow removal and shadow generation. The generator includes a visual self-attention model, and its multi-head self-attention layer is placed before the layer normalization layer. The discriminator includes a domain classifier to distinguish whether the image comes from a shadowed or shadowless satellite image domain. The network is trained based on the shadowed and shadowless satellite image domains. The generator G_A2B performs the shadow removal task, the generator G_B2A performs the shadow generation task, the discriminator D_A determines whether the input image belongs to the shadowed satellite image domain, and the discriminator D_B determines whether the input image belongs to the shadowless satellite image domain. S3: The loss function used in training includes generative adversarial loss based on the generator and discriminator. Based on the cycle consistency loss between the generated image and the original image produced by the two generators. The inherent loss is based on the generated image and the original image produced by this image domain generator. Boundary smoothing loss based on shaded and unshaded images Shadow removal loss based on shadow-removed image and shadow-free image Multiple loss mechanisms work together to constrain the training of the model.

2. The satellite image shadow removal method according to claim 1, characterized in that, The satellite image domain dataset with / without shadows created in step S1 first uses high-resolution images with reduced resolution as shadow-free satellite images. Then, based on the paired shadow dataset of natural scenes, the Lab space is selected for shadow feature and shadow texture information analysis. A shadow coefficient histogram is constructed in the L space of the Lab space, and the distribution fitting is studied by threshold segmentation to summarize shadow change characteristics. Statistical analysis is performed on the a and b spaces in the Lab space. Then, the analysis results are used to randomly generate shadows of different sizes and shapes on the shadow-free satellite images, thus realizing the creation of the satellite image domain dataset with / without shadows, which contains 2000 training sets and 1450 test sets.

3. The satellite image shadow removal method according to claim 1, characterized in that, The generator described in step S2 inserts 12 non-traditional encoder modules after the fully connected layer in the visual self-attention model. Compared with traditional modules, this module swaps the multi-head self-attention layer and layer normalization, multilayer perceptron and layer normalization, so that the network has stronger modeling ability and can better transfer the features of the shadowed image to the features of the shadowless image, thereby achieving the shadow removal task.

4. The satellite image shadow removal method according to claim 1, characterized in that, The discriminator mentioned in step S2 is a Markov discriminator containing a domain classifier. The addition of the domain classifier provides a self-attention map, which enables the discriminator to better determine whether the input image belongs to the image domain discriminator classifies.

5. The satellite image shadow removal method according to claim 1, characterized in that, The loss function part mentioned in step S3 includes shadow removal loss. It is a supervised loss based on removing shadows from the image, and its formulaic expression is: 'a' represents a shaded satellite image, and 'b' represents a unshaded satellite image. The image is obtained after generating G_A2B, with boundary smoothing loss. It is a loss function based on soft shadow masks, and its formulaic expression is: ,in It is the shadow mask tensor. It is the RGB mean tensor of pixel differences. Indicates the horizontal direction. Indicates the vertical direction. These are gradients, which enable generators that remove shadows to remove them more accurately, generators that generate shadows to generate them more accurately, discriminators with shadowed regions to determine whether an input image is a shadowed image more accurately, and discriminators without shadowed regions to determine whether an image is a shadowless image more accurately.

Images