Method and device for shadow removal from remote sensing images based on detail enhancement and edge reconstruction

Abstract

The application discloses a shadow removal method and device for remote sensing images based on detail enhancement and edge reconstruction. The method comprises the following steps: obtaining a remote sensing image training set, performing shadow detection on the shadow image to obtain an original shadow mask, performing morphological operation on the original shadow mask to obtain a divided shadow mask and a half-shadow-free image; inputting the divided shadow mask and the half-shadow-free image into a remote sensing image removal model to obtain a shadow removal result, constructing a hybrid loss function based on the shadow removal result and a real shadow-free image, and training the remote sensing image removal model based on the hybrid loss function; obtaining a to-be-detected remote sensing image, inputting the to-be-detected remote sensing image into the trained remote sensing image removal model after performing shadow detection on the to-be-detected remote sensing image, and obtaining a shadow-removed remote sensing image. The application can improve the quality of shadow removal of remote sensing images, has obvious advantages in overall image color recovery quality, shadow area micro detail structure recovery and edge pseudo mark processing.

Description

Technical Field

[0001] This application relates to the field of remote sensing image processing, and in particular to a method, apparatus, storage medium, and electronic device for shadow removal of remote sensing images based on detail enhancement and edge reconstruction. Background Technology

[0002] Shadows are very common in high spatial resolution remote sensing imagery. They can cause local spectral distortion and information loss, significantly reducing the accuracy of downstream tasks such as urban target detection and land cover classification. Therefore, performing shadow correction on remote sensing imagery is an essential preprocessing step that can effectively improve the usability and information content of remote sensing data.

[0003] Traditional remote sensing shadow removal methods can be mainly categorized into two types: local matching models and global optimization models. Local matching models treat individual shadows as the processing object, enhancing information by searching for matching illuminated regions. This method is computationally simple but prone to post-correction color distortion and edge artifacts in complex scenes. Global models, on the other hand, constrain the surface and illumination components separately through physical priors, constructing an energy functional and using the globally optimal solution of the surface component as the final correction result. These methods can effectively address the heterogeneity and mixing of shadows, but they still suffer from limitations such as difficulty in determining regularization parameters and insufficient computational efficiency when processing large amounts of remote sensing data.

[0004] In recent years, deep learning-based shadow removal methods have emerged in the field of remote sensing image processing. These methods learn complex nonlinear mapping relationships in data through efficient and automated feature extraction, thereby achieving high-precision shadow removal. However, existing deep learning remote sensing shadow removal models are usually directly transferred from shadow removal tasks in natural images, and generally do not consider prior knowledge of remote sensing illumination or imaging mechanisms. When processing remote sensing images of complex urban scenes, they suffer from common and significant problems such as blurred details within shadows and residual shadows or highlight artifacts at shadow edges. Summary of the Invention

[0005] This application provides a method, apparatus, storage medium, and electronic device for shadow removal in remote sensing images based on detail enhancement and edge reconstruction. It can improve the quality of shadow removal in remote sensing images and has significant advantages in overall image color restoration quality, restoration of minute details in shadow areas, and edge artifact processing.

[0006] This application provides a method for shadow removal in remote sensing images based on detail enhancement and edge reconstruction, including:

[0007] Obtain a training set of remote sensing images; wherein, the training set of remote sensing images includes shadowed images and real shadowless images of the same area corresponding to the shadowed images;

[0008] The image with shadows is subjected to shadow detection to obtain an original shadow mask. Morphological operations are then performed on the original shadow mask to obtain a shadow mask that distinguishes between the umbra and penumbra regions, and a penumbra-free image that includes both the umbra and the non-shadowed regions.

[0009] The segmented shadow mask and the penumbra-free image are input into the remote sensing image removal model to obtain the shadow removal result. A hybrid loss function is constructed based on the shadow removal result and the real shadow-free image. The remote sensing image removal model is trained based on the hybrid loss function.

[0010] The remote sensing image to be detected is acquired, and after shadow detection is performed on the remote sensing image to be detected, it is input into the trained remote sensing image removal model to obtain the shadow-removed remote sensing image.

[0011] Furthermore, in the aforementioned remote sensing image shadow removal method based on detail enhancement and edge reconstruction, the remote sensing image removal model includes an umbra region detail enhancement model, a generator, and a discriminator; the step of inputting the segmented shadow mask and the penumbra-free image into the remote sensing image removal model to obtain the shadow removal result includes:

[0012] The segmented shadow mask and the non-penumbra map are input into the umbra region detail enhancement model to obtain the detail enhancement map;

[0013] The shadowless image, the detail-enhanced image, and the segmented shadow mask are input into the generator to obtain the shadow removal result;

[0014] The shadow removal result and the real shadow-free image are input into the discriminator to distinguish between real and fake images. Based on the discrimination result, the generator is assisted in outputting a high-fidelity shadow removal result.

[0015] Furthermore, the aforementioned method for shadow removal in remote sensing images based on detail enhancement and edge reconstruction includes performing morphological operations on the original shadow mask to obtain a shadow mask that distinguishes between the umbra and penumbra regions, and a penumbra-free image containing both the umbra and non-shadow regions, including:

[0016] The original shadow mask is eroded by several pixels to obtain the local shadow region;

[0017] After performing a dilation operation of several pixels on the original shadow mask, the difference between the original shadow mask and the umbra region is calculated, and the difference is used as the penumbra region.

[0018] By assigning different category values ​​to the umbra region and the penumbra region, a shadow mask that distinguishes between the umbra region and the penumbra region is obtained.

[0019] A penumbra-free image is obtained based on the penumbra region and the shadowed image.

[0020] Furthermore, in the aforementioned method for shadow removal from remotely sensed images based on detail enhancement and edge reconstruction, the step of inputting the segmented shadow mask into the umbra region detail enhancement model to obtain a detail enhancement map includes:

[0021] The divided shadow mask is dilated by several pixels to obtain a shadowless region adjacent to the shadow region.

[0022] Based on the umbra region and the shadowless region of the non-penumbra image, a normalized cumulative histogram is calculated band by band, and a gray value lookup table is constructed.

[0023] Based on the grayscale lookup table, the grayscale values ​​of the umbra region of the image without penumbra are adjusted, and the adjusted image without penumbra is converted into a single-channel grayscale image to obtain a detail-enhanced image.

[0024] Furthermore, in the above-mentioned method for shadow removal of remote sensing images based on detail enhancement and edge reconstruction, the generator includes a color encoder, a detail encoder, a shallow feature adaptive fusion module, a deep feature cross-fusion module, and a cascaded decoder.

[0025] The penumbra-free image, the detail enhancement image, and the segmented shadow mask are input into the generator to obtain the shadow removal result, including:

[0026] The penumbra-free image and the segmented shadow mask are input into the color encoder to extract multi-scale color features, resulting in a color feature map.

[0027] The enhanced detail map and the segmented shadow mask are input into the detail encoder to extract multi-scale detail features, resulting in a detail feature map.

[0028] The color feature map and the detail feature map are input into the shallow feature adaptive fusion module to obtain shallow fused features, and the color feature map and the detail feature map are input into the deep feature cross fusion module to obtain high-dimensional fused features;

[0029] The shallow fusion features and the high-dimensional fusion features are input into the cascaded decoder to obtain the shadow removal result.

[0030] Furthermore, in the above-mentioned method for shadow removal of remote sensing images based on detail enhancement and edge reconstruction, the color encoder includes multiple cascaded convolutional modules, the detail encoder includes multiple cascaded convolutional modules, and the number of shallow feature adaptive fusion modules is the same as the number of convolutional modules.

[0031] In the color encoder and detail encoder, the color feature map and detail feature map output by each convolutional module are input into the corresponding shallow feature adaptive fusion module for processing to obtain shallow fused features.

[0032] Furthermore, in the above-mentioned method for shadow removal of remote sensing images based on detail enhancement and edge reconstruction, the shallow feature adaptive fusion module includes a channel attention module, which is sequentially connected to a global pooling layer, a 1×1 convolutional layer, a ReLU activation function, a 1×1 convolutional layer, and a Sigmoid function.

[0033] In the shallow feature adaptive fusion module, channel attention weights are calculated on the input color feature map and detail feature map, and then weighted fusion is performed to obtain shallow fused features.

[0034] Furthermore, in the above-mentioned method for shadow removal of remote sensing images based on detail enhancement and edge reconstruction, the deep feature cross-fusion module includes an attention module, which includes a 1×1 convolutional layer, a ReLU activation function, a 1×1 convolutional layer, and a Sigmoid function.

[0035] In the deep feature cross-fusion module, attention weights are calculated for the input color feature map and detail feature map, and then fused with each other's original feature maps to obtain high-dimensional fused features.

[0036] Furthermore, in the above-mentioned remote sensing image shadow removal method based on detail enhancement and edge reconstruction, the hybrid loss function is:

[0037]

[0038] in , , and These are the set weight parameters. For a mixed loss function, For pixel reconstruction loss, For feature consistency loss, For color ratio loss, To generate adversarial losses.

[0039] This application also provides a remote sensing image shadow removal device based on detail enhancement and edge reconstruction, including:

[0040] An acquisition module is used to acquire a remote sensing image training set; wherein, the remote sensing image training set includes shadowed images and real shadowless images of the same area corresponding to the shadowed images;

[0041] The processing module is used to perform shadow detection on the image with shadows to obtain an original shadow mask, and to perform morphological operations on the original shadow mask to obtain a shadow mask that distinguishes between the umbra and penumbra regions and a penumbra-free image that includes the umbra and the non-shadow region.

[0042] The training module is used to input the segmented shadow mask and the penumbra-free image into the remote sensing image removal model to obtain the shadow removal result, construct a hybrid loss function based on the shadow removal result and the real shadow-free image, and train the remote sensing image removal model based on the hybrid loss function.

[0043] The shadow removal module is used to acquire the remote sensing image to be detected, perform shadow detection on the remote sensing image, and input it into the trained remote sensing image removal model to obtain the shadow-removed remote sensing image.

[0044] This application also provides a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute any of the above-described remote sensing image shadow removal methods based on detail enhancement and edge reconstruction.

[0045] This application also provides an electronic device, including a processor and a memory, wherein the processor is electrically connected to the memory, the memory is used to store instructions and data, and the processor is used in the steps of the remote sensing image shadow removal method based on detail enhancement and edge reconstruction described above.

[0046] This application provides a method, apparatus, storage medium, and electronic device for shadow removal from remote sensing images based on detail enhancement and edge reconstruction. This application performs morphological processing on remote sensing images to obtain a penumbra-free image with no shadow areas. The penumbra-free image is then applied to a remote sensing image removal model to obtain a shadow-removed image. This invention has the following technical advantages and beneficial effects:

[0047] ① This invention adopts a method that couples illumination enhancement statistics with generative adversarial learning mode, which simply and effectively solves the common problem of difficulty in restoring ground feature details under large areas of shadow occlusion, and effectively improves the usability of shadow removal methods for remote sensing images based on deep learning.

[0048] ② This invention processes the penumbra region with complex lighting conditions and the umbra with relatively uniform lighting separately. In the penumbra region, the powerful context-based reconstruction capability of generative adversarial networks is used to achieve accurate penumbra region filling, so that the shadow correction result transitions naturally at the edge, overcoming the edge artifacts and penumbra residue problems that are difficult to solve by existing methods.

[0049] ③ The modules based on shallow feature adaptive fusion and deep feature cross-fusion enable the network to adaptively fuse color information and detail information in different dimensions, thereby improving the accuracy of color restoration and detail clarity in remote sensing shadow removal.

[0050] ④ The hybrid loss function proposed in this invention includes pixel reconstruction loss, feature consistency loss, color ratio loss, and generative adversarial loss, which can guide model training and optimization from multiple aspects such as pixel, feature, and color channel relationship, thereby accelerating the convergence speed of the model.

[0051] ⑤ This invention does not require complex parameter adjustments or manual intervention, and is not limited to processing visible light images. It can also process multispectral images, including those in the near-infrared band. It has strong automation, generalization and accuracy, and is easy to put into practical use. Attached Figure Description

[0052] The technical solution and other beneficial effects of this application will become apparent from the following detailed description of specific embodiments in conjunction with the accompanying drawings.

[0053] Figure 1 A flowchart of a remote sensing image shadow removal method based on detail enhancement and edge reconstruction provided in an embodiment of this application.

[0054] Figure 2 This is a flowchart illustrating the generation of the partitioned shadow mask and the penumbra image provided in the embodiments of this application.

[0055] Figure 3 This is a schematic diagram of the structure of the remote sensing image removal model provided in the embodiments of this application.

[0056] Figure 4 This is a schematic diagram of the structure of the shallow feature adaptive fusion module and the deep feature cross-fusion module.

[0057] Figure 5 The image shows a comparison of the shadow removal effect before and after, as provided in the embodiments of this application.

[0058] Figure 6 The accompanying diagram shows a comparison between the proposed method and existing shadow removal methods, as illustrated in the embodiments of this application.

[0059] Figure 7 A schematic diagram of the structure of the remote sensing image shadow removal device based on detail enhancement and edge reconstruction provided in the embodiments of this application.

[0060] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0061] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0062] This application provides a method, apparatus, storage medium, and electronic device for shadow removal of remote sensing images based on detail enhancement and edge reconstruction. The shadow removal apparatus for remote sensing images based on detail enhancement and edge reconstruction provided in this application can be integrated into an electronic device, such as a terminal or server. The terminal can include a tablet computer, laptop computer, personal computer (PC), microprocessor box, or other devices.

[0063] Please see Figure 1 , Figure 1 The flowchart illustrates a method for shadow removal from remotely sensed images based on detail enhancement and edge reconstruction, provided in this application embodiment. This method, applied in an electronic device, includes the following steps:

[0064] S1, Obtain the remote sensing image training set; wherein, the remote sensing image training set includes shadowed images and real shadowless images of the same area corresponding to the shadowed images.

[0065] Specifically, the remote sensing image training set should include multiple pairs of high spatial resolution optical remote sensing images with shadows and their corresponding shadowless remote sensing images. Meanwhile, the images with shadows should contain shadows projected by multiple scenes and types of ground features, so as to improve the removal accuracy and generalization when training the remote sensing image shadow removal model.

[0066] S2, perform shadow detection on the image with shadows to obtain the original shadow mask, and perform morphological operations on the original shadow mask to obtain the shadow mask after dividing the umbra and penumbra regions and the penumbra-free image containing the umbra and the non-shadow region.

[0067] Figure 2 The flowchart for generating the partitioned shadow mask and penumbra image provided in the embodiments of this application is as follows: Figure 2 As shown, step S2 may include the following steps:

[0068] S21, perform an erosion operation on the original shadow mask for several pixels to obtain the local shadow region;

[0069] S22, after the original shadow mask is dilated by several pixels, the difference between it and the umbra region is calculated, and the difference is used as the penumbra region.

[0070] S23, assign different category values ​​to the umbra and penumbra regions to obtain a shadow mask that distinguishes between the umbra and penumbra regions;

[0071] S24, obtain a penumbra-free image based on the penumbra region and the shadowed image.

[0072] Specifically, the first step is to perform shadow detection on the image with shadows and generate a binary shadow mask (i.e., the original shadow mask) marked with shadow locations. Then, morphological processing is performed on the original shadow mask and the image with shadows: first, an erosion operation of φ pixels is performed on the original shadow mask to determine the umbra region (e.g., ...). Figure 2 The process involves using the white portion of the shadow mask to distinguish between the umbra and penumbra regions (as described in the text). Then, a dilation operation of φ pixels is performed on the original shadow mask to remove the portion overlapping with the umbra region. The remaining portion with a value can then be identified as the penumbra region (e.g., ...). Figure 2 The gray portion of the shadow mask (distinguishing between the umbra and penumbra regions as described above) is used to assign different category values ​​to the divided umbra and penumbra regions, thus obtaining the shadow mask after distinguishing between the umbra and penumbra regions. Finally, the determined penumbra regions are used to cover the original shadowed image to remove penumbra information interference, resulting in a penumbra-free image. In this embodiment, the value of φ is 2.

[0073] S3. Input the segmented shadow mask and penumbra-free image into the remote sensing image removal model to obtain the shadow removal result. Construct a hybrid loss function based on the shadow removal result and the real shadow-free image, and train the remote sensing image removal model based on the hybrid loss function.

[0074] Figure 3 This is a schematic diagram of the structure of the remote sensing image removal model provided in the embodiments of this application, as shown below. Figure 3 As shown, the remote sensing image shadow removal model innovatively couples an illumination statistical model with a generative adversarial learning mode. The remote sensing image shadow removal model includes an umbra detail enhancement model (UDE Model), a generator G, and a discriminator D. Step S3 specifically includes the following steps:

[0075] S31, input the segmented shadow mask and the non-penumbra map into the umbra region detail enhancement model to obtain the detail enhancement map.

[0076] In one embodiment, step S31 includes the following steps:

[0077] S311, Perform a dilation operation of several pixels on the divided shadow mask to obtain a shadowless region adjacent to the shadow region;

[0078] S312, based on the umbra region and the shadowless region of the non-penumbra image, calculates the normalized cumulative histogram band by band and constructs a gray value lookup table;

[0079] S313, adjust the grayscale value of the umbra region of the non-penumbra image based on the grayscale value lookup table, and convert the adjusted non-penumbra image into a single-channel grayscale image to obtain a detail-enhanced image.

[0080] Specifically, such as Figure 3 As shown, with no penumbra image I u And the shadow mask after dividing the umbra and penumbra regions M a As input, first, a shadow mask is created after dividing the umbra and penumbra regions. M a A dilation operation of Ψ pixels is performed on the entire image to obtain a shadowless region adjacent to the shadow region. In this embodiment, the value of Ψ is 15. This is based on a penumbra-free image. I u For the umbra region and the adjacent unshaded region, normalized cumulative histograms are calculated band by band, and a grayscale lookup table (LUT) is constructed. The grayscale values ​​of the umbra region are adjusted band by band using the constructed LUT, and the adjusted results are then converted into single-channel grayscale images to reduce potential color distortion problems and obtain a detail-enhanced image. D e .

[0081] S32 inputs the penumbra image, the detail enhancement image, and the segmented shadow mask into the generator to obtain the shadow removal result.

[0082] Specifically, the generator G includes a color encoder, a detail encoder, n shallow feature adaptive fusion modules, a deep feature cross-fusion module, and a cascaded decoder. Step S32 includes the following steps:

[0083] S321: The non-penumbra image and the segmented shadow mask are stitched together along the channel dimension and then input into the color encoder to extract multi-scale color features, resulting in a color feature map.

[0084] S322: After stitching the detail enhancement map and the segmented shadow mask together in the channel dimension, it is input into the detail encoder to extract multi-scale detail features and obtain a detail feature map.

[0085] In one embodiment, the color encoder includes multiple cascaded convolutional modules, the detail encoder includes multiple cascaded convolutional modules, and the number of shallow feature adaptive fusion modules is the same as the number of convolutional modules. In both the color encoder and the detail encoder, the color feature map and detail feature map output from each layer of convolutional modules are input to the corresponding layer's shallow feature adaptive fusion module for processing to obtain shallow fused features.

[0086] Specifically, the color encoder comprises n concatenated convolutional modules, each including a 4×4 convolutional layer, a batch normalization layer, and a LeakyReLU activation function for extracting multi-scale color features; no penumbra image. I u And the shadow mask after dividing the umbra and penumbra regions M a After concatenation along the channel dimension, the data is input into the color encoder to encode multi-channel color features individually. The feature map generated by the i-th convolutional module has a size of [size missing]. ,in i It is the hierarchical index of each convolutional module in the encoder (1≤ i ≤n), C It is the number of channels input to the first convolutional module. H The height of the input image. W n is the width of the input image; in this embodiment, n is 4. C It is 64.

[0087] The detail encoder maintains the same network structure as the color encoder, but its feature information flow remains independent. (Detail enhancement map) D e And the shadow mask after dividing the umbra and penumbra regions M a After being concatenated along the channel dimension, it is input into the detail encoder to encode multi-scale detail features separately.

[0088] S323: Input the color feature map and detail feature map into the shallow feature adaptive fusion module to obtain shallow fused features, and input the color feature map and detail feature map into the deep feature cross fusion module to obtain high-dimensional fused features.

[0089] Figure 4 This is a schematic diagram of the structure of the shallow feature adaptive fusion module and the deep feature cross-fusion module, as shown below. Figure 4 As shown, the shallow feature adaptive fusion module includes a channel attention module, which is sequentially connected to a global pooling layer, a 1×1 convolutional layer, a ReLU activation function, another 1×1 convolutional layer, and a Sigmoid function. In the shallow feature adaptive fusion module, channel attention weights are calculated on the input color feature map and detail feature map, and then weighted and fused to obtain the shallow fused features.

[0090] Specifically, the number of shallow feature adaptive fusion modules is the same as the number of convolutional modules, that is, there are n shallow feature adaptive fusion modules (AFF), corresponding to the n levels of convolutional modules in the color encoder and detail encoder (in this embodiment, n is 4). In each shallow feature adaptive fusion module, the color feature map and detail feature map of the same level are weighted and fused after calculating the channel attention weights through the channel attention module (including global pooling - 1×1 convolution - ReLU activation function - 1×1 convolution - Sigmoid function) to generate shallow fused features. These output shallow fused features are then fed into the corresponding level of the decoder module for subsequent feature decoding and reconstruction.

[0091] like Figure 4 As shown, the deep feature cross-fusion module includes an attention module, which comprises a 1×1 convolutional layer, a ReLU activation function, another 1×1 convolutional layer, and a Sigmoid function. In the deep feature cross-fusion module, attention weights are calculated for the input color feature map and detail feature map, and then fused with the corresponding original feature maps to obtain high-dimensional fused features.

[0092] Specifically, the deep feature cross-fusion module calculates attention weights for the color feature map output from the last convolutional module of the color encoder and the detail feature map from the last convolutional module of the detail encoder, respectively, through an attention module (including 1×1 convolution-ReLU activation function-1×1 convolution-Sigmoid function). Then, each attention weight is multiplied by the other's original feature map (meaning the processed detail feature map is multiplied by the input color feature map, and vice versa). The two sets of multiplied feature maps are concatenated along the channel dimension to generate high-dimensional fused features deep within the network, which are also used for subsequent feature decoding and reconstruction.

[0093] S324 inputs the shallow fusion features and high-dimensional fusion features into the cascaded decoder to obtain the shadow removal result.

[0094] Please continue reading. Figure 3The cascaded decoder uses multiple transposed convolutional blocks and upsampling blocks for feature decoding and reconstruction. More specifically, the cascaded decoder first refines the high-dimensional fused features output by the deep feature cross-fusion module using multiple convolutional modules. Then, the cascaded decoder uses the high-dimensional fused features through multiple transposed convolutional blocks to achieve penumbra region context reconstruction and feature map size restoration. Subsequently, it uses strided connections to cascade and fuse the shallow fused features output by the shallow feature adaptive fusion module, avoiding the loss of original details and color information during feature upsampling and downsampling operations. Each transposed convolutional block includes a transposed convolutional layer, a batch normalization layer, and a ReLU activation function. Finally, an upsampling layer and a convolutional layer are used to upsample the feature map to the same H×W size as the input image, and the Tanh function is used to normalize it to -1 to 1. Inverse normalization is then used to output the final shadow removal result.

[0095] S33, input the shadow removal result and the real shadowless image into the discriminator to distinguish between real and fake, and based on the discrimination result, assist the generator in outputting a high-fidelity shadow removal result.

[0096] Specifically, the discriminator D is designed as a PatchGAN network structure, which is a network composed of 5 convolutional modules. It is used to distinguish the shadow removal results output by the generator from the real shadowless image. During model training, it assists the generator G in outputting high-fidelity shadow removal results by calculating the generative adversarial loss.

[0097] Furthermore, the hybrid loss function is a weighted combination of pixel reconstruction loss feature consistency loss, color ratio loss, and generative adversarial loss:

[0098]

[0099] in , , and These are the set weight parameters. For a mixed loss function, For pixel reconstruction loss, For feature consistency loss, For color ratio loss, To generate adversarial losses.

[0100] Among them, pixel reconstruction loss Aiming to minimize the model output shadow removal results And real reference shadowless image The pixel-level difference between them. Pixel reconstruction loss. It can be represented as:

[0101]

[0102] in, Expressing expectations, This represents the L1 distance.

[0103] Feature consistency loss Remove the result by minimizing the output shadow. And real reference shadowless image The distance in the feature space, used to enhance the constraint on deep network features during model training, can be expressed as:

[0104]

[0105] in, It is the first generation of the pre-trained VGG19 network. i Feature maps generated by the layer.

[0106] Color ratio loss To mitigate color distortion issues that easily occur during shadow removal, it can be represented as:

[0107]

[0108] in, c Indicates the total number of channels. k Indicates the current number k A two-dimensional image with multiple channels. Color ratio loss is minimized by removing the output shadows. And real reference shadowless image The percentage difference in each channel value at the same pixel enhances the network's sensitivity to subtle color differences.

[0109] Generate adversarial loss The aim is to generate adversarial learning patterns and train the discriminator alternately. D and generator G This helps the generator produce visually higher quality harmonious shadow removal results, which can be represented as:

[0110]

[0111] When training the remote sensing shadow removal model, the network model parameters are continuously optimized through the stochastic gradient descent algorithm and backpropagation mechanism until the training ends when the mixed loss function of shadow removal results and shadowless image determination converges.

[0112] S4. Acquire the remote sensing image to be detected, perform shadow detection on the remote sensing image, and input it into the trained remote sensing image removal model to obtain the shadow-removed remote sensing image.

[0113] Specifically, shadow detection is performed on the remote sensing image to be detected to obtain an original shadow mask. Morphological operations are then performed on the original shadow mask to obtain a shadow mask that distinguishes between the umbra and penumbra regions, and a penumbra-free image that includes both the umbra and the non-shadowed regions. The segmented shadow mask and the penumbra-free image are then input into a trained remote sensing image removal model to obtain a shadow-free remote sensing image with clear details and natural edges. Figure 5 These are before-and-after comparison images of shadow removal provided in an embodiment of this application. Figure 6 The comparison diagrams provided in this application embodiment show the effect of this method compared with existing shadow removal methods. Figure 5 and Figure 6 It is evident that the method proposed in this invention has significant advantages in overall image color restoration quality, restoration of minute details in shadow areas, and edge artifact processing. Furthermore, it can process high-resolution visible light, multispectral, and other multi-source images, demonstrating strong versatility.

[0114] Compared with existing methods, the present invention has the following technical advantages and beneficial effects:

[0115] ① This invention adopts a method that couples illumination enhancement statistics with generative adversarial learning mode, which simply and effectively solves the common problem of difficulty in restoring ground feature details under large areas of shadow occlusion, and effectively improves the usability of shadow removal methods for remote sensing images based on deep learning.

[0116] ② This invention processes the penumbra region with complex lighting conditions and the umbra with relatively uniform lighting separately. In the penumbra region, the powerful context-based reconstruction capability of generative adversarial networks is used to achieve accurate penumbra region filling, so that the shadow correction result transitions naturally at the edge, overcoming the edge artifacts and penumbra residue problems that are difficult to solve by existing methods.

[0117] ③ The modules based on shallow feature adaptive fusion and deep feature cross-fusion enable the network to adaptively fuse color information and detail information in different dimensions, thereby improving the accuracy of color restoration and detail clarity in remote sensing shadow removal.

[0118] ④ The hybrid loss function proposed in this invention includes pixel reconstruction loss, feature consistency loss, color ratio loss, and generative adversarial loss, which can guide model training and optimization from multiple aspects such as pixel, feature, and color channel relationship, thereby accelerating the convergence speed of the model.

[0119] ⑤ This invention does not require complex parameter adjustments or manual intervention, and is not limited to processing visible light images. It can also process multispectral images, including those in the near-infrared band. It has strong automation, generalization and accuracy, and is easy to put into practical use.

[0120] Based on the method described in the above embodiments, this embodiment will further describe the remote sensing image shadow removal device based on detail enhancement and edge reconstruction. The remote sensing image shadow removal device based on detail enhancement and edge reconstruction can be implemented as an independent entity or integrated into an electronic device. The electronic device can be a terminal, server, or other device. The terminal can include a tablet computer, a laptop computer, a personal computer (PC), a microprocessor box, or other devices.

[0121] Please see Figure 7 , Figure 7 This application provides a detailed description of a remote sensing image shadow removal device based on detail enhancement and edge reconstruction, which is applied in an electronic device. The device may include:

[0122] An acquisition module is used to acquire a remote sensing image training set; wherein, the remote sensing image training set includes shadowed images and real shadowless images of the same area corresponding to the shadowed images;

[0123] The processing module is used to perform shadow detection on the image with shadows to obtain an original shadow mask, and to perform morphological operations on the original shadow mask to obtain a shadow mask that distinguishes between the umbra and penumbra regions and a penumbra-free image that includes the umbra and the non-shadow region.

[0124] The training module is used to input the segmented shadow mask and the penumbra-free image into the remote sensing image removal model to obtain the shadow removal result, construct a hybrid loss function based on the shadow removal result and the real shadow-free image, and train the remote sensing image removal model based on the hybrid loss function.

[0125] The shadow removal module is used to acquire the remote sensing image to be detected, perform shadow detection on the remote sensing image, and input it into the trained remote sensing image removal model to obtain the shadow-removed remote sensing image.

[0126] In specific implementation, the above modules and / or units can be implemented as independent entities, or they can be arbitrarily combined and implemented as the same or several entities. For the specific implementation of the above modules and / or units, please refer to the previous method embodiments. For the specific beneficial effects that can be achieved, please also refer to the beneficial effects in the previous method embodiments, which will not be repeated here.

[0127] In addition, this application also provides an electronic device, which may be a computer, tablet computer, or other similar device. This electronic device can implement the steps of any embodiment of the remote sensing image shadow removal method based on detail enhancement and edge reconstruction provided in this application. Therefore, it can achieve the beneficial effects achievable by any remote sensing image shadow removal method based on detail enhancement and edge reconstruction provided in this invention, as detailed in the preceding embodiments, and will not be repeated here.

[0128] Figure 8 A specific structural block diagram of an electronic device provided in an embodiment of the present invention is shown. This electronic device can be used to implement the remote sensing image shadow removal method based on detail enhancement and edge reconstruction provided in the above embodiments. The electronic device 500 can be a terminal, server, or other device. The terminal can include a tablet computer, laptop computer, personal computer (PC), microprocessor box, or other devices.

[0129] RF circuit 510 is used to receive and transmit electromagnetic waves, converting electromagnetic waves into electrical signals and vice versa, thereby enabling communication with communication networks or other devices. RF circuit 510 may include various existing circuit elements used to perform these functions, such as antennas, radio frequency transceivers, digital signal processors, encryption / decryption chips, subscriber identity modules (SIM cards), memory, etc. RF circuit 510 can communicate with various networks such as the Internet, corporate intranets, and wireless networks, or communicate with other devices via wireless networks. The aforementioned wireless networks may include cellular telephone networks, wireless local area networks (WLANs), or metropolitan area networks (MANs). The aforementioned wireless networks may use various communication standards, protocols, and technologies, including but not limited to Global System for Mobile Communication (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (WCDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Wireless Fidelity (Wi-Fi) (such as IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, and / or IEEE 802.11n), Voice over Internet Protocol (VoIP), Worldwide Interoperability for Microwave Access (Wi-Max), other protocols for email, instant messaging, and short messages, and any other suitable communication protocols, including those that have not yet been developed.

[0130] The memory 520 can be used to store software programs and modules, such as the program instructions / modules corresponding to those in the above embodiments. The processor 580 executes various functional applications and data processing by running the software programs and modules stored in the memory 520, such as taking pictures with the front-facing camera, processing the captured images, and switching the display colors of the content displayed on the screen. The memory 520 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 520 may further include memory remotely located relative to the processor 580, and these remote memories can be connected to the electronic device 500 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0131] The input unit 530 can be used to receive input numeric or character information, and to generate a keyboard and mouse related to user settings and function control.

[0132] Display unit 540 can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces, which can be composed of graphics, text, icons, video, and any combination thereof. Display unit 540 may include display panel 541, which may optionally be configured in the form of LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), or other similar forms.

[0133] Audio circuitry 560, speaker 561, and microphone 562 provide an audio interface between the user and electronic device 500. Audio circuitry 560 converts received audio data into electrical signals and transmits them to speaker 561, where speaker 561 converts them into sound signals for output. Conversely, microphone 562 converts collected sound signals into electrical signals, which are then received by audio circuitry 560, converted back into audio data, and processed by processor 580. The audio data is then transmitted via RF circuitry 510 to, for example, another terminal, or output to memory 520 for further processing. Audio circuitry 560 may also include an earphone jack to facilitate communication between external headphones and electronic device 500.

[0134] Electronic device 500, through transmission module 570 (e.g., Wi-Fi module), can help users receive requests, send information, etc., providing users with wireless broadband internet access. Although transmission module 570 is shown in the figure, it is understood that it is not an essential component of electronic device 500 and can be omitted as needed without changing the essence of the invention.

[0135] The processor 580 is the control center of the electronic device 500. It connects to various parts of the phone via various interfaces and lines, and performs various functions and processes data of the electronic device 500 by running or executing software programs and / or modules stored in the memory 520, and by calling data stored in the memory 520, thereby providing overall monitoring of the electronic device. Optionally, the processor 580 may include one or more processing cores; in some embodiments, the processor 580 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into the processor 580.

[0136] Electronic device 500 also includes a power supply 590 (such as a battery) that supplies power to various components. In some embodiments, the power supply may be logically connected to processor 580 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 590 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0137] Although not shown, the electronic device 500 also includes cameras (such as front-facing cameras and rear-facing cameras), Bluetooth modules, etc., which will not be described in detail here. Specifically, in this embodiment, the display unit of the electronic device is a touch screen display, and the mobile terminal also includes a memory and one or more programs, wherein one or more programs are stored in the memory and configured to be executed by one or more processors. One or more programs contain instructions for performing the following operations:

[0138] Obtain a training set of remote sensing images; wherein, the training set of remote sensing images includes shadowed images and real shadowless images of the same area corresponding to the shadowed images;

[0139] The image with shadows is subjected to shadow detection to obtain an original shadow mask. Morphological operations are then performed on the original shadow mask to obtain a shadow mask that distinguishes between the umbra and penumbra regions, and a penumbra-free image that includes both the umbra and the non-shadowed regions.

[0140] The segmented shadow mask and the penumbra-free image are input into the remote sensing image removal model to obtain the shadow removal result. A hybrid loss function is constructed based on the shadow removal result and the real shadow-free image. The remote sensing image removal model is trained based on the hybrid loss function.

[0141] The remote sensing image to be detected is acquired, and after shadow detection is performed on the remote sensing image to be detected, it is input into the trained remote sensing image removal model to obtain the shadow-removed remote sensing image.

[0142] In practice, the above modules can be implemented as independent entities or combined in any way to be implemented as the same or several entities. For the specific implementation of the above modules, please refer to the previous method implementation examples, which will not be repeated here.

[0143] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor. Therefore, embodiments of the present invention provide a storage medium storing a plurality of instructions that can be loaded by a processor to execute the steps of any embodiment of the remote sensing image shadow removal method based on detail enhancement and edge reconstruction provided by the present invention.

[0144] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0145] Since the instructions stored in the storage medium can execute the steps in any embodiment of the remote sensing image shadow removal method based on detail enhancement and edge reconstruction provided in the embodiments of the present invention, the beneficial effects that any remote sensing image shadow removal method based on detail enhancement and edge reconstruction provided in the embodiments of the present invention can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0146] The foregoing has provided a detailed description of a method, apparatus, storage medium, and electronic device for shadow removal of remote sensing images based on detail enhancement and edge reconstruction, as provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for shadow removal in remote sensing images based on detail enhancement and edge reconstruction, characterized in that, The method includes: Obtain a training set of remote sensing images; wherein, the training set of remote sensing images includes shadowed images and real shadowless images of the same area corresponding to the shadowed images; The image with shadows is subjected to shadow detection to obtain an original shadow mask. Morphological operations are performed on the original shadow mask to obtain a shadow mask that distinguishes between the umbra and penumbra regions and a penumbra-free image that includes the umbra and the non-shadow region. The segmented shadow mask and the penumbra-free image are input into the remote sensing image removal model to obtain the shadow removal result. A hybrid loss function is constructed based on the shadow removal result and the real shadow-free image. The remote sensing image removal model is then trained based on the hybrid loss function. The remote sensing image removal model includes an umbra region detail enhancement model, a generator, and a discriminator. The processing steps of the remote sensing image removal model include: The segmented shadow mask and the penumbra-free image are input into the umbra region detail enhancement model to obtain a detail enhancement image; the penumbra-free image, the detail enhancement image, and the segmented shadow mask are input into the generator to obtain a shadow removal result; the shadow removal result and the real shadow-free image are input into the discriminator to distinguish between real and fake images, and the discriminator assists the generator in outputting a high-fidelity shadow removal result based on the discrimination result; The generator includes a color encoder, a detail encoder, a shallow feature adaptive fusion module, a deep feature cross-fusion module, and a cascaded decoder. The penumbra-free image, the detail-enhanced image, and the segmented shadow mask are input into the generator to obtain shadow removal results, including: The penumbra-free image and the segmented shadow mask are input into the color encoder to extract multi-scale color features, resulting in a color feature map. The detail enhancement image and the segmented shadow mask are input into the detail encoder to extract multi-scale detail features, resulting in a detail feature map. The color feature map and the detail feature map are input into the shallow feature adaptive fusion module to obtain shallow fusion features. The color feature map and the detail feature map are input into the deep feature cross-fusion module to obtain high-dimensional fusion features. The shallow fusion features and the high-dimensional fusion features are input into the cascaded decoder to obtain the shadow removal result. The remote sensing image to be detected is acquired, and after shadow detection is performed on the remote sensing image to be detected, it is input into the trained remote sensing image removal model to obtain the shadow-removed remote sensing image.

2. The method for shadow removal in remote sensing images based on detail enhancement and edge reconstruction according to claim 1, characterized in that, Morphological operations are performed on the original shadow mask to obtain a shadow mask that distinguishes between the umbra and penumbra regions, and a penumbra-free image that includes both the umbra and the non-shadowed regions, including: The original shadow mask is eroded by several pixels to obtain the local shadow region; After performing a dilation operation of several pixels on the original shadow mask, the difference between the original shadow mask and the umbra region is calculated, and the difference is used as the penumbra region. By assigning different category values ​​to the umbra region and the penumbra region, a shadow mask that distinguishes between the umbra region and the penumbra region is obtained. A penumbra-free image is obtained based on the penumbra region and the shadowed image.

3. The method for shadow removal in remote sensing images based on detail enhancement and edge reconstruction according to claim 1, characterized in that, The step of inputting the segmented shadow mask into the umbra region detail enhancement model to obtain a detail enhancement map includes: The divided shadow mask is dilated by several pixels to obtain a shadowless region adjacent to the shadow region. Based on the umbra region and the shadowless region of the non-penumbra image, a normalized cumulative histogram is calculated band by band, and a gray value lookup table is constructed. Based on the grayscale lookup table, the grayscale values ​​of the umbra region of the image without penumbra are adjusted, and the adjusted image without penumbra is converted into a single-channel grayscale image to obtain a detail-enhanced image.

4. The method for shadow removal in remote sensing images based on detail enhancement and edge reconstruction according to claim 1, characterized in that, The color encoder includes multiple cascaded convolutional modules, the detail encoder includes multiple cascaded convolutional modules, and the number of shallow feature adaptive fusion modules is the same as the number of convolutional modules. In the color encoder and detail encoder, the color feature map and detail feature map output by each convolutional module are input into the corresponding shallow feature adaptive fusion module for processing to obtain shallow fused features.

5. The method for shadow removal in remote sensing images based on detail enhancement and edge reconstruction according to claim 4, characterized in that, The shallow feature adaptive fusion module includes a channel attention module, which is sequentially connected to a global pooling layer, a 1×1 convolutional layer, a ReLU activation function, a 1×1 convolutional layer, and a Sigmoid function; In the shallow feature adaptive fusion module, channel attention weights are calculated on the input color feature map and detail feature map, and then weighted fusion is performed to obtain shallow fused features.

6. The method for shadow removal in remote sensing images based on detail enhancement and edge reconstruction according to claim 5, characterized in that, The deep feature cross-fusion module includes an attention module, which includes a 1×1 convolutional layer, a ReLU activation function, a 1×1 convolutional layer, and a Sigmoid function. In the deep feature cross-fusion module, attention weights are calculated for the input color feature map and detail feature map, and then fused with each other's original feature maps to obtain high-dimensional fused features.

7. The method for shadow removal in remote sensing images based on detail enhancement and edge reconstruction according to claim 1, characterized in that, The hybrid loss function is: in , , and These are the set weight parameters. For a mixed loss function, For pixel reconstruction loss, For feature consistency loss, For color ratio loss, To generate adversarial losses.

8. A remote sensing image shadow removal device based on detail enhancement and edge reconstruction, wherein the remote sensing image shadow removal device based on detail enhancement and edge reconstruction is used to implement the remote sensing image shadow removal method based on detail enhancement and edge reconstruction as described in claim 1, characterized in that, include: An acquisition module is used to acquire a remote sensing image training set; wherein, the remote sensing image training set includes shadowed images and real shadowless images of the same area corresponding to the shadowed images; The processing module is used to perform shadow detection on the image with shadows to obtain an original shadow mask, and to perform morphological operations on the original shadow mask to obtain a shadow mask that distinguishes between the umbra and penumbra regions and a penumbra-free image that includes the umbra and the non-shadow region. The training module is used to input the segmented shadow mask and the penumbra-free image into the remote sensing image removal model to obtain the shadow removal result, construct a hybrid loss function based on the shadow removal result and the real shadow-free image, and train the remote sensing image removal model based on the hybrid loss function. The shadow removal module is used to acquire the remote sensing image to be detected, perform shadow detection on the remote sensing image, and input it into the trained remote sensing image removal model to obtain the shadow-removed remote sensing image.

Images