Remote sensing image shadow removal method and device based on detail enhancement and edge reconstruction

Abstract

The invention discloses a remote sensing image shadow removal method and device based on detail enhancement and edge reconstruction. The method comprises the following steps: acquiring a remote sensing image training set, performing shadow detection on a shadow image to obtain an original shadow mask, and performing morphological operation on the original shadow mask to obtain a divided shadow mask and a penumbra-free image; inputting the divided shadow mask and penumbra-free image into a remote sensing image removal model to obtain a shadow removal result, constructing a mixed loss function based on the shadow removal result and the real shadow-free image, and training the remote sensing image removal model based on the mixed loss function; and obtaining a to-be-detected remote sensing image, performing shadow detection on the to-be-detected remote sensing image, and inputting the to-be-detected remote sensing image into the trained remote sensing image removal model to obtain a shadow-removed remote sensing image. The remote sensing image shadow removal quality can be improved, and the method has obvious advantages in overall image color recovery quality, shadow area tiny detail structure recovery and edge artifact processing.

Description

Technical Field

[0001] This application relates to the field of remote sensing image processing, and particularly to a method, device, storage medium, and electronic device for removing shadows from remote sensing images based on detail enhancement and edge reconstruction. Background Art

[0002] Shadows are very common in high-spatial-resolution remote sensing images, which can cause local spectral distortion and information loss in remote sensing images, thereby significantly reducing the accuracy of downstream tasks such as urban target detection and land cover classification. Therefore, performing shadow correction on remote sensing images is an extremely necessary preprocessing step for remote sensing data, which can effectively improve the usability and effective information content of remote sensing data.

[0003] Traditional remote sensing shadow removal methods can be mainly classified into two categories: local matching models and global optimization models. Among them, the local matching model takes a single shadow as the processing object and enhances information by searching for a matching illumination area. This method is computationally simple but is extremely prone to color distortion and edge artifacts after correction in complex scenes. The global model constrains the surface and illumination components respectively through physical priors to construct an energy functional, and takes the global optimal solution of the surface component as the final correction result. This type of method can better solve the problems of inhomogeneity and mixing of shadows, but there are still limitations in determining regularization parameters and insufficient computational efficiency when processing large remote sensing data.

[0004] In recent years, shadow removal methods based on deep learning have begun to appear in the field of remote sensing image processing. These methods learn complex non-linear mapping relationships in the data through efficient and automated feature extraction methods to achieve high-precision shadow removal. However, existing deep learning remote sensing shadow removal models are usually directly migrated from natural image shadow removal tasks and generally do not consider remote sensing illumination priors or imaging mechanisms. When actually processing remote sensing images of complex urban scenes, there are common and significant problems such as blurred internal details of shadows and residual shadow edges or bright artifacts. Summary of the Invention

[0005] Embodiments of this application provide a method, device, storage medium, and electronic device for removing shadows from remote sensing images based on detail enhancement and edge reconstruction, which can improve the quality of remote sensing image shadow removal and have obvious advantages in the overall image color restoration quality, restoration of tiny detail structures in shadow areas, and edge artifact processing.

[0006] Embodiments of this application provide a method for removing shadows from remote sensing images based on detail enhancement and edge reconstruction, including: Obtain a remote sensing image training set; wherein, the remote sensing image training set includes shadow images and true shadowless images of the same area corresponding to the shadow images; Shadow detection is performed on the shaded image to obtain an original shadow mask. Morphological operations are performed on the original shadow mask to obtain a divided shadow mask that distinguishes the umbra region and the penumbra region, and a no-penumbra image that includes the umbra region and the shadow-free region; The divided shadow mask and the no-penumbra image are input into a remote sensing image removal model to obtain a shadow removal result. A hybrid loss function is constructed based on the shadow removal result and the true shadow-free image, and the remote sensing image removal model is trained based on the hybrid loss function; A remote sensing image to be detected is obtained. After shadow detection is performed on the remote sensing image to be detected, it is input into the trained remote sensing image removal model to obtain a shadow-removed remote sensing image.

[0007] Furthermore, in the above remote sensing image shadow removal method based on detail enhancement and edge reconstruction, the remote sensing image removal model includes an umbra region detail enhancement model, a generator, and a discriminator; the process of inputting the divided shadow mask and the no-penumbra image into the remote sensing image removal model to obtain a shadow removal result includes: The divided shadow mask and the no-penumbra image are input into the umbra region detail enhancement model to obtain a detail-enhanced image; The detail-enhanced image is input into the generator to obtain a shadow removal result; The shadow removal result and the true shadow-free image are input into the discriminator to determine authenticity, and the discriminator result is used to assist the generator to output a high-fidelity shadow removal result.

[0008] Furthermore, in the above remote sensing image shadow removal method based on detail enhancement and edge reconstruction, the process of performing morphological operations on the original shadow mask to obtain a divided shadow mask that distinguishes the umbra region and the penumbra region, and a no-penumbra image that includes the umbra region and the shadow-free region includes: Erosion operation on the original shadow mask by a certain number of pixels to obtain the umbra region; After dilation operation on the original shadow mask by a certain number of pixels, the difference is taken with the umbra region, and the difference value is used as the penumbra region; Different class values are assigned to the umbra region and the penumbra region to obtain a divided shadow mask that distinguishes the umbra region and the penumbra region; A no-penumbra image is obtained based on the penumbra region and the shaded image.

[0009] Furthermore, in the above remote sensing image shadow removal method based on detail enhancement and edge reconstruction, the process of inputting the divided shadow mask into the umbra region detail enhancement model to obtain a detail-enhanced image includes: Perform dilation operations on the divided shadow mask by several pixels to obtain a shadow-free area adjacent to the shadow area; Based on the umbra region of the penumbra-free image and the shadow-free area, calculate the normalized cumulative histogram band by band and construct a gray value lookup table; Based on the gray value lookup table, adjust the gray value of the umbra region of the penumbra-free image, and convert the adjusted penumbra-free image into a single-channel grayscale image to obtain a detail-enhanced image.

[0010] Furthermore, for the above remote sensing image shadow removal method based on detail enhancement and edge reconstruction, the generator includes a color encoder, a detail encoder, a shallow feature adaptive fusion module, a deep feature cross fusion module, and a cascaded decoder; Inputting the detail-enhanced image into the generator to obtain a shadow removal result includes: Input the detail-enhanced image into the color encoder to extract multi-scale color features and obtain a color feature map; Input the detail-enhanced image into the detail encoder to extract multi-scale detail features and obtain a detail feature map; Input the color feature map and the detail feature map into the shallow feature adaptive fusion module to obtain a shallow fusion feature, and input the color feature map and the detail feature map into the deep feature cross fusion module to obtain a high-dimensional fusion feature; Input the shallow fusion feature and the high-dimensional fusion feature into the cascaded decoder to obtain a shadow removal result.

[0011] Furthermore, for the above remote sensing image shadow removal method based on detail enhancement and edge reconstruction, the color encoder includes a plurality of cascaded convolution modules, the detail encoder includes a plurality of cascaded convolution modules, and the number of shallow feature adaptive fusion modules is the same as the number of convolution modules; In the color encoder and the detail encoder, input the color feature map and the detail feature map output by each layer of convolution module into the shallow feature adaptive fusion module at the corresponding level for processing to obtain a shallow fusion feature.

[0012] Furthermore, for the above remote sensing image shadow removal method based on detail enhancement and edge reconstruction, the shallow feature adaptive fusion module includes a channel attention module, and the channel attention module is sequentially connected with a global pooling layer, a 1×1 convolution layer, a ReLU activation function, a 1×1 convolution layer, and a Sigmoid function; In the shallow feature adaptive fusion module, calculate the channel attention weight for the input color feature map and detail feature map and then perform weighted fusion to obtain a shallow fusion feature.

[0013] Furthermore, in the above remote sensing image shadow removal method based on detail enhancement and edge reconstruction, the deep feature cross-fusion module includes an attention module, and the attention module includes a 1×1 convolutional layer, a ReLU activation function, a 1×1 convolutional layer, and a Sigmoid function; In the deep feature cross-fusion module, after calculating the attention weights for the input color feature map and detail feature map, they are respectively fused with the original feature maps of each other to obtain high-dimensional fusion features.

[0014] Furthermore, in the above remote sensing image shadow removal method based on detail enhancement and edge reconstruction, the hybrid loss function is:

[0015] where 、 、 and are set weight parameters, is the hybrid loss function, is the pixel reconstruction loss, is the feature consistency loss, is the color ratio loss, is the generative adversarial loss.

[0016] An embodiment of the present application also provides a remote sensing image shadow removal device based on detail enhancement and edge reconstruction, including: An acquisition module, configured to acquire a remote sensing image training set; wherein, the remote sensing image training set includes shadow images and real shadow-free images of the same area corresponding to the shadow images; A processing module, configured to perform shadow detection on the shadow image to obtain an original shadow mask, and perform morphological operations on the original shadow mask to obtain a divided shadow mask that distinguishes the umbra region and the penumbra region and a semi-shadow-free image that includes the umbra region and the shadow-free region; A training module, configured to input the divided shadow mask and the semi-shadow-free image into a remote sensing image removal model to obtain a shadow removal result, construct a hybrid loss function based on the shadow removal result and the real shadow-free image, and train the remote sensing image removal model based on the hybrid loss function; A shadow removal module, configured to acquire a remote sensing image to be detected, perform shadow detection on the remote sensing image, and input it into the trained remote sensing image removal model to obtain a shadow-removed remote sensing image.

[0017] The embodiments of the present application further provide a computer-readable storage medium, in which multiple instructions are stored, and the instructions are adapted to be loaded by a processor to execute any one of the above-mentioned remote sensing image shadow removal methods based on detail enhancement and edge reconstruction.

[0018] The embodiments of the present application further provide an electronic device, including a processor and a memory. The processor is electrically connected to the memory. The memory is used to store instructions and data, and the processor is used for the steps in any one of the above-mentioned remote sensing image shadow removal methods based on detail enhancement and edge reconstruction.

[0019] For the remote sensing image shadow removal method, device, storage medium and electronic device provided by the present application, the present application performs morphological processing on the remote sensing image to obtain a penumbra-free image of the shadow-free area, and inputs the penumbra-free image of the shadow-free area into the remote sensing image removal model to obtain a shadow-removed image. The present invention has the following technical advantages and beneficial effects: ① The present invention adopts a method of coupling the illumination enhancement statistical method with the generative adversarial learning mode, which simply and effectively solves the common problem that it is difficult to restore the details of ground objects under large-area shadow occlusion, and effectively improves the usability of the remote sensing image shadow removal method based on deep learning.

[0020] ② The present invention separately processes the penumbra area with complex lighting conditions and the umbra with relatively uniform lighting. In the penumbra area, the powerful context-based reconstruction ability of the generative adversarial network is utilized to achieve accurate retouching of the penumbra area, making the shadow correction result transition naturally at the edge, and overcoming the edge artifacts and penumbra residue problems that are difficult to solve by existing methods.

[0021] ③ Based on the module of adaptive fusion of shallow features and cross-fusion of deep features, the network can adaptively fuse color information and detail information in different dimensions, thereby improving the color recovery accuracy and detail clarity of remote sensing shadow removal.

[0022] ④ The hybrid loss function proposed by the present invention, including pixel reconstruction loss, feature consistency loss, color ratio loss and generative adversarial loss, can guide model training and optimization from multiple aspects such as pixels, features, and color channel relationships, thereby accelerating the convergence speed of the model.

[0023] ⑤ The present invention does not require complex parameter adjustment and manual intervention, and is not limited to processing visible light images. It can also process multi-spectral images including near-infrared bands, etc. It has strong automation, generalization and accuracy, and is easy to be put into practical use. Description of the Drawings

[0024] Combined with the accompanying drawings, through a detailed description of the specific embodiments of the present application, the technical solutions and other beneficial effects of the present application will become obvious.

[0025] Figure 1 It is a flowchart of a remote sensing image shadow removal method based on detail enhancement and edge reconstruction provided by an embodiment of the present application.

[0026] Figure 2 It is a flowchart of generating a divided shadow mask and a penumbra-free image provided by an embodiment of the present application.

[0027] Figure 3 It is a schematic structural diagram of a remote sensing image removal model provided by an embodiment of the present application.

[0028] Figure 4 It is a schematic structural diagram of a shallow feature adaptive fusion module and a deep feature cross fusion module.

[0029] Figure 5 It is a comparison effect diagram before and after shadow removal provided by an embodiment of the present application.

[0030] Figure 6 It is a comparison effect diagram of the present method and the existing shadow removal method provided by an embodiment of the present application.

[0031] Figure 7 It is a schematic structural diagram of a remote sensing image shadow removal device based on detail enhancement and edge reconstruction provided by an embodiment of the present application.

[0032] Figure 8 It is a schematic structural diagram of an electronic device provided by an embodiment of the present application. Specific Embodiments

[0033] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of protection of the present application.

[0034] An embodiment of the present application provides a remote sensing image shadow removal method, device, storage medium, and electronic device based on detail enhancement and edge reconstruction. A remote sensing image shadow removal device based on detail enhancement and edge reconstruction provided by an embodiment of the present application can be integrated in an electronic device, and the electronic device can be a device such as a terminal, a server, etc. Among them, the terminal can include a tablet computer, a notebook computer, a personal computer (PC), a microprocessing box, or other devices, etc.

[0035] Please refer toFigure 1 , Figure 1 A flowchart of a remote sensing image shadow removal method based on detail enhancement and edge reconstruction provided in an embodiment of the present application, which is applied to an electronic device, and the remote sensing image shadow removal method based on detail enhancement and edge reconstruction comprises the following steps: S1, obtaining a remote sensing image training set; wherein the remote sensing image training set includes shadow images and real shadow-free images of the same area corresponding to the shadow images.

[0036] Specifically, the remote sensing image training set should include multiple pairs of high spatial resolution optical remote sensing shadowed images and their corresponding shadow-free remote sensing images. At the same time, the shadowed images should contain shadows cast by multiple scenes and types of objects, so as to improve the removal accuracy and generalization when training the remote sensing image shadow removal model.

[0037] S2, performing shadow detection on the shadowed image to obtain an original shadow mask, performing morphological operations on the original shadow mask to obtain a shadow mask that distinguishes the umbra area and the penumbra area and a non-penumbra image that includes the umbra area and the non-shadow area.

[0038] Figure 2 The flowchart of generating the shadow mask and the non-penumbra image after division provided in the embodiment of the present application is as follows: Figure 2 As shown, step S2 may include the following steps: S21, performing an erosion operation on a number of pixels of the original shadow mask to obtain an umbra area; S22, after dilating the original shadow mask by several pixels, subtracting the original shadow mask from the umbra area, and taking the difference as the penumbra area; S23, assigning different category values ​​to the umbra area and the penumbra area to obtain a shadow mask after division that distinguishes the umbra area and the penumbra area; S24, obtaining a non-penumbra image based on the penumbra area and the shadowed image.

[0039] Specifically, the shadow detection is first performed on the shadowed image and a binary shadow mask (i.e., the original shadow mask) with the shadow position marked is generated; then the original shadow mask and the shadowed image are morphologically processed: the original shadow mask is first eroded by φ pixels to determine the umbra area (e.g., Figure 2 The original shadow mask is then expanded by φ pixels to remove the part that overlaps with the umbra area. The remaining valued part can be determined as the penumbra area (such as Figure 2(the gray part of the shadow mask that distinguishes the umbra and penumbra regions described in [reference]), by assigning different category values to the divided umbra region and penumbra region, the divided shadow mask that distinguishes the umbra and penumbra regions can be obtained; finally, the determined penumbra region is used to cover the original shadowed image to eliminate the interference of penumbra information and obtain an image without penumbra. In this embodiment, the value of φ is 2.

[0040] S3. Input the divided shadow mask and the image without penumbra into the remote sensing image removal model to obtain a shadow removal result. Based on the shadow removal result and the true shadowless image, construct a hybrid loss function, and train the remote sensing image removal model based on the hybrid loss function.

[0041] Figure 3 is a schematic structural diagram of the remote sensing image removal model provided by the embodiment of the present application. As Figure 3 shown, the remote sensing image shadow removal model innovatively couples a lighting statistical model with a generative adversarial learning mode. The remote sensing image shadow removal model includes a umbra region detail enhancement model (UDE Model), a generator G, and a discriminator D. Step S3 specifically includes the following steps: S31. Input the divided shadow mask and the image without penumbra into the umbra region detail enhancement model to obtain a detail enhanced image.

[0042] In one embodiment, step S31 includes the following steps: S311. Perform a dilation operation on the divided shadow mask by a certain number of pixels to obtain a shadowless region adjacent to the shadow region; S312. Based on the umbra region and the shadowless region of the image without penumbra, calculate the normalized cumulative histogram band by band and construct a gray value lookup table; S313. Adjust the gray value of the umbra region of the image without penumbra based on the gray value lookup table, and convert the adjusted image without penumbra into a single-channel grayscale image to obtain a detail enhanced image.

[0043] Specifically, as Figure 3 shown, taking the image without penumbra I u and the divided shadow mask that distinguishes the umbra and penumbra regions M a as inputs, first perform a dilation operation on the divided shadow mask that distinguishes the umbra and penumbra regions M a as a whole by Ψ pixels to obtain a shadowless region adjacent to the shadow region. In this embodiment, the value of Ψ is 15. Based on the image without penumbra I uFor the umbra region and the shadowless region adjacent to the shadow region, calculate the normalized cumulative histogram band by band, and construct a grayscale value lookup table (LUT). Use the constructed lookup table to adjust the grayscale value of the umbra region band by band, and then convert the adjusted result into a single-channel grayscale image to reduce possible color distortion problems and obtain a detail-enhanced image. D e 。

[0044] S32. Input the detail-enhanced image into the generator to obtain the shadow removal result.

[0045] Specifically, the generator G includes a color encoder, a detail encoder, n shallow feature adaptive fusion modules, a deep feature cross-fusion module, and a cascaded decoder. Step S32 includes the following steps: S321. Input the detail-enhanced image into the color encoder to extract multi-scale color features and obtain a color feature map.

[0046] S322. Input the detail-enhanced image into the detail encoder to extract multi-scale detail features and obtain a detail feature map.

[0047] In one embodiment, the color encoder includes multiple cascaded convolution modules, the detail encoder includes multiple cascaded convolution modules, and the number of shallow feature adaptive fusion modules is the same as the number of convolution modules. In the color encoder and the detail encoder, the color feature map and the detail feature map output by each layer of the convolution module are input into the corresponding-level shallow feature adaptive fusion module for processing to obtain shallow fusion features.

[0048] Specifically, the color encoder includes n cascaded convolution modules. Each convolution module includes a 4×4 convolutional layer, a batch normalization layer, and a LeakyReLU activation function for extracting multi-scale color features; the non-penumbra image I u and the divided shadow mask for distinguishing the umbra and penumbra regions M a are concatenated in the channel dimension and input into the color encoder to separately encode multi-channel color features. The feature map generated by the i-th convolution module has a size of , where i is the layer index of each convolution module in the encoder (1 ≤ i ≤ n), C is the number of channels input to the first convolution module, H is the height of the input image, W is the width of the input image; in this embodiment, n is 4, C is 64.

[0049] The detail encoder is the same as the color encoder in terms of network structure construction, but the feature information flow remains independent. Detail enhancement map D e and the shadow mask after dividing and distinguishing the umbra and penumbra regions M a are concatenated in the channel dimension and then input into the detail encoder to separately encode multi-scale detail features.

[0050] S323, the color feature map and the detail feature map are input into the shallow feature adaptive fusion module to obtain shallow fusion features, and the color feature map and the detail feature map are input into the deep feature cross-fusion module to obtain high-dimensional fusion features.

[0051] Figure 4 FIG. is a schematic diagram of the structures of the shallow feature adaptive fusion module and the deep feature cross-fusion module, as Figure 4 shown. The shallow feature adaptive fusion module includes a channel attention module, and the channel attention module is sequentially connected with a global pooling layer, a 1×1 convolutional layer, a ReLU activation function, a 1×1 convolutional layer, and a Sigmoid function. In the shallow feature adaptive fusion module, the channel attention weights are calculated for the input color feature map and detail feature map and then weighted fusion is performed to obtain shallow fusion features.

[0052] Specifically, the number of shallow feature adaptive fusion modules is the same as the number of convolutional modules, that is, there are n shallow feature adaptive fusion modules (AFF) in total, corresponding to the convolutional modules of n levels of the color encoder and the detail encoder (in this embodiment, n is 4). In each shallow feature adaptive fusion module, the color feature map and the detail feature map of the same level are respectively weighted and fused after calculating the channel attention weights through the channel attention module (including global pooling - 1×1 convolution - ReLU activation function - 1×1 convolution – Sigmoid function) to generate shallow fusion features. These output shallow fusion features are then fed into the decoder modules of the corresponding levels for subsequent feature decoding and reconstruction.

[0053] As Figure 4 shown, the deep feature cross-fusion module includes an attention module, and the attention module includes a 1×1 convolutional layer, a ReLU activation function, a 1×1 convolutional layer, and a Sigmoid function. In the deep feature cross-fusion module, the attention weights are calculated for the input color feature map and detail feature map and then fused with the corresponding original feature maps respectively to obtain high-dimensional fusion features.

[0054] Specifically, the deep feature cross-fusion module calculates the attention weights of the color feature map output by the last convolutional module of the color encoder and the detail feature map of the last convolutional module of the detail encoder through the attention module (including 1×1 convolution - ReLU activation function - 1×1 convolution – Sigmoid function), and multiplies the respective attention weights with the original feature maps of the other party (referring to multiplying the processed detail feature map with the input color feature map, and multiplying the processed color feature map with the input detail feature map). The two groups of multiplied feature maps are concatenated in the channel dimension to generate high-dimensional fusion features in the deep layer of the network, which are also used for subsequent feature decoding and reconstruction.

[0055] S324, input the shallow fusion feature and the high-dimensional fusion feature into the cascaded decoder to obtain the shadow removal result.

[0056] Please continue to refer to Figure 3 , the cascaded decoder uses multiple transposed convolution blocks and upsampling blocks for feature decoding and reconstruction. More specifically, the cascaded decoder first refines the high-dimensional fusion features output by the deep feature cross-fusion module using multiple convolutional modules. Thereafter, the cascaded decoder uses the high-dimensional fusion features through multiple transposed convolution blocks to achieve penumbra region context reconstruction and feature map size restoration, and then uses stride connections to cascade and fuse the shallow fusion features output by the shallow feature adaptive fusion module, avoiding the loss of original detail and color information in the operations of feature upsampling and downsampling. Among them, each transposed convolution block includes a transposed convolution layer, a batch normalization layer, and a ReLU activation function. Finally, use 1 upsampling layer and 1 convolution layer to upsample the feature map to the same H×W size as the input image, and use the Tanh function to normalize it to -1 to 1, and use the inverse normalization operation to output the final shadow removal result.

[0057] S33, input the shadow removal result and the real shadow-free image into the discriminator to distinguish true and false, and assist the generator to output a high-fidelity shadow removal result based on the discrimination result.

[0058] Specifically, the discriminator D is designed as a PatchGAN network structure, a network composed of 5 convolutional modules, which is used to distinguish the shadow removal result output by the generator from the real shadow-free image, and assist the generator G to output a high-fidelity shadow removal result by calculating the generative adversarial loss during model training.

[0059] Furthermore, the hybrid loss function is a weighted combination of pixel reconstruction loss, feature consistency loss, color ratio loss, and generative adversarial loss:

[0060] Among them , , and are the set weight parameters, is the hybrid loss function, is the pixel reconstruction loss, is the feature consistency loss, is the color ratio loss, is the generative adversarial loss.

[0061] Among them, the pixel reconstruction loss aims to minimize the pixel-level gap between the shadow removal result of the model output and the true reference shadow-free image. The pixel reconstruction loss can be expressed as:

[0062] where, represents the expectation, represents the L1 distance.

[0063] The feature consistency loss enhances the constraint on the deep features of the network during model training by minimizing the distance in the feature space between the output shadow removal result and the true reference shadow-free image and can be expressed as:

[0064] where, i is the feature map generated by the th layer of the pre-trained VGG19 network.

[0065] The color ratio loss is used to alleviate the color distortion problem that is likely to occur during the shadow removal process and can be expressed as:

[0066] c where, k represents the total number of channels, k represents the 2D image of the current th channel. The color ratio loss enhances the sensitivity of the network to subtle color differences by minimizing the percentage difference of each channel value at the same pixel between the output shadow removal result and the true reference shadow-free image

[0067] The generative adversarial loss aims to, through the generative adversarial learning mode, alternately train the discriminator D and the generator G ​​, to assist the generator in generating visually higher-quality and harmonious shadow removal results, which can be expressed as:

[0068] When training the remote sensing shadow removal model, the network model parameters are continuously optimized through the stochastic gradient descent algorithm and the backpropagation mechanism until the training ends when the mixed loss function of the shadow removal result and the shadow-free image is determined to converge.

[0069] S4. Obtain the remote sensing image to be detected, perform shadow detection on the remote sensing image, and input it into the trained remote sensing image removal model to obtain a shadow-removed remote sensing image.

[0070] Specifically, perform shadow detection on the remote sensing image to be detected to obtain the original shadow mask, perform morphological operations on the original shadow mask to obtain the divided shadow mask that distinguishes the umbra region and the penumbra region and the penumbra-free image that includes the umbra region and the shadow-free region. Input the divided shadow mask and the penumbra-free image into the trained remote sensing image removal model, and a shadow-removed remote sensing image with clear details and natural edges can be obtained. Figure 5 This is the comparison effect diagram before and after shadow removal provided by the embodiment of the present application. Figure 6 This is the comparison effect diagram of the method of the present application and the existing shadow removal methods. From Figure 5 and Figure 6 it can be seen that the method proposed by the present invention has obvious advantages in the overall image color restoration quality, the restoration of tiny detail structures in the shadow region, and the processing of edge artifacts, and can process multi-source images such as high-resolution visible light and multispectral images, with strong versatility.

[0071] Compared with the existing methods, the present invention has the following technical advantages and beneficial effects: ① The present invention adopts a method of coupling the illuminance enhancement statistical method with the generative adversarial learning mode, which simply and effectively solves the common problem that it is difficult to restore the details of ground objects under large-area shadow occlusion, and effectively improves the usability of the remote sensing image shadow removal method based on deep learning.

[0072] ② The present invention separately processes the penumbra region with complex lighting conditions and the umbra with relatively uniform lighting. In the penumbra region, the powerful context-based reconstruction ability of the generative adversarial network is used to achieve precise painting of the penumbra region, making the shadow correction result transition naturally at the edge, and overcoming the edge artifact and penumbra residue problems that are difficult to solve by the existing methods.

[0073] ③ The module based on the adaptive fusion of shallow features and the cross-fusion of deep features enables the network to adaptively fuse color information and detail information in different dimensions, thereby improving the color restoration accuracy and detail clarity of remote sensing shadow removal.

[0074] ④ The hybrid loss function proposed by the present invention, including pixel reconstruction loss, feature consistency loss, color ratio loss, and generative adversarial loss, can guide model training and optimization from multiple aspects such as pixels, features, and color channel relationships, thereby accelerating the convergence speed of the model.

[0075] ⑤ The present invention does not require complex parameter adjustment and manual intervention, and is not limited to processing visible light images. It can also process multi-spectral images including, for example, the near-infrared band, and has strong automation, generalization, and accuracy, and is easy to put into practical use.

[0076] According to the method described in the above embodiments, this embodiment will further describe from the perspective of a remote sensing image shadow removal device based on detail enhancement and edge reconstruction. The remote sensing image shadow removal device based on detail enhancement and edge reconstruction can be specifically implemented as an independent entity, or integrated in an electronic device. The electronic device can be a device such as a terminal or a server. Among them, the terminal can include a tablet computer, a laptop computer, a personal computer (PC), a micro-processing box, or other devices, etc.

[0077] Please refer to Figure 7 , Figure 7 which specifically describes the remote sensing image shadow removal device provided by the embodiments of the present application, applied in an electronic device. The remote sensing image shadow removal device based on detail enhancement and edge reconstruction may include: An acquisition module, configured to acquire a remote sensing image training set; wherein, the remote sensing image training set includes a shadow image and a true shadowless image of the same area corresponding to the shadow image; A processing module, configured to perform shadow detection on the shadowed image to obtain an original shadow mask, and perform morphological operations on the original shadow mask to obtain a divided shadow mask that distinguishes the umbra region and the penumbra region, and a semi-umbra-free image that includes the umbra region and the shadowless region; A training module, configured to input the divided shadow mask and the semi-umbra-free image into a remote sensing image removal model to obtain a shadow removal result, construct a hybrid loss function based on the shadow removal result and the true shadowless image, and train the remote sensing image removal model based on the hybrid loss function; A shadow removal module, configured to acquire a remote sensing image to be detected, perform shadow detection on the remote sensing image, and then input it into the trained remote sensing image removal model to obtain a shadow-removed remote sensing image.

[0078] In specific implementation, each of the above modules and / or units can be implemented as an independent entity, or can be arbitrarily combined and implemented as the same or several entities. For the specific implementation of each of the above modules and / or units, please refer to the foregoing method embodiments. For the beneficial effects that can be specifically achieved, please also refer to the beneficial effects in the foregoing method embodiments, which will not be elaborated herein.

[0079] In addition, an embodiment of the present application further provides an electronic device, which can be a device such as a computer or a tablet computer. The electronic device can implement the steps in any embodiment of the remote sensing image shadow removal method based on detail enhancement and edge reconstruction provided by the embodiments of the present application. Therefore, the beneficial effects that can be achieved by any of the remote sensing image shadow removal methods based on detail enhancement and edge reconstruction provided by the embodiments of the present invention can be realized. For details, please refer to the foregoing embodiments, which will not be elaborated herein.

[0080] Figure 8 The specific structural block diagram of the electronic device provided by the embodiment of the present invention is shown. The electronic device can be used to implement the remote sensing image shadow removal method based on detail enhancement and edge reconstruction provided in the above embodiments. The electronic device 500 can be a device such as a terminal or a server. Among them, the terminal can include a tablet computer, a notebook computer, a personal computer (PC), a microprocessing box, or other devices, etc.

[0081] The RF circuit 510 is used to receive and transmit electromagnetic waves, realizing the mutual conversion between electromagnetic waves and electrical signals, so as to communicate with a communication network or other devices. The RF circuit 510 may include various existing circuit components for performing these functions. For example, an antenna, a radio frequency transceiver, a digital signal processor, an encryption / decryption chip, a subscriber identity module (SIM) card, a memory, and so on. The RF circuit 510 can communicate with various networks such as the Internet, an enterprise intranet, a wireless network or communicate with other devices through a wireless network. The above-mentioned wireless network may include a cellular phone network, a wireless local area network or a metropolitan area network. The above-mentioned wireless network can use various communication standards, protocols and technologies, including but not limited to Global System for Mobile Communication (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (WCDMA), Code Division Access (CDMA), Time Division Multiple Access (TDMA), Wireless Fidelity (Wi-Fi) (such as Institute of Electrical and Electronics Engineers standards IEEE 802.11a, IEEE 802.11b, IEEE 802.11g and / or IEEE 802.11n), Voice over Internet Protocol (VoIP), Worldwide Interoperability for Microwave Access (Wi-Max), other protocols for email, instant messaging and short messages, and any other suitable communication protocols, and may even include those protocols that have not yet been developed currently.

[0082] The memory 520 can be used to store software programs and modules, such as the corresponding program instructions / modules in the above embodiments. The processor 580 executes various functional applications and data processing by running the software programs and modules stored in the memory 520, that is, realizes functions such as taking pictures with the front camera, processing the captured images, and switching the display colors of the display content on the display screen. The memory 520 may include a high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memories. In some instances, the memory 520 may further include a memory remotely provided with respect to the processor 580, and these remote memories can be connected to the electronic device 500 through a network. Examples of the above network include but are not limited to the Internet, enterprise intranet, local area network, mobile communication network, and combinations thereof.

[0083] The input unit 530 can be used to receive input digital or character information, as well as generate a keyboard and a mouse related to user settings and function controls. The display unit 540 can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces, and these graphical user interfaces can be composed of graphics, text, icons, videos, and any combination thereof. The display unit 540 may include a display panel 541. Optionally, the display panel 541 can be configured in the form of an LCD (Liquid Crystal Display) or an OLED (Organic Light-Emitting Diode).

[0084] The audio circuit 560, the speaker 561, and the microphone 562 can provide an audio interface between the user and the electronic device 500. The audio circuit 560 can transmit the electrical signal converted from the received audio data to the speaker 561, and the speaker 561 converts it into a sound signal for output; on the other hand, the microphone 562 converts the collected sound signal into an electrical signal, which is received by the audio circuit 560 and then converted into audio data. After the audio data is output to the processor 580 for processing, it is sent to another terminal, such as through the RF circuit 510, or the audio data is output to the memory 520 for further processing. The audio circuit 560 may also include an earphone jack to provide communication between the peripheral earphone and the electronic device 500.

[0085] The electronic device 500 can help the user receive requests, send information, etc. through the transmission module 570 (such as a Wi-Fi module), and it provides the user with wireless broadband Internet access. Although the transmission module 570 is shown in the figure, it can be understood that it does not belong to the essential composition of the electronic device 500 and can be completely omitted within the scope of not changing the essence of the invention according to needs.

[0086] The processor 580 is the control center of the electronic device 500, connecting various parts of the entire mobile phone through various interfaces and circuits. By running or executing software programs and / or modules stored in the memory 520, and by calling the data stored in the memory 520, it executes various functions of the electronic device 500 and processes data, thereby monitoring the electronic device as a whole. Optionally, the processor 580 may include one or more processing cores; in some embodiments, the processor 580 may integrate an application processor and a modem processor. Among them, the application processor mainly processes the operating system, user interface, application programs, etc., and the modem processor mainly processes wireless communication. It can be understood that the above-mentioned modem processor may not be integrated into the processor 580 either.

[0087] The electronic device 500 further includes a power supply 590 (such as a battery) for supplying power to each component. In some embodiments, the power supply can be logically connected to the processor 580 through a power management system, so as to realize functions such as management of charging, discharging, and power consumption management through the power management system. The power supply 590 may also include any components such as one or more DC or AC power supplies, a recharge system, a power failure detection circuit, a power converter or inverter, and a power status indicator.

[0088] Although not shown, the electronic device 500 further includes a camera (such as a front camera and a rear camera), a Bluetooth module, etc., which will not be elaborated here. Specifically, in this embodiment, the display unit of the electronic device is a touch screen display, and the mobile terminal further includes a memory, and one or more programs, where one or more programs are stored in the memory and are configured to be executed by one or more processors. The one or more programs include instructions for performing the following operations: Obtain a remote sensing image training set; wherein, the remote sensing image training set includes shadow images and true shadowless images of the same area corresponding to the shadow images; Perform shadow detection on the shadow images to obtain an original shadow mask, and perform morphological operations on the original shadow mask to obtain a divided shadow mask that distinguishes the umbra region and the penumbra region and a non-penumbra image that includes the umbra region and the shadowless region; Input the divided shadow mask and the non-penumbra image into a remote sensing image removal model to obtain a shadow removal result, construct a hybrid loss function based on the shadow removal result and the true shadowless image, and train the remote sensing image removal model based on the hybrid loss function; Obtain a remote sensing image to be detected, perform shadow detection on the remote sensing image to be detected, and then input it into the trained remote sensing image removal model to obtain a shadow-removed remote sensing image.

[0089] In specific implementation, each of the above modules can be implemented as an independent entity, or can be arbitrarily combined and implemented as the same or several entities. For the specific implementation of each of the above modules, reference can be made to the foregoing method embodiments, which will not be elaborated herein.

[0090] Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructions, or by controlling related hardware through instructions. The instructions can be stored in a computer-readable storage medium and loaded and executed by a processor. For this purpose, an embodiment of the present invention provides a storage medium in which multiple instructions are stored, and the instructions can be loaded by a processor to execute the steps of any one of the embodiments of the method for removing remote sensing image shadows based on detail enhancement and edge reconstruction provided by the embodiments of the present invention.

[0091] Among them, the computer-readable storage medium may include: read-only memory (ROM, Read Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk, etc.

[0092] Since the instructions stored in the storage medium can execute the steps in any one of the embodiments of the method for removing remote sensing image shadows based on detail enhancement and edge reconstruction provided by the embodiments of the present invention, the beneficial effects that can be achieved by any of the methods for removing remote sensing image shadows based on detail enhancement and edge reconstruction provided by the embodiments of the present invention can be realized. For details, refer to the foregoing embodiments, which will not be elaborated herein.

[0093] The above has introduced in detail a method, device, storage medium, and electronic device for removing remote sensing image shadows based on detail enhancement and edge reconstruction provided by the embodiments of the present application. Specific examples are used in this article to elaborate on the principle and implementation manner of the present application. The description of the above embodiments is only used to help understand the method and its core idea of the present application; at the same time, for those skilled in the art, according to the idea of the present application, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to the present application.

Claims

1. A remote sensing image shadow removal method based on detail enhancement and edge reconstruction, characterized in that: The method comprises: Acquire a remote sensing image training set; wherein the remote sensing image training set includes a shadow image and a real shadow-free image of the same area corresponding to the shadow image; Performing shadow detection on the shadowed image to obtain an original shadow mask, and performing morphological operations on the original shadow mask to obtain a shadow mask that is divided to distinguish between an umbra area and a penumbra area, and a non-penumbra image that includes an umbra area and a non-shadow area; Inputting the divided shadow mask and the penumbra-free image into a remote sensing image removal model to obtain a shadow removal result, constructing a hybrid loss function based on the shadow removal result and the true shadow-free image, and training the remote sensing image removal model based on the hybrid loss function; A remote sensing image to be detected is obtained, and shadow detection is performed on the remote sensing image to be detected and then the image is input into a trained remote sensing image removal model to obtain a shadow-removed remote sensing image.

2. The remote sensing image shadow removal method based on detail enhancement and edge reconstruction according to claim 1 is characterized in that: The remote sensing image removal model includes a umbra area detail enhancement model, a generator and a discriminator; the divided shadow mask and the non-penumbra image are input into the remote sensing image removal model to obtain a shadow removal result, including: Inputting the divided shadow mask and the non-penumbra image into the umbra region detail enhancement model to obtain a detail enhancement image; Inputting the detail enhancement map into the generator to obtain a shadow removal result; The shadow removal result and the real shadow-free image are input into the discriminator to determine whether they are true or false, and based on the determination result, the generator is assisted to output a high-fidelity shadow removal result.

3. The remote sensing image shadow removal method based on detail enhancement and edge reconstruction according to claim 1 is characterized in that: Performing morphological operations on the original shadow mask to obtain a shadow mask that is divided to distinguish the umbra area and the penumbra area and a non-penumbra image that includes the umbra area and the non-shadow area, including: Performing an erosion operation on a number of pixels of the original shadow mask to obtain an umbra area; After dilating the original shadow mask by several pixels, subtract it from the umbra area and use the difference as the penumbra area; Assigning different category values ​​to the umbra region and the penumbra region to obtain a shadow mask after division that distinguishes the umbra region from the penumbra region; A non-penumbra image is obtained based on the penumbra area and the shadowed image.

4. The remote sensing image shadow removal method based on detail enhancement and edge reconstruction according to claim 2 is characterized in that: The step of inputting the divided shadow mask into the umbra region detail enhancement model to obtain a detail enhancement map comprises: Performing a dilation operation on the divided shadow mask by several pixels to obtain a shadow-free area adjacent to the shadow area; Based on the umbra area of ​​the non-penumbra image and the non-shadow area, a normalized cumulative histogram is calculated band by band, and a grayscale value lookup table is constructed; The grayscale value of the umbra area of ​​the non-penumbra image is adjusted based on the grayscale value lookup table, and the adjusted non-penumbra image is converted into a single-channel grayscale image to obtain a detail enhanced image.

5. The remote sensing image shadow removal method based on detail enhancement and edge reconstruction according to claim 3 is characterized in that: The generator includes a color encoder, a detail encoder, a shallow feature adaptive fusion module, a deep feature cross fusion module and a cascade decoder; The step of inputting the detail enhancement map into the generator to obtain a shadow removal result comprises: Inputting the detail enhancement map into the color encoder to extract multi-scale color features to obtain a color feature map; Inputting the detail enhancement map into the detail encoder to extract multi-scale detail features to obtain a detail feature map; Inputting the color feature map and the detail feature map into the shallow feature adaptive fusion module to obtain shallow fusion features, and inputting the color feature map and the detail feature map into the deep feature cross fusion module to obtain high-dimensional fusion features; The shallow fusion features and the high-dimensional fusion features are input into the cascade decoder to obtain a shadow removal result.

6. The remote sensing image shadow removal method based on detail enhancement and edge reconstruction according to claim 5 is characterized in that: The color encoder includes a plurality of convolution modules connected in series, the detail encoder includes a plurality of convolution modules connected in series, and the number of the shallow feature adaptive fusion modules is the same as the number of the convolution modules; In the color encoder and detail encoder, the color feature map and detail feature map output by each layer of convolution module are input into the shallow feature adaptive fusion module of the corresponding level for processing to obtain shallow fusion features.

7. The remote sensing image shadow removal method based on detail enhancement and edge reconstruction according to claim 6 is characterized in that: The shallow feature adaptive fusion module includes a channel attention module, which is sequentially connected to a global pooling layer, a 1×1 convolution layer, a ReLU activation function, a 1×1 convolution layer, and a Sigmoid function; In the shallow feature adaptive fusion module, the channel attention weights of the input color feature map and detail feature map are calculated and then weighted fusion is performed to obtain the shallow fusion feature.

8. The remote sensing image shadow removal method based on detail enhancement and edge reconstruction according to claim 6 is characterized in that: The deep feature cross-fusion module includes an attention module, and the attention module includes a 1×1 convolution layer, a ReLU activation function, a 1×1 convolution layer and a Sigmoid function; In the deep feature cross-fusion module, the attention weights of the input color feature map and detail feature map are calculated and then fused with the original feature maps of each other to obtain high-dimensional fusion features.

9. The remote sensing image shadow removal method based on detail enhancement and edge reconstruction according to claim 1, characterized in that: The hybrid loss function is: in , , and is the weight parameter set, is the mixed loss function, is the pixel reconstruction loss, is the feature consistency loss, is the color ratio loss, To generate adversarial loss.

10. A remote sensing image shadow removal device based on detail enhancement and edge reconstruction, characterized in that: include: An acquisition module is used to acquire a remote sensing image training set; wherein the remote sensing image training set includes a shadow image and a real shadow-free image of the same area corresponding to the shadow image; A processing module, configured to perform shadow detection on the shadowed image to obtain an original shadow mask, and perform morphological operations on the original shadow mask to obtain a shadow mask that is divided to distinguish between an umbra area and a penumbra area, and a non-penumbra image that includes an umbra area and a non-shadow area; A training module, used for inputting the divided shadow mask and the penumbra-free image into a remote sensing image removal model to obtain a shadow removal result, constructing a hybrid loss function based on the shadow removal result and the real shadow-free image, and training the remote sensing image removal model based on the hybrid loss function; The shadow removal module is used to obtain the remote sensing image to be detected, perform shadow detection on the remote sensing image, and then input the shadow into the trained remote sensing image removal model to obtain the shadow-removed remote sensing image.

Images