An unmanned aerial vehicle remote sensing image shadow removal method and system based on residual prototype contrast self-supervised learning

Abstract

The application relates to a UAV remote sensing image shadow removal method and system based on residual prototype contrast self-supervised learning. The method comprises the following steps: acquiring a UAV remote sensing image, and generating an enhanced image by adopting a shadow synthesis enhancement method based on retinal decomposition; extracting online residual errors and target residual errors through a twin generator network; training the twin generator network by jointly outputting consistency loss, shadow-free area protection loss and frequency domain focus loss; training a cross-scene discriminator network by calculating an adversarial loss, and iteratively training the twin generator network to obtain a trained twin generator network; and inputting a to-be-processed image into the trained twin generator network to output a final shadow removal result. By the method and system, the problems of artifacts and incomplete shadow removal in the UAV remote sensing image shadow removal process can be effectively solved, the recovery quality of the UAV remote sensing image is improved, and the manpower and material resources consumed for acquiring paired data of the UAV remote sensing image are effectively reduced.

Description

Technical Field

[0001] This invention relates to the technical field of shadow removal in UAV remote sensing images, and more specifically, to a method and system for shadow removal in UAV remote sensing images based on residual prototype contrastive self-supervised learning. Background Technology

[0002] Compared to conventional photographic imagery, UAV remote sensing imagery provides richer spatial and color information, and these images are widely used in many fields such as earth science, agriculture, urban planning, and disaster monitoring. However, in natural scenes, due to unavoidable factors such as objects blocking sunlight, shadow areas often form, leading to problems such as color distortion and loss of detail in the images. These degraded images directly restrict the performance of downstream advanced tasks such as urban security, wildlife monitoring, and post-disaster recovery management. Therefore, developing UAV remote sensing imagery processing methods that can effectively enhance the brightness and contrast of shadow areas is of significant practical importance.

[0003] Traditional methods for shadow removal in remote sensing imagery typically rely on physical models, statistical models, or variational models based on specific illumination assumptions. However, these artificially designed assumptions often fail to hold true in complex real-world scenarios. While deep learning techniques have made significant strides in shadow removal from ordinary photographic images, achieving high-fidelity shadow removal in UAV imagery remains a major challenge. This stems primarily from several domain-specific difficulties: complex shadow morphology and structure, the high diversity of ground features, and significant domain shifts caused by factors such as sensor differences, geographical location, seasonal variations, and atmospheric conditions.

[0004] To address these challenges, current deep learning methods can be broadly categorized into three paradigms: fully supervised, unsupervised, and self-supervised. Fully supervised methods require training on large-scale shadow-to-shadow-free paired datasets and typically incorporate advanced architectures such as convolutional neural networks, attention mechanisms, and Mamba networks. These architectures ensure training stability and achieve optimal performance by modeling the physical imaging relationship between shadow and non-shadow regions. However, this method heavily relies on expensive paired data, resulting in limited generalization ability and significant performance degradation in domain-shifted scenarios. Unsupervised methods are usually based on recurrent consistency generative adversarial networks (GANs), which learn the mapping relationship between the shadow and non-shadow domains to overcome the dependence on paired data. Some studies have attempted to alleviate the data scarcity problem by constructing synthetic shadow datasets; these methods are more flexible and data-efficient. However, the training process based on CycleGANs requires proxy tasks such as shadow generation, which can easily lead to training oscillations and instability, potentially causing artifacts or incomplete shadow removal.

[0005] To bridge the gap between the data dependency of fully supervised methods and the training instability of unsupervised paradigms, self-supervised methods learn the intrinsic laws of shadow formation and removal by utilizing the physical constraints and internal statistical properties inherent in single images or unpaired datasets. The core advantage of these methods lies in their ability to learn robust domain-invariant representations without relying on paired data or shadow masks, achieving an ideal balance between the high performance of supervised learning and the flexibility of unsupervised learning. Although their test accuracy on some datasets may be slightly lower than supervised methods, their superior cross-domain robustness provides a more practical solution for real-world applications. The emergence of several self-supervised methods in recent years indicates that effectively integrating domain knowledge has become a key research direction. Among them, contrastive self-supervised learning, as one of the mainstream paradigms, aims to maximize the mutual information between different views through a contrastive loss function. By bringing similar samples closer together and pushing dissimilar samples further apart, it learns separable features, thereby benefiting downstream tasks with more discriminative representations. This characteristic effectively eliminates the core contradiction in the field of UAV shadow removal between the difficulty of acquiring paired data and the instability of image restoration model training. Summary of the Invention

[0006] Therefore, it is necessary to provide a method and system for shadow removal from UAV remote sensing images based on residual prototype contrastive self-supervised learning to address the above-mentioned technical problems and improve the restoration quality of shadowed UAV remote sensing images.

[0007] One method for shadow removal in UAV remote sensing images based on residual prototype contrastive self-supervised learning includes: S1: Acquire UAV remote sensing images and generate enhanced images using a shadow synthesis enhancement method based on retinal decomposition; S2: Construct a twin generator network to process UAV remote sensing images and enhanced images respectively, and extract online residuals and target residuals; S3: Construct a residual prototype comparison loss based on online residuals and target residuals, and train the twin generator network by jointly outputting consistency loss, no-shading-area protection loss and frequency domain focus loss; S4: Construct a cross-scene discriminator, use the cross-scene discriminator to calculate the adversarial loss to train the cross-scene discriminator network, drive the generated results to evolve towards the shadowless image style, iterate the training of the Siamese generator network, and obtain the trained Siamese generator network. S5: Input the image to be processed into the trained Siamese generator network and output the final shadow removal result.

[0008] Further, in step S1, specifically: Using the UAV remote sensing image as the original image I, initial illumination-reflection decomposition is performed on the original image I to obtain illumination map L and reflection map R:

[0009] in Represents the retinal decomposition function. The initial shadow mask I is obtained by performing Otsu thresholding on the illumination map L. m Calculate the average illumination value in the non-shaded area:

[0010] Based on this, a unified pseudo-shadowless lighting map is constructed:

[0011] Overlay random shadow mask With decay factor Generate a composite lighting map of shadows:

[0012] in Control the intensity of the synthesized shadows. The final synthesized image retains the shadows while enhancing the image:

[0013]

[0014] Furthermore, in step S2, the twin generator network includes: an online generator and a target generator. via online generator Online residual extraction from original images; online generator The middle has parameters ; Through the target generator Extracting target residuals from enhanced images; target generator Medium parameters for The exponential moving average.

[0015] Further, in step S3, specifically... Step S3.1: Input the original image and its enhanced image into the twin generator network, and output the online residual and target residual at the end of the network; Step S3.2: Based on the obtained target residual, retrieve the most similar one from the residual dictionary. The residual prototypes constitute a candidate set of positive samples, and the most similar prototype is assigned an anchor weight of 1, while the remaining prototypes are assigned a weight of 1. Each auxiliary positive sample is assigned a dynamic weight that is adaptively adjusted by entropy, constructing an algorithm... Pseudo-labels for norm normalization; Step S3.3: Based on the dynamic weighted pseudo-label, the online residuals and all prototypes in the positive sample candidate set are used to form a weighted positive sample pair. The remaining prototypes in the residual dictionary are regarded as negative samples. The residual prototype contrast loss in the form of weighted cross-entropy is constructed. Step S3.4: The online residual is added to the original image to obtain the online restored image, and the target residual is added to the enhanced image to obtain the target restored image. The online restored image and the target restored image are used to construct the output consistency loss. Step S3.5: Compare the original image with the online restored image, compare the enhanced image with the target restored image, and construct the shadowless area protection loss; perform Fast Fourier Transform on the online restored image and the target restored image and calculate the frequency domain focus loss; train the online generator network by combining all loss terms.

[0016] Furthermore, in step S3, the process of constructing a residual prototype comparison loss based on the online residual and the target residual is as follows: Maintain a residual prototype memory in the target generator. Each of them Represents a stored residual prototype in the memory bank, encoding an erroneous shadow removal pattern that the current generator may produce during the shadow removal process; For small batches Shadow images in The online generator produces online residuals. = ( ),pass = + The image is repaired; the target generator processes and enhances the image. syn Generate stable Standard residual ; Based on target residual Based on the similarity distribution with all prototypes in the memory bank, select the one with the highest similarity. The prototypes constitute a positive sample candidate set. And construct pseudo tags according to the following rules and do Norm normalization :

[0017] conditional probability Estimate using the following formula:

[0018] symbol Indicates two The inner product of normalized vectors, For temperature parameters; Dynamic weights Based on the prototype selection probability distribution Entropy adaptive adjustment:

[0019] in ;entropy The larger the value, the more uncertain the prototype selection. Automatic decay is used to reduce the contribution of auxiliary positive samples to the loss function and avoid mistaking hard negative samples as positive samples; Based on the aforementioned dynamically weighted pseudo-labels, the residual prototype contrastive loss... Defined as a weighted cross-entropy form:

[0020] In the formula For online residuals, For online residuals and prototypes The matching probability, calculated as follows: Same; except in the residual dictionary The remaining prototypes other than the original prototypes are used as negative prototypes.

[0021] Furthermore, the dictionary is updated by comparing the loss gradient-driven cooperative-adversarial process. The process of residual prototyping includes: loss function For the prototype The gradient is given by the partial derivative of the residual prototype-comparative loss:

[0022] in Corresponding positive sample candidate set The prototype in correspond Other than the negative prototype; The prototype is then updated using a hybrid gradient strategy, forming a cooperative-adversarial dynamic mechanism: a positive sample candidate set. The prototype in the model updates via gradient descent to more closely approximate the online residual (cooperative), while The negative prototype outside of this process is updated via gradient ascent of the loss function to move away from the online residual (adversarial):

[0023]

[0024] Because when hour This update is equivalent to updating along the similarity. The gradient ascent is used to maximize the similarity between the negative prototype and the online residual, thereby increasing the difficulty for the model to distinguish the negative prototype.

[0025] Furthermore, the process of training the generator network by jointly applying the output consistency loss, the shadowless protection loss, and the frequency domain focus loss includes: Define generator The adversarial objective function is as follows:

[0026] To ensure consistency between the two outputs of the Siamese network, this constraint is achieved through the L1 loss function:

[0027] Furthermore, to preserve information about shadowless areas, an Otsu thresholding method is used to calculate a mask based on the input-output image difference. Using its inverse mask Isolate and compare the shadowless regions between the input and output to obtain the shadowless region protection loss:

[0028] Where ⊙ represents the Hadamarda.

[0029] Furthermore, to constrain the output quality of the generator in the frequency domain, a frequency domain focus loss is constructed. : Online image recovery With target recovery image Perform Fast Fourier Transform on each to obtain its spectrum. ; Calculate the frequency domain weight matrix ,in This is the focusing factor, which controls the penalty intensity for components with large spectral differences. Frequency domain coordinates; Frequency domain focus loss is defined as weighted spectral distance:

[0030] In the formula, N is the total number of pixels in the spectrum; through this loss, the model adaptively applies higher weights to the low-frequency and mid-frequency difference components related to shadows in the spectrum, focuses on optimizing the frequency regions where prediction errors are concentrated, and suppresses shadow residue and artifacts.

[0031] Finally, the total generator loss function of this invention can be expressed as:

[0032]

[0033] in , , This is the loss weighting coefficient.

[0034] Furthermore, in step S4, constructing the cross-scene discriminator specifically involves: A fully convolutional cross-scene discriminator is constructed using the PatchGAN structure. The cross-scene discriminator maps the input image into an N×N matrix, where each element corresponds to the authenticity judgment result of a local receptive field region in the input image. Spectral normalization is applied after each convolutional layer of the cross-scene discriminator to constrain the discriminator's Lipschitz constant and stabilize the adversarial training process.

[0035] Furthermore, the discriminator network is trained by calculating adversarial loss using a cross-scene discriminator, driving the generated results to evolve towards a shadowless image style. The iterative generator network training process includes: Hinge loss is used as the adversarial loss function, where the discriminator loss... Fighting loss against generators They are defined as follows:

[0036]

[0037] In the formula, This represents a true reference image without shadows. This indicates the generated restored image. This is the mean of the output matrix of the cross-scene discriminator.

[0038] The spectral normalization operation is defined as follows: for the weight matrix of each convolutional layer in the cross-scene discriminator... Replace it with:

[0039] in Represents the weight matrix The maximum singular value is estimated by exponential iteration; this operation forces the Lipschitz constant of each convolutional layer in the discriminator to be constrained to 1, thus eliminating the need for additional gradient penalty terms, and works in conjunction with Hinge loss to achieve stable adversarial training, driving the generated image to evolve towards a shadowless image style; the generator and discriminator are iteratively updated until convergence.

[0040] Furthermore, this invention also provides a shadow removal system for UAV remote sensing images based on residual prototype contrastive self-supervised learning, specifically: The data acquisition module is used to acquire UAV remote sensing images and generate enhanced images using a shadow synthesis enhancement method based on retinal decomposition. The data augmentation module is used to construct a twin generator network, which processes UAV remote sensing images and augmented images respectively, and extracts online residuals and target residuals. The residual extraction module is used to construct a residual prototype comparison loss based on the online residual and the target residual, and jointly output the consistency loss, the shadowless area protection loss and the frequency domain focus loss to train the twin generator network; The model training module is used to build a cross-scene discriminator, calculate the adversarial loss using the cross-scene discriminator to train the cross-scene discriminator network, drive the generated results to evolve towards a shadowless image style, iterate the training of the Siamese generator network, and obtain the trained Siamese generator network. The shadow removal module is used to input the image to be processed into the trained Siamese generator network and output the final shadow removal result.

[0041] This invention extracts online and target residuals through a Siamese generator network, effectively capturing differences between images and helping the model focus on shadow region restoration. A loss function built based on residual prototype contrastive learning guides the model to learn the optimal shadow removal mode, improving shadow removal quality. Jointly training the generator with output consistency loss, shadow-free area protection loss, and frequency domain focus loss protects shadow-free area content and enhances model robustness. This method effectively solves the problems of artifacts and incomplete shadow removal, significantly improving image restoration quality while greatly reducing reliance on paired labeled data, thus reducing manpower and material costs. Attached Figure Description

[0042] Figure 1 A flowchart illustrating the steps of a UAV remote sensing image shadow removal method based on residual prototype contrastive self-supervised learning, provided in this application embodiment; Figure 2 This is a schematic diagram of the residual prototype comparison self-supervised network model structure provided in the embodiments of this application; Figure 3 A schematic diagram of a shadow synthesis enhancement method based on retinal decomposition provided in an embodiment of this application; Figure 4 This is a schematic diagram of the generator network structure provided in an embodiment of this application; Figure 5 Example diagram of shadow removal test for UAV remote sensing images provided in the embodiments of this application; Figure 6 This is a schematic diagram of the structure of a UAV remote sensing image shadow removal system based on residual prototype contrastive self-supervised learning, provided as an embodiment of this application. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0044] Example 1: This embodiment provides, as follows: Figure 1 The method for shadow removal in UAV remote sensing images based on residual prototype contrastive self-supervised learning, as shown, includes: S1: Acquire UAV remote sensing images and generate enhanced images using a shadow synthesis enhancement method based on retinal decomposition; S2: Construct a twin generator network to process UAV remote sensing images and enhanced images respectively, and extract online residuals and target residuals; S3: Construct a residual prototype comparison loss based on online residuals and target residuals, and train the twin generator network by jointly outputting consistency loss, no-shading-area protection loss and frequency domain focus loss; S4: Construct a cross-scene discriminator, use the cross-scene discriminator to calculate the adversarial loss to train the cross-scene discriminator network, drive the generated results to evolve towards the shadowless image style, iterate the training of the Siamese generator network, and obtain the trained Siamese generator network. S5: Input the image to be processed into the trained Siamese generator network and output the final shadow removal result.

[0045] This embodiment extracts online residuals and target residuals through a Siamese generator network, which can effectively capture differences between images and help the model focus on the restoration of shadow areas. The loss function built based on residual prototype contrastive learning can guide the model to learn the optimal shadow removal mode and improve the quality of shadow removal. The generator is trained by jointly output consistency loss, shadow-free area protection loss and frequency domain focus loss, which can protect the content of shadow-free areas and enhance the robustness of the model. This method can effectively solve the problems of artifacts and incomplete removal in the shadow removal process, significantly improve the quality of image restoration, and at the same time greatly reduce the dependence on paired labeled data, reducing human and material costs.

[0046] Example 2: This embodiment further discloses information based on Embodiment 1: In step S1, the process of generating enhanced images from the UAV remote sensing images using a shadow synthesis enhancement method based on retinal decomposition includes: Perform initial illumination-reflection decomposition on the original image I to obtain illumination map L and reflection map R:

[0047] in Represents the retinal decomposition function. The initial shadow mask I is obtained by performing Otsu thresholding on the illumination map L. m Calculate the average illumination value in the non-shaded area:

[0048] Based on this, a unified pseudo-shadowless lighting map is constructed:

[0049] Overlay random shadow mask With decay factor Generate a composite lighting map of shadows:

[0050] in Control the intensity of the synthesized shadows. The final synthesized image retains the shadows while enhancing the image:

[0051]

[0052] In the above technical solution, the shadow synthesis enhancement method based on retinal decomposition is used to generate enhanced images from the acquired UAV remote sensing images, which can expand the training data and provide effective self-supervised signals.

[0053] In a preferred embodiment, step S2, which involves processing the original image and the enhanced image separately using a twin generator network to extract the online residual and the target residual for image inpainting, includes: via online generator (parameter Extracting online residuals from the original image; Through the target generator (parameter for The target residual of the enhanced image is extracted using the exponential moving average of the image.

[0054] In this embodiment, by performing shadow synthesis enhancement based on retinal decomposition on the acquired UAV remote sensing images, the training data can be expanded and effective self-supervised signals can be provided; by extracting online residuals and target residuals through a twin generator network, the differences between images can be effectively captured, helping the model focus on the repair of shadow areas.

[0055] Example 3: This embodiment further discloses information based on Embodiment 1: This embodiment further explains steps S3 to S5 in Embodiment 1, as follows: In a preferred embodiment, step S3, the process of constructing a residual prototype comparison loss based on the online residual and the target residual, includes: See Figure 2 Maintain a residual dictionary at the target generator terminal. Each of them This represents a stored "residual prototype" in the memory bank, encoding an erroneous shadow removal pattern that the current generator might produce during shadow removal. During the training of the generator network, the dictionary... The residual prototype in the model is updated through a contrastive loss gradient-driven cooperative-adversarial process, making it the most challenging "error elimination mode" at the decision boundary of the discriminator across scenarios. This is achieved using a target generator. The target residual from The positive prototype is retrieved from the database, and the online residuals are aligned with the positive prototype and moved away from the negative prototype by contrast loss.

[0056] For small batches Shadow images in The online generator produces online residuals. = ( ),pass = + Obtain the restored image. The target generator processes the enhanced image. syn Generate stable target residuals .

[0057] Based on target residual Based on the similarity distribution with all prototypes in the memory bank, select the one with the highest similarity. The prototypes constitute a positive sample candidate set. And construct pseudo tags according to the following rules and do Norm normalization :

[0058] conditional probability Estimate using the following formula:

[0059] symbol Indicates two The inner product of normalized vectors, For temperature parameters; Dynamic weights Based on the prototype selection probability distribution Entropy adaptive adjustment:

[0060] in ;entropy The larger the value, the more uncertain the prototype selection. Automatic decay is used to reduce the contribution of auxiliary positive samples to the loss function and avoid mistaking hard negative samples as positive samples; Based on the aforementioned dynamically weighted pseudo-labels, the residual prototype contrastive loss... Defined as a weighted cross-entropy form:

[0061] In the formula For online residuals, For online residuals and prototypes The matching probability, calculated as follows: same; In the residual dictionary, except The remaining prototypes other than the original prototypes are used as negative prototypes.

[0062] Specifically, the generator network based on residual prototype comparison includes: drone input raw image and its enhanced images syn ; Online generator trained online and momentum update target generator ; Maintained by receiving updates via a cooperative-adversarial process driven by the contrastive loss gradient. ; Original image and enhanced images syn The online residuals of the output image are obtained after passing through an online generator and a target generator, respectively. and target residual Target residual and dictionary Perform cosine similarity calculation on all prototypes and retrieve the top prototypes with the highest similarity scores. One residual prototype is a positive prototype, and the rest are negative prototypes. Together with the online residual construct prototype residual comparison loss, they are used to train the online generator network.

[0063] Furthermore, the dictionary is updated through a cooperative-adversarial process driven by comparing loss gradients. The process of residual prototyping includes: loss function For the prototype The gradient is given by the partial derivative of the residual prototype-comparative loss:

[0064] in Corresponding positive sample candidate set The prototype in correspond Other than the negative prototype; The prototype is then updated using a hybrid gradient strategy, forming a cooperative-adversarial dynamic mechanism: a positive sample candidate set. The prototype in the model updates via gradient descent to more closely approximate the online residual (cooperative), while The negative prototype outside of this process is updated via gradient ascent of the loss function to move away from the online residual (adversarial):

[0065]

[0066] Because when hour This update is equivalent to updating along the similarity. The gradient ascent is used to maximize the similarity between the negative prototype and the online residual, thereby increasing the difficulty for the model to distinguish the negative prototype.

[0067] Understandably, this cooperative-adversarial update strategy, which can be seen as a variant of competitive learning, continuously drives the memory bank. The residual prototype in the process evolved into a more challenging hard case representation. This process forces the online generator... Learning enables online residuals It not only accurately reflects the image residuals but also maintains the greatest difference from all difficult negative modes, thereby improving the quality of image restoration.

[0068] In a preferred embodiment, in step S3, see... Figure 2 The process of constructing a residual prototype contrast loss based on the online residual and the target residual, and jointly training the generator network with the output consistency loss, shadowless area protection loss, and frequency domain focus loss includes: Step S31: Input the original image and its enhanced image into the twin generator network, and output the online residual and target residual at the end of the network; Step S32: Based on the obtained target residual, retrieve the most similar one from the residual dictionary. The residual prototypes constitute a candidate set of positive samples, and the most similar prototype is assigned an anchor weight of 1, while the remaining prototypes are assigned a weight of 1. Each auxiliary positive sample is assigned a dynamic weight that is adaptively adjusted by entropy, constructing an algorithm... Pseudo-labels for norm normalization; Step S33: Based on the dynamic weighted pseudo-label, the online residuals and all prototypes in the positive sample candidate set are used to form a weighted positive sample pair. The remaining prototypes in the residual dictionary are regarded as negative samples. The residual prototype contrast loss in the form of weighted cross-entropy is constructed. Step S34: The online residual is added to the original image to obtain the online restored image, and the target residual is added to the enhanced image to obtain the target restored image. The online restored image and the target restored image are used to construct the output consistency loss. Step S35: Compare the original image with the online restored image, compare the enhanced image with the target restored image, and construct the shadowless area protection loss; perform Fast Fourier Transform on the online restored image and the target restored image and calculate the frequency domain focus loss; train the online generator network by combining all loss terms.

[0069] Specifically, in step S34, the process of constructing the output consistency loss training generator network includes: Define generator The adversarial objective function is as follows:

[0070] To ensure consistency between the two outputs of the Siamese network, this constraint is achieved through the L1 loss function:

[0071] Specifically, in step S35, the process of training the generator network by combining the shadowless region protection loss and the frequency domain focus loss includes: To preserve information about shadowless areas, an Otsu thresholding method is used to calculate a mask based on the input-output image difference. Using its inverse mask Isolate and compare the shadowless regions between the input and output to obtain the shadowless region protection loss:

[0072] Where ⊙ represents the Hadamarda.

[0073] Online image recovery With target recovery image Perform Fast Fourier Transform on each to obtain its spectrum. ; Calculate the frequency domain weight matrix ,in This is the focusing factor, which controls the penalty intensity for components with large spectral differences. Frequency domain coordinates; Frequency domain focus loss is defined as weighted spectral distance:

[0074] In the formula, N is the total number of pixels in the spectrum; through this loss, the model adaptively applies higher weights to the low-frequency and mid-frequency difference components related to shadows in the spectrum, focuses on optimizing the frequency regions where prediction errors are concentrated, and suppresses shadow residue and artifacts.

[0075] Finally, the total generator loss function of this invention can be expressed as:

[0076]

[0077] in , , This is the loss weighting coefficient.

[0078] Understandably, the residual prototype contrast loss function, combined joint output consistency loss, shadowless area protection loss, and frequency domain focus loss can effectively optimize the parameters of the generator network during iterative training, thereby improving the discriminability of the generator network for correct UAV shadow repair residuals and the quality of shadow elimination, while maintaining the invariance of the overall information of the shadowless area.

[0079] Furthermore, the discriminator network is trained by calculating adversarial loss using a cross-scene discriminator, driving the generated results to evolve towards a shadowless image style. The iterative generator network training process includes: A fully convolutional cross-scene discriminator is constructed using the PatchGAN structure. The cross-scene discriminator maps the input image into an N×N matrix, where each element corresponds to the authenticity judgment result of a local receptive field region in the input image. Spectral normalization is applied after each convolutional layer of the cross-scene discriminator to constrain the discriminator's Lipschitz constant and stabilize the adversarial training process.

[0080] Hinge loss is used as the adversarial loss function, where the discriminator loss... Fighting loss against generators They are defined as follows:

[0081]

[0082] In the formula, This represents a true reference image without shadows. This indicates the generated restored image. This is the mean of the output matrix of the cross-scene discriminator.

[0083] The spectral normalization operation is defined as follows: for the weight matrix of each convolutional layer in the cross-scene discriminator... Replace it with:

[0084] in Represents the weight matrix The maximum singular value is estimated by exponential iteration; this operation forces the Lipschitz constant of each convolutional layer in the discriminator to be constrained to 1, thus eliminating the need for additional gradient penalty terms, and works in conjunction with Hinge loss to achieve stable adversarial training, driving the generated image to evolve towards a shadowless image style; the generator and discriminator are iteratively updated until convergence.

[0085] For example, constructing as Figure 2 The self-supervised network shown has the generator network structure as follows: Figure 4 As shown, the encoder consists of an initial 7×7 Conv convolutional layer, a batch normalization (BN) layer, a ReLU activation layer, and three parts: the encoder, ResNetBlock convolutional blocks, and the decoder. The encoder contains three repeated 3×3 Conv convolutional layers, a batch normalization (BN) layer, and a ReLU activation layer. The ResNetBlock convolutional block contains four repeated basic convolutional blocks, each containing one 3×3 Conv convolution, one BN layer, and one ReLU activation layer, followed by another 3×3 Conv convolution and one batch normalization layer. The decoder has the same structure as the encoder. The entire self-supervised framework uses a Siamese network structure, including two identically constructed generator branches. and ,in Update using slow momentum updates In the encoder, convolutional layers extract local features, batch normalization layers normalize these features, and ReLU layers perform non-linear activation. Shadow removal is not performed during network pre-training; after pre-training, an online generator is used to perform shadow removal on the test image.

[0086] Without loss of generality, The image sample of the drone to be tested, To restore the image after removing shadows, It can be represented as:

[0087] In this embodiment, refer to Figure 5 Example images of shadow removal tests for drone images are provided.

[0088] In an embodiment, by means of, Figure 3The method described above employs retinal decomposition-based shadow synthesis enhancement on acquired UAV remote sensing images, which expands training data and provides effective self-supervised signals. Extracting online and target residuals through a Siamese generator network effectively captures differences between images, helping the model focus on shadow region restoration. A loss function built based on residual prototype contrastive learning guides the model to learn the optimal shadow removal mode, improving shadow removal quality. Jointly training the generator with output consistency loss, shadow-free area protection loss, and frequency domain focus loss protects shadow-free area content and enhances model robustness. Adversarial training using a cross-scene discriminator reduces reliance on paired data, enabling the model to learn effective features from unpaired reference images, thus constructing a self-supervised shadow removal model capable of autonomously understanding shadow changes in UAV remote sensing images. This method effectively solves the problems of artifacts and incomplete shadow removal, significantly improving image restoration quality while greatly reducing reliance on paired labeled data and lowering human and material costs.

[0089] Example 4: This embodiment provides, as follows: Figure 6 The system shown is a shadow removal system for UAV remote sensing images based on residual prototype contrastive self-supervised learning, comprising: The data acquisition module is used to acquire UAV remote sensing images and generate enhanced images using a shadow synthesis enhancement method based on retinal decomposition. The data augmentation module is used to construct a twin generator network, which processes UAV remote sensing images and augmented images respectively, and extracts online residuals and target residuals. The residual extraction module is used to construct a residual prototype comparison loss based on the online residual and the target residual, and jointly output the consistency loss, the shadowless area protection loss and the frequency domain focus loss to train the twin generator network; The model training module is used to build a cross-scene discriminator, calculate the adversarial loss using the cross-scene discriminator to train the cross-scene discriminator network, drive the generated results to evolve towards a shadowless image style, iterate the training of the Siamese generator network, and obtain the trained Siamese generator network. The shadow removal module is used to input the image to be processed into the trained Siamese generator network and output the final shadow removal result.

[0090] This embodiment extracts online residuals and target residuals through a Siamese generator network, which can effectively capture differences between images and help the model focus on the restoration of shadow areas. The loss function built based on residual prototype contrastive learning can guide the model to learn the optimal shadow removal mode and improve the quality of shadow removal. The generator is trained by jointly output consistency loss, shadow-free area protection loss and frequency domain focus loss, which can protect the content of shadow-free areas and enhance the robustness of the model. This method can effectively solve the problems of artifacts and incomplete removal in the shadow removal process, significantly improve the quality of image restoration, and at the same time greatly reduce the dependence on paired labeled data, reducing human and material costs.

Claims

1. A method for shadow removal in UAV remote sensing images based on residual prototype contrastive self-supervised learning, characterized in that, include: S1: Acquire UAV remote sensing images and generate enhanced images using a shadow synthesis enhancement method based on retinal decomposition; S2: Construct a twin generator network to process UAV remote sensing images and enhanced images respectively, and extract online residuals and target residuals; S3: Construct a residual prototype comparison loss based on online residuals and target residuals, and train the twin generator network by jointly outputting consistency loss, no-shading-area protection loss and frequency domain focus loss; S4: Construct a cross-scene discriminator, use the cross-scene discriminator to calculate the adversarial loss to train the cross-scene discriminator network, drive the generated results to evolve towards the shadowless image style, iterate the training of the Siamese generator network, and obtain the trained Siamese generator network. S5: Input the image to be processed into the trained Siamese generator network and output the final shadow removal result.

2. The method for shadow removal in UAV remote sensing images based on residual prototype contrastive self-supervised learning according to claim 1, characterized in that, In step S1, specifically: Using UAV remote sensing imagery as the raw imagery For the original image Perform initial illumination-reflection decomposition to obtain the illumination map. With reflection map : in Represents the retinal decomposition function; through the illumination diagram Otsu thresholding is performed to obtain the initial shadow mask I. m Calculate the average illumination value in the non-shaded area: Based on this, a unified pseudo-shadowless lighting map is constructed: Overlay random shadow mask With decay factor Generate a composite lighting map of shadows: in Control the intensity of the synthesized shadows; the final synthesized image retains the shadows while enhancing the image: 。 3. The method for shadow removal in UAV remote sensing images based on residual prototype contrastive self-supervised learning according to claim 1, characterized in that, In step S2, the twin generator network includes: an online generator and a target generator. via online generator Online residual extraction from original images; online generator The middle has parameters ; Through the target generator Extracting target residuals from enhanced images; target generator Medium parameters for The exponential moving average.

4. The method for shadow removal in UAV remote sensing images based on residual prototype contrastive self-supervised learning according to claim 3, characterized in that, In step S3, specifically... Step S3.1: Input the original image and its enhanced image into the twin generator network, and output the online residual and target residual at the end of the network; Step S3.2: Based on the obtained target residual, retrieve the most similar one from the residual dictionary. The residual prototypes constitute a candidate set of positive samples, and the most similar prototype is assigned an anchor weight of 1, while the remaining prototypes are assigned a weight of 1. Each auxiliary positive sample is assigned a dynamic weight that is adaptively adjusted by entropy, constructing an algorithm... Pseudo-labels for norm normalization; Step S3.3: Based on the dynamic weighted pseudo-label, the online residuals and all prototypes in the positive sample candidate set are used to form a weighted positive sample pair. The remaining prototypes in the residual dictionary are regarded as negative samples. The residual prototype contrast loss in the form of weighted cross-entropy is constructed. Step S3.4: The online residual is added to the original image to obtain the online restored image, and the target residual is added to the enhanced image to obtain the target restored image. The online restored image and the target restored image are used to construct the output consistency loss. Step S3.5: Compare the original image with the online restored image, compare the enhanced image with the target restored image, and construct the shadowless area protection loss; perform Fast Fourier Transform on the online restored image and the target restored image and calculate the frequency domain focus loss; train the online generator network by combining all loss terms.

5. The method for shadow removal in UAV remote sensing images based on residual prototype contrastive self-supervised learning according to claim 3, characterized in that, In step S3, the process of constructing a residual prototype comparison loss based on the online residual and the target residual is as follows: Maintain a residual prototype memory in the target generator. Each of them Represents a stored residual prototype in the memory bank, encoding an erroneous shadow removal pattern that the current generator may produce during the shadow removal process; For small batches Shadow images in The online generator produces online residuals. = ( ),pass = + The image is repaired; the target generator processes and enhances the image. syn Generate stable Standard residual ; Based on target residual Based on the similarity distribution with all prototypes in the memory bank, select the one with the highest similarity. The prototypes constitute a positive sample candidate set. And construct pseudo tags according to the following rules and do Norm normalization : conditional probability Estimate using the following formula: symbol Indicates two The inner product of normalized vectors, For temperature parameters; Dynamic weights Based on the prototype selection probability distribution Entropy adaptive adjustment: in ;entropy The larger the value, the more uncertain the prototype selection. Automatic decay is used to reduce the contribution of auxiliary positive samples to the loss function and avoid mistaking difficult negative samples as positive samples; Based on the aforementioned dynamically weighted pseudo-labels, the residual prototype contrastive loss... Defined as a weighted cross-entropy form: In the formula For online residuals and prototypes The matching probability, calculated as follows: Same; except in the residual dictionary The remaining prototypes other than the original prototypes are used as negative prototypes.

6. The method for shadow removal in UAV remote sensing images based on residual prototype contrastive self-supervised learning according to claim 5, characterized in that, The dictionary is updated by comparing the loss gradient-driven cooperative-adversarial process. The process of residual prototyping includes: loss function For the prototype The gradient is given by the partial derivative of the residual prototype-comparative loss: in Corresponding positive sample candidate set The prototype in correspond Other than the negative prototype; The prototype is then updated using a hybrid gradient strategy, forming a cooperative-adversarial dynamic mechanism: a positive sample candidate set. The prototype in the model updates via gradient descent to more closely approximate the online residual (cooperative), while The negative prototype outside of this process is updated via gradient ascent of the loss function to move away from the online residual (adversarial): Because when hour This update is equivalent to updating along the similarity. The gradient ascent is used to maximize the similarity between the negative prototype and the online residual, thereby increasing the difficulty for the model to distinguish the negative prototype.

7. The method for shadow removal in UAV remote sensing images based on residual prototype contrastive self-supervised learning according to claim 1, characterized in that, The process of training the generator network using joint output consistency loss, shadowless protection loss, and frequency domain focus loss includes: Define generator The adversarial objective function is as follows: To ensure consistency between the two outputs of the Siamese network, this constraint is achieved through the L1 loss function: Furthermore, to preserve information about shadowless areas, an Otsu thresholding method is used to calculate a mask based on the input-output image difference. Using its inverse mask Isolate and compare the shadowless regions between the input and output to obtain the shadowless region protection loss: Where ⊙ represents the Hadamard volume; Furthermore, to constrain the output quality of the generator in the frequency domain, a frequency domain focus loss is constructed. : Online image recovery With target recovery image Perform Fast Fourier Transform on each to obtain its spectrum. ; Calculate the frequency domain weight matrix ,in This is the focusing factor, which controls the penalty intensity for components with large spectral differences. Frequency domain coordinates; Frequency domain focus loss is defined as weighted spectral distance: In the formula, N is the total number of pixels in the spectrum; through this loss, the model adaptively applies higher weights to the low-frequency and mid-frequency difference components related to shadows in the spectrum, focuses on optimizing the frequency regions where prediction errors are concentrated, and suppresses shadow residue and artifacts; Finally, the total generator loss function can be expressed as: in , , This is the loss weighting coefficient.

8. The method for shadow removal in UAV remote sensing images based on residual prototype contrastive self-supervised learning according to claim 1, characterized in that, In step S4, the construction of the cross-scene discriminator is specifically as follows: A fully convolutional cross-scene discriminator is constructed using the PatchGAN structure. The cross-scene discriminator maps the input image into an N×N matrix, where each element corresponds to the authenticity judgment result of a local receptive field region in the input image. Spectral normalization is applied after each convolutional layer of the cross-scene discriminator to constrain the discriminator's Lipschitz constant and stabilize the adversarial training process.

9. A method for shadow removal in UAV remote sensing images based on residual prototype contrastive self-supervised learning according to claim 8, characterized in that, The discriminator network is trained by calculating adversarial loss using a cross-scene discriminator, driving the generated results to evolve towards a shadowless image style. The iterative generator network training process includes: Hinge loss is used as the adversarial loss function, where the discriminator loss... Fighting loss against generators They are defined as follows: In the formula, This represents a true reference image without shadows. This indicates the generated restored image. The mean of the output matrix for the cross-scene discriminator; The spectral normalization operation is defined as follows: for the weight matrix of each convolutional layer in the cross-scene discriminator... Replace it with: in Represents the weight matrix The maximum singular value is estimated by exponential iteration; this operation forces the Lipschitz constant of each convolutional layer in the discriminator to be constrained to 1, thus eliminating the need for additional gradient penalty terms, and works in conjunction with Hinge loss to achieve stable adversarial training, driving the generated image to evolve towards a shadowless image style; the generator and discriminator are iteratively updated until convergence.

10. A shadow removal system for UAV remote sensing images based on residual prototype contrastive self-supervised learning, characterized in that, Specifically: The data acquisition module is used to acquire UAV remote sensing images and generate enhanced images using a shadow synthesis enhancement method based on retinal decomposition. The data augmentation module is used to construct a twin generator network, which processes UAV remote sensing images and augmented images respectively, and extracts online residuals and target residuals. The residual extraction module is used to construct a residual prototype comparison loss based on the online residual and the target residual, and jointly output the consistency loss, the shadowless area protection loss and the frequency domain focus loss to train the twin generator network; The model training module is used to build a cross-scene discriminator, calculate the adversarial loss using the cross-scene discriminator to train the cross-scene discriminator network, drive the generated results to evolve towards a shadowless image style, iterate the training of the Siamese generator network, and obtain the trained Siamese generator network. The shadow removal module is used to input the image to be processed into the trained Siamese generator network and output the final shadow removal result.

Images