SAR (Synthetic Aperture Radar)-to-optical image dual-fidelity conversion method based on unidirectional knowledge migration

Abstract

The invention relates to an SAR-to-optical image dual-fidelity conversion method based on unidirectional knowledge migration. The method comprises the following steps: inputting an SAR image sample into a detail reconstruction sub-network and a semantic preserving sub-network for basic capability training; constructing a detail feature migration channel from the detail reconstruction sub-network to the semantic maintenance sub-network by using a one-way consistency loss function, or constructing a global semantic migration channel from the semantic maintenance sub-network to the detail reconstruction sub-network, and obtaining the semantic maintenance sub-network or the detail reconstruction sub-network based on one-way knowledge migration; and converting the SAR image into an optical image by using a semantic preserving sub-network or a detail reconstruction sub-network based on unidirectional knowledge migration. The method can overcome the internal contradiction between global semantic maintenance and local texture structure detail reconstruction of a single-branch generative adversarial network, the disjunction problem between a multi-scale discriminator and an internal functional module of a generator, and the optimization target conflict caused by lack of directional constraint in double-branch feature fusion.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image processing, specifically to a SAR to optical image double-fidelity conversion method based on one-way knowledge transfer. Background Technology

[0002] SAR (Synthetic Aperture Radar) images and optical images differ significantly in imaging mechanisms, physical characteristics, and image morphology. These differences severely restrict their application and analysis in subsequent tasks. SAR-to-optical image conversion, a data processing technique that transforms heterogeneous remote sensing images into images with homogeneous features, effectively addresses the modal differences between SAR and optical images. This provides crucial technical support for subsequent applications and analyses such as SAR-to-optical image matching, SAR-to-optical image transformation detection, SAR image target recognition, SAR-to-optical image wildfire assessment, and early crop growth monitoring. Therefore, research on SAR-to-optical image conversion is of great significance. However, existing image conversion methods still face significant challenges in SAR-to-optical image conversion. First, due to the complex and diverse types of ground features in remote sensing images, different ground features exhibit significant differences in feature representation, requiring models to possess strong global semantic understanding and conversion capabilities. Second, the speckle noise unique to SAR imaging not only interferes with image quality but also blurs the edge structure information of ground features, necessitating a balance between denoising effectiveness and edge preservation accuracy during the conversion process. Furthermore, the inter-domain differences caused by different imaging conditions, sensor parameters, and regional characteristics also place higher demands on the model's generalization ability.

[0003] To overcome these challenges, researchers have proposed a series of generative adversarial networks (GANs) for SAR-to-optical image conversion tasks to improve the quality and fidelity of generated images. Researchers initially focused on optimizing the generator (e.g., ...). Figure 1 (a): A generator-optimized network framework that operates within a supervised learning paradigm. L IS (Representing image similarity loss) By introducing feature enhancement modules, attention mechanisms, and multi-scale feature fusion into the generator, the ability of the generator to capture and utilize SAR image features is enhanced, thereby improving the quality of the generated images. However, this type of optimization method has significant limitations. The model struggles to simultaneously consider two distinct tasks: global semantic information and local texture structure detail reconstruction. This leads to irreconcilable internal conflicts within the network during optimization, hindering further improvements in generation quality. To overcome this limitation, researchers have proposed two new optimization methods: one is to introduce a multi-scale discriminator on the basis of optimizing the generator, providing hierarchical supervision from global to local (e.g., ...). Figure 1(b): A network framework that introduces a multi-scale discriminator, which operates under a supervised learning paradigm. L IS The first method represents image similarity loss; the second method extracts global semantic features and local texture features of the image in the bottleneck layer of the generator, and then performs feature-level complementary fusion to simultaneously improve the reconstruction quality of global semantic information and local texture details (e.g., Figure 1 (c): The network framework of the dual-branch feature fusion method, which is carried out under the supervised learning paradigm. L IS (This represents image similarity loss). However, both of these optimization methods still have certain limitations. First, while introducing a multi-scale discriminator improves the visual fidelity and structural integrity of the generated images by discriminating at different spatial scales, this multi-scale discrimination method is disconnected from the internal functional modules of the generator, making it difficult to perform targeted optimization on specific modules within the generator that perform different tasks. Second, although the dual-branch feature fusion method constructs parallel feature extraction channels to extract global semantic features and local texture features of the image respectively, improving the feature richness of the generated image, the loss function only applies to the final output after fusion. It cannot apply targeted constraints to the global semantic and local texture features extracted by the dual branches. This can lead to the global branch weakening the consistency of global semantic information due to the network's excessive focus on local details, and the local branch losing fine texture structure details due to the lack of clear detail reconstruction constraints. This limits the simultaneous improvement of both global semantic accuracy and local texture structure realism in the generated image, thus restricting the overall visual fidelity of the generated image. Summary of the Invention

[0004] This invention provides a SAR-to-optical image dual-fidelity conversion method based on unidirectional knowledge transfer, which overcomes the inherent contradiction between global semantic preservation and local texture structure detail reconstruction in existing single-branch generative adversarial networks, the problem of disconnect between the internal functional modules of the multi-scale discriminator and the generator, and the optimization objective conflict caused by the lack of directional constraints in dual-branch feature fusion methods.

[0005] The technical solution of this invention to solve the above-mentioned technical problems is as follows: A SAR-to-optical image double-fidelity conversion method based on one-way knowledge transfer, which utilizes a one-way knowledge transfer generative adversarial network with a bi-branch generative framework to convert the SAR image to be processed into an optical image. The one-way knowledge transfer generative adversarial network includes a detail reconstruction sub-network and a semantic preservation sub-network of the bi-branch generative framework; the SAR-to-optical image double-fidelity conversion method includes: SAR image samples are input into the detail reconstruction subnetwork and the semantic preservation subnetwork respectively to generate two sets of optical images that focus on the local texture structure details of the image and optical images that focus on the global semantic information of the image, respectively. Adversarial discrimination is performed on the two sets of generated optical images to guide the detail reconstruction subnetwork and the semantic preservation subnetwork to focus on the reconstruction of local texture structure details and the preservation of global semantic information, respectively, thereby realizing the basic capability training of the detail reconstruction subnetwork and the semantic preservation subnetwork. Based on the completion of basic capability training of the detail reconstruction subnetwork and the semantic preservation subnetwork, a detail feature transfer channel from the detail reconstruction subnetwork to the semantic preservation subnetwork is constructed using a one-way consistency loss function. This transfers the ability of the detail reconstruction subnetwork to reconstruct local texture structure details to the semantic preservation subnetwork, resulting in a semantic preservation subnetwork based on one-way knowledge transfer. Alternatively, a global semantic transfer channel from the semantic preservation subnetwork to the detail reconstruction subnetwork is constructed using a one-way consistency loss function. This transfers the ability of the semantic preservation subnetwork to preserve global semantic information to the detail reconstruction subnetwork, resulting in a detail reconstruction subnetwork based on one-way knowledge transfer. The SAR image to be processed is converted into the corresponding optical image by using a semantic preservation subnetwork based on one-way knowledge transfer or a detail reconstruction subnetwork based on one-way knowledge transfer.

[0006] Based on the above technical solution, the present invention can be further improved as follows.

[0007] Furthermore, the detailed reconstruction subnetwork includes a generator for image transformation and a discriminator for determining the authenticity of the generated image; The generator of the detailed reconstruction subnetwork uses the Unet network. In the Unet network, the layers of the encoder and the layers of the decoder are connected in a skip connection to perform multi-scale fusion of the feature maps in the encoder and the feature maps in the decoder. The Unet network adopts a progressive feature expansion strategy, which gradually increases the channel dimension of the features while keeping the spatial scale of the feature maps unchanged. The innermost layer of the encoder and decoder in the Unet network is embedded with a CBAM module to embed the channel attention mechanism and the spatial attention mechanism into the innermost layer of the encoder and decoder, and automatically learns the weights of each feature channel through the channel attention mechanism. The generator and discriminator of the detailed reconstruction subnetwork adopt the PatchGAN architecture, with the number of convolutional layers in the PatchGAN architecture set to 2.

[0008] Furthermore, the semantic preservation subnetwork includes a generator for image transformation and a discriminator for determining the authenticity of the generated image; The generator of the semantically preserving subnetwork uses a ResNet network. The bottleneck layer of the ResNet network embeds a Transformer encoding module with positional encoding. The Transformer encoding module is used to preserve the key spatial structure information of the feature map by introducing a learnable positional encoding mechanism, and to use a multi-head attention mechanism to compute attention weights in parallel in multiple independent representation subspaces to simultaneously capture long-range dependencies at different scales. Residual modules are embedded in the downsampling and upsampling layers of the encoder and decoder in the ResNet network. The discriminator of the semantic preservation subnetwork adopts the PatchGAN architecture, and the number of convolutional layers in the PatchGAN architecture is set to 4.

[0009] Furthermore, both the detail reconstruction subnetwork and the semantic preservation subnetwork include a generator for image transformation and a discriminator for determining the authenticity of the generated image; The total loss function of the unidirectional knowledge transfer generative adversarial network is expressed as follows: ; In the formula, The total loss function for the generative adversarial network for the unidirectional knowledge transfer is given. , , , and The weights are the proportions of the loss function. The image similarity loss function for the generator in the detailed reconstruction subnetwork is used. The generative adversarial loss function of the discriminator in the detailed reconstructed subnetwork is used to reconstruct the discriminator. Let be the image similarity loss function of the generator in the semantically preserving subnetwork. Let G be the generative adversarial loss function of the discriminator in the semantically preserving subnetwork. Let be the one-way consistency loss function.

[0010] Furthermore, the image similarity loss function of the detailed reconstruction subnetwork is expressed as: ; In the formula, It is a norm of 1. For real SAR images, For true optical images, The optical image generated by the detailed reconstruction subnetwork.

[0011] Furthermore, the generative adversarial loss function of the detailed reconstruction subnetwork is expressed as: ; In the formula, For the expected operation, The discriminator reconstructs the subnetwork for the aforementioned details. For real SAR images, For true optical images, The optical image generated by the detailed reconstruction subnetwork.

[0012] Furthermore, the image similarity loss function of the semantically preserving sub-network is expressed as: ; In the formula, and The weights are the proportions of the loss function. Let the content consistency loss function be... The feature consistency loss is used; where: ; ; In the formula, For real SAR images, For true optical images, The optical image generated for the semantically preserving subnetwork. It is a norm of 1. It is a 2-norm. The weighting parameters between feature maps, To obtain the true optical image extracted using VGG in the first... l On the layer feature map Features at a spatial location The optical image generated by the semantic-preserving subnetwork extracted using VGG is in the first... l On the layer feature map Features at a spatial location and For the first l The height and width of the layer feature map. and These are the spatial coordinates of the feature map.

[0013] Furthermore, the generative adversarial loss function of the semantic-preserving sub-network is expressed as: ; In the formula, For the expected operation, For the discriminator of the semantically preserving subnetwork, For real SAR images, For true optical images, Optical images generated for the semantically preserving subnetwork.

[0014] Furthermore, when the ability of the detail reconstruction subnetwork to reconstruct local texture structure details is transferred to the semantic preservation subnetwork, the one-way consistency loss function is expressed as: ; In the formula, The one-way consistency loss function from the detailed reconstruction subnetwork to the semantically preserving subnetwork is given. To stop gradient operations and prevent model parameters from being propagated back, For real SAR images, The optical image generated by the detailed reconstruction subnetwork. The optical image generated for the semantically preserving subnetwork. It is a norm of 1. An optical image generated to stop gradient operations on the detailed reconstruction subnetwork and to prevent model parameters from being backpropagated.

[0015] Furthermore, when the ability of the semantic preservation subnetwork to preserve global semantic information is transferred to the detail reconstruction subnetwork, the one-way consistency loss function is expressed as: ; In the formula, The one-way consistency loss function from the semantically preserving subnetwork to the detail reconstruction subnetwork is given. and The weights are the proportions of the loss function. To stop gradient operations and prevent model parameters from being propagated back, For real SAR images, The optical image generated by the detailed reconstruction subnetwork. The optical image generated for the semantically preserving subnetwork. It is a norm of 1. It is a 2-norm. The weighting parameters between feature maps, The optical image generated by the VGG-extracted detail reconstruction subnetwork is in the first... l On the layer feature map Features at a spatial location and For the first l The height and width of the layer feature map. and For the spatial coordinates of the feature mapping, An optical image generated to stop gradient operations on the semantically preserving subnetwork and without backpropagating model parameters. The optical image generated to stop gradient operations on the semantically preserving subnetwork and to prevent model parameters from backpropagating is in the first... l On the layer feature map Features at a spatial location.

[0016] The beneficial effects of this invention are as follows: The SAR to optical image dual-fidelity conversion method based on unidirectional knowledge transfer first adopts a dual-branch generation framework, which assigns two fundamentally different tasks, local texture structure detail reconstruction and global semantic information preservation, to the detail reconstruction subnetwork and the semantic preservation subnetwork. Structurally, this overcomes the inherent limitation of single-branch generative adversarial networks, which cannot simultaneously take into account the consistency of global semantic information and the fidelity of local texture structure details. Building upon this, a proprietary dual-branch discriminator and task-oriented loss function are employed. The detail reconstruction sub-network is equipped with a shallow discriminator focused on local texture structure discrimination, and pixel-level loss constraints are applied to ensure the authenticity of local texture structure detail reconstruction. Simultaneously, the semantic preservation sub-network is equipped with a deep discriminator responsible for global semantic information consistency evaluation, supplemented by pixel-level and feature-level joint loss constraints to ensure the consistency of global semantic information. This achieves directional optimization of the two sub-networks and fundamentally solves the problem of disconnect between the multi-scale discriminator and the internal functional modules of the generator, avoiding the optimization objective conflict caused by the lack of directional constraints in the dual-branch feature fusion method. Finally, by utilizing one-way consistency loss, the ability of the detail reconstruction subnetwork to capture and reconstruct both global semantic information and local texture details is transferred to the semantic preservation subnetwork. This ensures the dual fidelity of the optical images generated by the semantic preservation subnetwork, which simultaneously possesses both global semantic information and local texture details, thereby improving the overall quality of images generated by the semantic preservation subnetwork based on one-way knowledge transfer. Alternatively, by utilizing one-way consistency loss, the ability of the semantic preservation subnetwork to preserve global semantic information is transferred to the detail reconstruction subnetwork. This ensures the dual fidelity of the optical images generated by the detail reconstruction subnetwork, which simultaneously possesses both global semantic information and local texture details, thereby improving the overall quality of images generated by the detail reconstruction subnetwork based on one-way knowledge transfer. Attached Figure Description

[0017] Figure 1 A schematic diagram of a conditional generative adversarial network framework for SAR-to-optical image conversion in existing technologies; Figure 2 A schematic diagram of the network framework for a generative adversarial network for unidirectional knowledge transfer; Figure 3 A schematic diagram of the overall framework for generative adversarial networks for unidirectional knowledge transfer; Figure 4 This is a flowchart of a SAR-to-optical image double-fidelity conversion method based on one-way knowledge transfer according to the present invention; Figure 5 A schematic diagram of the CBAM module. Figure 6 This is a schematic diagram of the PE-Transformer module and the residual module; Figure 7 Examples of images for different land cover types in the SEN1-2 and WHU-OPT-SAR datasets. Detailed Implementation

[0018] The principles and features of the present invention are described below with reference to the accompanying drawings. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0019] A dual-fidelity SAR-to-optical image conversion method based on one-way knowledge transfer is proposed. This method utilizes a one-way knowledge transfer generative adversarial network (UKT-GAN) with a dual-branch generative framework to convert the SAR image to be processed into an optical image. The network framework of the one-way knowledge transfer generative adversarial network is as follows: Figure 2 As shown; the unidirectional knowledge transfer generative adversarial network includes a detail reconstruction subnetwork and a semantic preservation subnetwork within a dual-branch generative framework. The overall framework of the unidirectional knowledge transfer generative adversarial network is as follows: Figure 3 As shown; like Figure 4 As shown, the SAR to optical image double-fidelity conversion method includes: SAR image samples are input into the detail reconstruction subnetwork and the semantic preservation subnetwork respectively to generate two sets of optical images that focus on the local texture structure details of the image and optical images that focus on the global semantic information of the image, respectively. Adversarial discrimination is performed on the two sets of generated optical images to guide the detail reconstruction subnetwork and the semantic preservation subnetwork to focus on the reconstruction of local texture structure details and the preservation of global semantic information, respectively, thereby realizing the basic capability training of the detail reconstruction subnetwork and the semantic preservation subnetwork. Based on the completion of basic capability training of the detail reconstruction subnetwork and the semantic preservation subnetwork, a detail feature transfer channel from the detail reconstruction subnetwork to the semantic preservation subnetwork is constructed using a one-way consistency loss function. This transfers the ability of the detail reconstruction subnetwork to reconstruct local texture structure details to the semantic preservation subnetwork, resulting in a semantic preservation subnetwork based on one-way knowledge transfer. Alternatively, a global semantic transfer channel from the semantic preservation subnetwork to the detail reconstruction subnetwork is constructed using a one-way consistency loss function. This transfers the ability of the semantic preservation subnetwork to preserve global semantic information to the detail reconstruction subnetwork, resulting in a detail reconstruction subnetwork based on one-way knowledge transfer. The SAR image to be processed is converted into the corresponding optical image by using a semantic preservation subnetwork based on one-way knowledge transfer or a detail reconstruction subnetwork based on one-way knowledge transfer.

[0020] The overall framework of the UKT-GAN for SAR to optical image conversion in this invention is as follows: Figure 2 As shown, this network employs a supervised learning method, trained using paired SAR and optical images. It mainly consists of a detail reconstruction subnetwork and a semantic preservation subnetwork trained in parallel. Both subnetworks are designed based on the GAN architecture. The generator part uses a differentiated framework design, while the discriminator part, although sharing the same framework, differs in the specific configuration of the number of convolutional layers.

[0021] During training, SAR image data is first synchronously input into a detail reconstruction subnetwork and a semantic preservation subnetwork to generate two sets of optical images: one focusing on local texture details and the other on global semantic information. Then, discriminators with different receptive fields are used to adversarially discriminate between the two sets of generated optical images, guiding the detail reconstruction subnetwork and the semantic preservation subnetwork to focus on reconstructing local texture details and preserving global semantic information, respectively. Finally, after the detail reconstruction subnetwork and the semantic preservation subnetwork have completed basic capability training (e.g., more than 100 training rounds), a detailed feature transfer channel from the detail reconstruction subnetwork to the semantic preservation subnetwork is constructed using the designed unidirectional consistency loss function. This ensures that the semantic preservation subnetwork can continuously learn the fine texture structure details extracted by the detail reconstruction subnetwork during adversarial training, thereby enabling the semantic preservation subnetwork to acquire image transformation capabilities that take into account both global semantic information and local texture structure details. Alternatively, a global semantic transfer channel from the semantic preservation subnetwork to the detail reconstruction subnetwork is constructed using the designed unidirectional consistency loss function. This ensures that the detail reconstruction subnetwork can continuously learn the global semantic information maintained by the semantic preservation subnetwork during adversarial training, thereby enabling the detail reconstruction subnetwork to acquire image transformation capabilities that take into account both global semantic information and local texture structure details.

[0022] During testing, only a semantically preserving subnetwork based on one-way knowledge transfer or a detail reconstruction subnetwork based on one-way knowledge transfer was used to convert the input SAR image into the corresponding optical image.

[0023] The following section provides a detailed introduction to one-way knowledge transfer generative adversarial networks.

[0024] The detail reconstruction subnetwork for SAR-to-optical image conversion focuses on capturing and reconstructing local texture details of the image, providing a reference for optimizing the local texture structure for the semantic preservation subnetwork. For example... Figure 3As shown in (a), it mainly consists of a generator responsible for image conversion and a discriminator used to determine the authenticity of the generated image.

[0025] The generator for the detail reconstruction subnetwork employs a UNet network. It uses skip connections to fuse feature maps from the encoder and decoder at multiple scales, thus preserving richer spatial detail information. Furthermore, to avoid excessive compression of image information by downsampling in the bottleneck layer of the UNet network, a progressive feature expansion strategy is adopted. While maintaining the spatial scale of the feature maps, only the channel dimension of the features is gradually increased to avoid information loss and enhance the expressive power of the features. In addition, to further improve the feature expressive power of the shallow encoder-decoder network, this invention also utilizes... Figure 5 The CBAM module shown (a lightweight attention module proposed by Ferrari et al. in 2018) embeds channel attention and spatial attention mechanisms into the innermost layer of a shallow encoder-decoder network. It automatically learns the weights of each feature channel through the channel attention mechanism to enhance the expression of key feature channels, while combining the spatial attention mechanism to focus on important regions in the feature map, thereby further improving the quality of the local texture structure of the generated image.

[0026] The discriminator in the detail reconstruction subnetwork—a shallow discriminator—adopts the PatchGAN architecture with only two convolutional layers, giving it a limited local receptive field and focusing on judging the authenticity of local texture details in the image. Furthermore, through adversarial training between the generator and the discriminator, the generator is guided to learn to generate optical images with rich texture details.

[0027] A semantically preserving subnetwork for SAR-to-optical image conversion: The semantically preserving subnetwork focuses on maintaining the consistency of global semantic information in the image. It utilizes unidirectional consistency constraints to transfer the ability of the detail reconstruction subnetwork to capture and reconstruct local texture details, thereby improving the overall quality of the images generated by the semantically preserving subnetwork. For example... Figure 3 As shown in (b), it also consists of a generator that performs image transformation and a discriminator that determines the authenticity of an image.

[0028] The generator for the semantically preserving subnetwork uses a ResNet network to extract high-level semantic information from images. However, unlike the traditional ResNet framework that simply stacks residual modules in the bottleneck layer, this invention embeds a Transformer encoding module with positional encoding (PE-Transformer for short) in the bottleneck layer. Figure 6As shown in (a), the Transformer encoding module first introduces a learnable positional encoding mechanism to preserve key spatial structure information of the feature map, thus avoiding the loss of spatial positional information caused by pure attention operations. Then, it utilizes a multi-head attention mechanism to compute attention weights in parallel across multiple independent representation subspaces, simultaneously capturing long-range dependencies at different scales, thereby significantly overcoming the inherent limitations of traditional convolutional neural networks in global context modeling. Regarding the structural design of the encoder and decoder, this invention also embeds, during the downsampling and upsampling processes of the encoder and decoder, elements such as... Figure 6 (b) shows the residual module, which ensures that information can be stably transmitted in the multi-layer network to enhance the integrity of global feature extraction and avoid the gradient vanishing problem, thereby improving the quality of the generated image.

[0029] The discriminator of the semantically preserving subnetwork—the deep discriminator: The discriminator of the semantically preserving subnetwork is also based on the PatchGAN architecture, but its number of convolutional layers is set to 4, thereby obtaining a larger receptive field, enabling it to effectively evaluate the overall structural rationality and global semantic consistency of the generated image, thereby guiding the generator to generate optical images with higher visual realism on a global scale.

[0030] Loss Function: The UKT-GAN proposed in this invention mainly consists of two parts: a detail reconstruction subnetwork and a semantic preservation subnetwork. Its total loss function is mainly divided into three categories: the loss function of the detail reconstruction subnetwork, the loss function of the semantic preservation subnetwork, and the one-way consistency loss function.

[0031] During the training of the detail reconstruction subnetwork for SAR-to-optical image conversion, its loss function mainly includes two types: image similarity loss and generative adversarial loss. The image similarity loss function of the detail reconstruction subnetwork is expressed as: ; In the formula, It is a norm of 1. For real SAR images, For true optical images, The optical image generated by the detailed reconstruction subnetwork.

[0032] The generative adversarial loss function of the detailed reconstruction subnetwork is expressed as: ; In the formula, For the expected operation, The discriminator reconstructs the subnetwork for the aforementioned details. For real SAR images, For true optical images, The optical image generated by the detailed reconstruction subnetwork.

[0033] In semantic preservation networks, the overall loss function is also divided into two categories: image similarity loss and generative adversarial loss. Regarding image similarity loss, to further improve the global semantic preservation ability of semantic preservation networks, it not only utilizes... L 1 Loss Constructs Content Consistency Loss Function L content To ensure the accuracy of the generated images at the pixel level, a feature consistency loss function is also introduced. L feature This ensures the realism of the generated images within the deep feature space. The image similarity loss function of the semantically preserving subnetwork is expressed as: ; In the formula, and The weights are the proportions of the loss function. Let the content consistency loss function be... The feature consistency loss is used; where: ; ; In the formula, For real SAR images, For true optical images, The optical image generated for the semantically preserving subnetwork. It is a norm of 1. It is a 2-norm. The weighting parameters between feature maps, To extract real optical images using VGG (a convolutional neural network model proposed in 2014 by Karen Simonyan and Andrew Zisserman of the Visual Geometry Group at Oxford University) in the first... l On the layer feature map Features at a spatial location The optical image generated by the semantic-preserving subnetwork extracted using VGG is in the first... l On the layer feature map Features at a spatial location and For the first l The height and width of the layer feature map. and These are the spatial coordinates of the feature map. In this embodiment, λ1=100 and λ2=30 represent the spatial coordinates of the feature map. L content and Lfeature The percentage weight.

[0034] The generative adversarial loss function of the semantically preserving subnetwork is expressed as: ; In the formula, For the expected operation, For the discriminator of the semantically preserving subnetwork, For real SAR images, For true optical images, Optical images generated for the semantically preserving subnetwork.

[0035] To transfer the ability of the detail reconstruction subnetwork to capture and reconstruct local texture details of an image to the semantic preservation subnetwork, this invention designs a one-way consistency loss from the detail reconstruction subnetwork to the semantic preservation subnetwork. L UC(SP) This loss, based on the completion of basic capability training (more than 100 training epochs) of the detail reconstruction subnetwork and the semantic preservation subnetwork, encourages the semantic preservation subnetwork to continuously learn the fine texture structure details extracted by the detail reconstruction subnetwork by directionally constraining the difference between their outputs. Furthermore, to verify the effectiveness of the unidirectional knowledge transfer network in ablation experiments, this invention also designs a unidirectional consistency loss that transfers the global semantic information preservation capability of the semantic preservation subnetwork to the detail reconstruction subnetwork. L UC(DR) Similarly, after more than 100 training epochs, it performs unidirectional knowledge transfer from the semantically preserving subnetwork to the detail reconstruction subnetwork. The loss functions for these two unidirectional consistency are designed as follows: When the ability of the detail reconstruction subnetwork to reconstruct local texture structure details is transferred to the semantic preservation subnetwork, the one-way consistency loss function is expressed as: ; In the formula, The one-way consistency loss function from the detailed reconstruction subnetwork to the semantically preserving subnetwork is given. To stop gradient operations and prevent model parameters from being propagated back, For real SAR images, The optical image generated by the detailed reconstruction subnetwork. The optical image generated for the semantically preserving subnetwork. It is a norm of 1. An optical image generated to stop gradient operations on the detailed reconstruction subnetwork and to prevent model parameters from being backpropagated.

[0036] When the ability of the semantic preservation subnetwork to preserve global semantic information is transferred to the detail reconstruction subnetwork, the one-way consistency loss function is expressed as: ; In the formula, The one-way consistency loss function from the semantically preserving subnetwork to the detail reconstruction subnetwork is given. and The weights are the proportions of the loss function. To stop gradient operations and prevent model parameters from being propagated back, For real SAR images, The optical image generated by the detailed reconstruction subnetwork. The optical image generated for the semantically preserving subnetwork. It is a norm of 1. It is a 2-norm. The weighting parameters between feature maps, The optical image generated by the VGG-extracted detail reconstruction subnetwork is in the first... l On the layer feature map Features at a spatial location and For the first l The height and width of the layer feature map. and For the spatial coordinates of the feature mapping, An optical image generated to stop gradient operations on the semantically preserving subnetwork and without backpropagating model parameters. The optical image generated to stop gradient operations on the semantically preserving subnetwork and to prevent model parameters from backpropagating is in the first... l On the layer feature map Features at spatial location. In this embodiment, λ3=10 and λ4=5.

[0037] Since UKT-GAN utilizes the unidirectional consistency constraint to transfer the ability of the detail reconstruction subnetwork to capture and reconstruct local texture structure details of the image to the semantic preservation subnetwork, the total loss function of UKT-GAN is expressed as follows: ; In the formula, The total loss function for the generative adversarial network for the unidirectional knowledge transfer is given. , , , and The weights are the proportions of the loss function. The image similarity loss function for the generator in the detailed reconstruction subnetwork is used. The generative adversarial loss function of the discriminator in the detailed reconstructed subnetwork is used to reconstruct the discriminator. Let be the image similarity loss function of the generator in the semantically preserving subnetwork. Let λ5 be the generative adversarial loss function of the discriminator in the semantically preserving subnetwork. In this embodiment, λ5=100, λ6=1, λ7=1, λ8=1, λ9=10.

[0038] The ability of the semantic preserving subnetwork to preserve global semantic information is transferred to the detail reconstruction subnetwork. Therefore, the total loss function of UKT-GAN is expressed as follows: .

[0039] The following section presents experiments and results analysis of the UKT-GAN used in this invention.

[0040] Experimental Design: To verify the effectiveness of UKT-GAN, this embodiment conducts comparative experiments on two publicly available datasets, SEN1-2 and WHU-OPT-SAR. For the SEN1-2 dataset, this embodiment selects subsets of four typical land cover types: farmland, mountains, buildings, and forests (e.g., ...). Figure 7 (a) is shown to verify the conversion effect of UKT-GAN under different land cover types. For the WHU-OPT-SAR dataset (whose land cover type distribution is shown in [reference]), we can use the following datasets to verify the conversion effect of UKT-GAN under different land cover types. Figure 7 (b) In this embodiment, it is used as supplementary data for cross-sensor verification (the differences from the SEN1-2 dataset are detailed in Table 1), and its original images are uniformly cropped to a standard size of 256×256 to construct registration image pairs suitable for model training and testing, so as to evaluate whether UKT-GAN can still maintain excellent generalization performance and robustness when facing the differences in imaging characteristics of different satellite sensors.

[0041] Table 1: Parameter Comparison Table between SEN1-2 Dataset and WHU-OPT-SAR Dataset In addition, to evaluate the effectiveness of the proposed UKT-GAN, this embodiment uses eight image transformation methods for comparative analysis based on the two datasets mentioned above, namely Pix2pix, CycleGAN, BicycleGAN, CUT, FastCUT, DCLGAN, SimDCL and StegoGAN.

[0042] To comprehensively evaluate the performance of UKT-GAN and the eight conversion methods mentioned above in SAR-to-optical image conversion, this embodiment will evaluate the quality of the generated image from four aspects: global quality, structural similarity, deep feature fidelity, and pixel-level accuracy, based on four evaluation metrics: peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), learned perceptual image patch similarity (LPIPS), and root mean square error (RMSE). This will ensure a comprehensive analysis of the image conversion effect.

[0043] PSNR: This uses mean squared error to determine the difference between two images. The higher the PSNR value, the better the quality of the reconstructed image and the closer it is to the original image. Its calculation formula is as follows: ; in, y and y’ These represent the real optical image and the generated optical image, respectively. MAX This represents the maximum grayscale value in the composite image. MSE This represents the mean square error between images. W and H Indicates the number of rows and columns of the image, ( m , n () indicates the position of a pixel.

[0044] ; ; ; ; μ y and μ y’ express y and y’ The mean, σ y and σ y’ express y and y’ variance l(y,y') , c(y,y') and s(y,y') These represent the brightness, contrast, and structure of the images, respectively. and Set to 1.

[0045] LPIPS: This method extracts image features using a deep convolutional neural network and evaluates the similarity between two images using a weighted L2 distance. A lower L2 distance value indicates greater similarity between the two images. The calculation formula is as follows: ; in,y l and y’ l They represent the evaluation network respectively. y and y’ The image of the first Layer feature map, ω l This represents the weighting parameters between feature maps.

[0046] RMSE: Used to evaluate the difference in pixel values ​​between two images. It is sensitive to subtle changes in images; the lower the RMSE value, the more similar the two images are. Its calculation formula is as follows: ; This embodiment is based on the PyTorch framework and implemented in an NVIDIA RTX 3090 single-card environment. Both the generator and discriminator use... β 1 =0.5 and β 2 The network was optimized using Adam with a learning rate of 0.9, with 200 iterations and a batch size of 1. For the first 100 iterations, both the generator and discriminator were trained with a fixed learning rate of lr = 0.0002, which was then progressively decreased to 0 for the next 100 iterations.

[0047] Experimental Results and Analysis: Quantitative analysis On the SEN1-2 dataset, specifically on four typical land cover types—farmland, mountains, buildings, and forests—the performance of different image transformation methods was compared across four evaluation metrics: PSNR, SSIM, LPIPS, and RMSE. It was observed that UKT-GAN achieved the best results compared to the eight previously mentioned image transformation methods on these four datasets, demonstrating superior performance. This advantage can be attributed to two main aspects: First, UKT-GAN employs a dual-branch training framework based on unidirectional knowledge transfer, integrating the detail reconstruction subnetwork's ability to capture and reconstruct local texture details into the semantic preservation subnetwork. This ensures that the images generated by the semantic preservation subnetwork possess high fidelity in both global semantic information and local texture details. Second, UKT-GAN utilizes a multi-level optimization mechanism to specifically optimize both the detail reconstruction and semantic preservation subnetworks, improving the quality of the images generated by these subnetworks. Furthermore, a comparison of the quantitative results across the four subsets of SEN1-2 reveals that Pix2pix and Bicyclegan also demonstrate superior performance. This is primarily because Pix2pix and Bicyclegan employ a supervised image transformation method, adding image similarity constraints to the generated optical images using real optical images during model training, thereby improving the quality of the network-generated images. In contrast, unsupervised image transformation methods (CycleGAN, BicycleGAN, CUT, FastCUT, DCLGAN, SimDCL, and StegoGAN) typically rely on cycle consistency loss for indirect constraints, which can lead to issues such as blurred details, semantic distortion, or modal confusion in complex scenes, resulting in generally inferior performance in both qualitative and quantitative metrics compared to supervised image transformation methods.

[0048] Qualitative analysis By comparison, it can be found that UKT-GAN can generate optical images that are more similar to the texture structure of real optical images on the ground compared to the above 8 methods.

[0049] Generalization analysis To verify the generalization ability of UKT-GAN under different sensor characteristics and imaging configurations, this embodiment conducted further comparative experiments and analyses based on the WHU-OPT-SAR dataset. UKT-GAN outperformed other comparative methods in four metrics: PSNR, SSIM, LPIPS, and RMAE, indicating that it can still generate optical images with clearer structures and higher overall quality on the WHU-OPT-SAR dataset. Furthermore, the optical images generated by UKT-GAN better preserve texture details similar to real images, with lower generation errors than the eight image transformation methods used for comparison. The generated optical images are also closer to real images at the pixel level. These results collectively demonstrate that UKT-GAN maintains excellent performance and possesses good generalization ability even when faced with data distribution differences caused by different sensor characteristics and imaging configurations.

[0050] ablation experiment To verify the impact of the CBAM and PE-Transformer modules in the detail reconstruction subnetwork and semantic preservation subnetwork of the dual-branch generation framework based on unidirectional knowledge transfer in UKT-GAN, as well as the feature consistency loss and unidirectional consistency loss on the quality of the generated images, this embodiment conducted ablation experiments on the farmland subdataset in the SEN-12 dataset.

[0051] First, to verify the effectiveness of the dual-branch generation framework based on unidirectional knowledge transfer, this embodiment uses a separately trained detail reconstruction subnetwork, a separately trained semantic preservation subnetwork, and a detail reconstruction subnetwork based on unidirectional knowledge transfer (using... L UC(DR) Loss function for knowledge transfer) and semantic preservation sub-network based on unidirectional knowledge transfer (using L UC(SP)Quantitative results were recorded on the farmland subset of the SEN1-2 dataset using a loss function for knowledge transfer. Comparison of these results revealed that, under this one-way knowledge transfer mechanism, the detail reconstruction subnetwork based on one-way knowledge transfer successfully embedded the semantic preservation subnetwork's ability to preserve global semantic information, and the semantic preservation subnetwork also successfully embedded the detail reconstruction subnetwork's ability to generate local texture details, effectively improving the quality of the generated images. However, since the semantic preservation subnetwork based on one-way knowledge transfer has better image conversion performance, this embodiment uses a semantic preservation subnetwork based on one-way knowledge transfer (UKT-GAN) as the core backbone network to ensure optical image output that combines global semantic information consistency with rich local texture details. Furthermore, on the farmland subset of the SEN1-2 dataset, the separately trained semantic preservation subnetwork outperformed all comparative methods across all four metrics, and the separately trained detail reconstruction subnetwork also outperformed all comparative methods in SSIM and LPIPS, achieving second-best results in PSNR and RMSE, demonstrating the effectiveness of the semantic preservation subnetwork and detail reconstruction subnetwork design.

[0052] Secondly, to verify the impact of the CBAM and PE-Transformer modules in the detail reconstruction subnetwork and semantic preservation subnetwork on the quality of the generated images, this embodiment conducted comparative experiments on the separately trained detail reconstruction subnetwork and the separately trained semantic preservation subnetwork (basic semantic preservation subnetwork) using only content loss and generative adversarial loss. The comparison revealed that introducing the CBAM module into the detail reconstruction subnetwork comprehensively improved all four metrics: PSNR, SSIM, LPIPS, and RMSE. This indicates that the attention mechanism, through adaptive weighting of channel and spatial dimensions, effectively enhances the detail reconstruction subnetwork's ability to represent local texture and details. However, integrating the PE-Transformer module into the semantic preservation subnetwork improved the generated optical images in terms of PSNR, SSIM, and RMSE, but performed poorly in terms of LPIPS. This shows that embedding the PE-Transformer module into the semantic preservation network can effectively establish long-range dependencies, significantly improving the global structural rationality and pixel-level accuracy of the generated images, but sacrificing some deep feature fidelity.

[0053] Finally, to improve the fidelity of the semantic preservation network for deep features and the ability of the integrated detail reconstruction subnetwork to capture texture structure details, this embodiment introduces a feature consistency loss based on feature-aware similarity. L feature Compared with one-way consistency loss for knowledge transfer L UC(SP)Experimental results show that adding feature consistency loss to the semantically preserving subnetwork is effective. L feature Subsequently, the semantic preservation subnetwork improved in all four metrics: PSNR, SSIM, LPIPS, and RMSE, with a significant improvement in LPIPS (from 0.4606 to 0.4353), indicating that... L feature By constraining semantics to maintain consistency between the sub-network output and the real optical image in the deep feature space, the fidelity of the generated image's deep features can be effectively enhanced. Further, by introducing unidirectional consistency loss... L UC(SP) Subsequently, the semantically preserving subnetwork improved PSNR, SSIM, and LPIPS metrics, while decreasing RMSE, indicating that the addition of a one-way consistency loss improved performance. L UC(SP) Transferring the ability of the detail reconstruction subnetwork to capture and reconstruct local texture details to the semantic preservation subnetwork also helps to further improve the overall quality of the generated images, proving the effectiveness of unidirectional consistency loss. L UC(SP) The effectiveness of the design.

[0054] In summary, addressing the challenge that existing SAR-to-optical image conversion methods based on generative adversarial networks (GANs) struggle to simultaneously ensure global semantic consistency and local texture detail fidelity using a single generator, this invention proposes a unidirectional knowledge transfer GAN, UKT-GAN, for dual-fidelity SAR-to-optical image conversion. This network comprises a clearly defined sub-network for detail reconstruction and a sub-network for semantic preservation. The detail reconstruction sub-network employs a U-Net generator embedded with a CBAM module, a shallow discriminator, and pixel-level orientation constraints, focusing on reconstructing local texture details. The semantic preservation sub-network uses a ResNet generator embedded with a PE-Transformer module, a deep discriminator, and pixel-level and feature-level orientation constraints to ensure global semantic consistency of the generated image. Furthermore, the reconstruction capability of the detail reconstruction sub-network for local texture details is transferred to the semantic preservation sub-network through a unidirectional consistency loss, enabling the generation of optical images with both global semantic information and local texture details even when only a single semantic preservation sub-network is deployed in the test. To comprehensively evaluate the performance of UKT-GAN, this embodiment also selects eight mainstream image conversion methods as benchmarks. Quantitative evaluation and qualitative analysis were conducted on sub-datasets of four typical land cover types—farmland, mountains, buildings, and forests—from the SEN1-2 dataset, as well as on the WHU-OPT-SAR dataset. Experimental results show that the UKT-GAN proposed in this invention can generate optical images with clearer structures and better overall quality when dealing with different land cover types compared to other methods. Furthermore, UKT-GAN maintains stable conversion performance and demonstrates good generalization ability even when faced with data distribution differences caused by different sensor characteristics and imaging configurations.

[0055] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A SAR-to-optical image dual-fidelity conversion method based on one-way knowledge transfer, characterized in that, A unidirectional knowledge transfer generative adversarial network (GAN) based on a bi-branch generative framework is used to convert a SAR image to an optical image. The unidirectional knowledge transfer GAN includes a detail reconstruction sub-network and a semantic preservation sub-network within the bi-branch generative framework. The SAR-to-optical image double-fidelity conversion method includes: SAR image samples are input into the detail reconstruction subnetwork and the semantic preservation subnetwork respectively to generate two sets of optical images that focus on the local texture structure details of the image and optical images that focus on the global semantic information of the image, respectively. Adversarial discrimination is performed on the two sets of generated optical images to guide the detail reconstruction subnetwork and the semantic preservation subnetwork to focus on the reconstruction of local texture structure details and the preservation of global semantic information, respectively, thereby realizing the basic capability training of the detail reconstruction subnetwork and the semantic preservation subnetwork. Based on the completion of basic capability training of the detail reconstruction subnetwork and the semantic preservation subnetwork, a detail feature transfer channel from the detail reconstruction subnetwork to the semantic preservation subnetwork is constructed using a one-way consistency loss function. This transfers the ability of the detail reconstruction subnetwork to reconstruct local texture structure details to the semantic preservation subnetwork, resulting in a semantic preservation subnetwork based on one-way knowledge transfer. Alternatively, a global semantic transfer channel from the semantic preservation subnetwork to the detail reconstruction subnetwork is constructed using a one-way consistency loss function. This transfers the ability of the semantic preservation subnetwork to preserve global semantic information to the detail reconstruction subnetwork, resulting in a detail reconstruction subnetwork based on one-way knowledge transfer. The SAR image to be processed is converted into the corresponding optical image by using a semantic preservation subnetwork based on one-way knowledge transfer or a detail reconstruction subnetwork based on one-way knowledge transfer.

2. The SAR to optical image dual-fidelity conversion method based on one-way knowledge transfer of claim 1, wherein, The detailed reconstruction subnetwork includes a generator for image transformation and a discriminator for determining the authenticity of the generated image; The generator of the detailed reconstruction subnetwork uses the Unet network. In the Unet network, the layers of the encoder and the layers of the decoder are connected in a skip connection to perform multi-scale fusion of the feature maps in the encoder and the feature maps in the decoder. The Unet network adopts a progressive feature expansion strategy, which gradually increases the channel dimension of the features while keeping the spatial scale of the feature maps unchanged. In the Unet network, a CBAM module is embedded in the innermost layer of the encoder and decoder to embed the channel attention mechanism and the spatial attention mechanism into the innermost layer of the encoder and decoder, and the weights of each feature channel are automatically learned through the channel attention mechanism. The generator and discriminator of the detailed reconstruction subnetwork adopt the PatchGAN architecture, with the number of convolutional layers in the PatchGAN architecture set to 2.

3. The SAR to optical image dual-fidelity conversion method based on one-way knowledge distillation of claim 1, wherein, The semantic preservation subnetwork includes a generator for image transformation and a discriminator for determining the authenticity of the generated image; The generator of the semantic preservation subnetwork adopts a ResNet network. The bottleneck layer of the ResNet network embeds a Transformer encoding module with positional encoding. The Transformer encoding module is used to preserve the key spatial structure information of the feature map by introducing a learnable positional encoding mechanism, and to use a multi-head attention mechanism to compute attention weights in parallel in multiple independent representation subspaces to simultaneously capture long-range dependencies at different scales. Residual modules are embedded in the downsampling and upsampling layers of the encoder and decoder in the ResNet network; The discriminator of the semantic preservation subnetwork adopts the PatchGAN architecture, and the number of convolutional layers in the PatchGAN architecture is set to 4.

4. The SAR to optical image dual-fidelity conversion method based on one-way knowledge transfer of claim 1, wherein, Both the detail reconstruction subnetwork and the semantic preservation subnetwork include a generator for image transformation and a discriminator for determining the authenticity of the generated image; The total loss function of the unidirectional knowledge transfer generative adversarial network is expressed as follows: ； In the formula, is the total loss function of the one-way knowledge transfer generative adversarial network, , , , and is the loss function proportion weight, is the image similarity loss function of the generator in the detail reconstruction subnetwork, is the generative adversarial loss function of the discriminator in the detail reconstruction subnetwork, is the image similarity loss function of the generator in the semantic preservation subnetwork, is the generative adversarial loss function of the discriminator in the semantic preservation subnetwork, is the one-way consistency loss function.

5. The SAR-to-optical image double-fidelity conversion method based on one-way knowledge transfer according to claim 4, characterized in that, The image similarity loss function of the detailed reconstruction subnetwork is expressed as: ； In the formula, It is a norm of 1. For real SAR images, For true optical images, The optical image generated by the detailed reconstruction subnetwork.

6. The SAR-to-optical image double-fidelity conversion method based on one-way knowledge transfer according to claim 4, characterized in that, The generative adversarial loss function of the detailed reconstruction subnetwork is expressed as: ； In the formula, For the expected operation, The discriminator reconstructs the subnetwork for the aforementioned details. For real SAR images, For true optical images, The optical image generated by the detailed reconstruction subnetwork.

7. The SAR-to-optical image double-fidelity conversion method based on one-way knowledge transfer according to claim 4, characterized in that, The image similarity loss function of the semantically preserving subnetwork is expressed as: ； In the formula, and The weights are the proportions of the loss function. Let the content consistency loss function be... The feature consistency loss is used; where: ； ； In the formula, For real SAR images, For true optical images, The optical image generated for the semantically preserving subnetwork. It is a norm of 1. It is a 2-norm. The weighting parameters between feature maps, To obtain the true optical image extracted using VGG in the first... l On the layer feature map Features at a spatial location The optical image generated by the semantic-preserving subnetwork extracted using VGG is in the first... l On the layer feature map Features at a spatial location and For the first l The height and width of the layer feature map, and These are the spatial coordinates of the feature map.

8. The SAR-to-optical image double-fidelity conversion method based on one-way knowledge transfer according to claim 4, characterized in that, The generative adversarial loss function of the semantic-preserving subnetwork is expressed as: ； In the formula, For the expected operation, For the discriminator of the semantically preserving subnetwork, For real SAR images, For true optical images, Optical images generated for the semantically preserving subnetwork.

9. The SAR-to-optical image double-fidelity conversion method based on one-way knowledge transfer according to claim 4, characterized in that, When the ability of the detail reconstruction subnetwork to reconstruct local texture structure details is transferred to the semantic preservation subnetwork, the one-way consistency loss function is expressed as: ； In the formula, The one-way consistency loss function from the detailed reconstruction subnetwork to the semantically preserving subnetwork is given. To stop gradient operations and prevent model parameters from being propagated back, For real SAR images, The optical image generated by the detailed reconstruction subnetwork. The optical image generated for the semantically preserving subnetwork. It is a norm of 1. An optical image generated to stop gradient operations on the detailed reconstruction subnetwork and to prevent model parameters from being backpropagated.

10. The SAR-to-optical image double-fidelity conversion method based on one-way knowledge transfer according to claim 4, characterized in that, When the ability of the semantic preservation subnetwork to preserve global semantic information is transferred to the detail reconstruction subnetwork, the one-way consistency loss function is expressed as: ； In the formula, The one-way consistency loss function from the semantically preserving subnetwork to the detail reconstruction subnetwork is given. and The weights are the proportions of the loss function. To stop gradient operations and prevent model parameters from being propagated back, For real SAR images, The optical image generated by the detailed reconstruction subnetwork. The optical image generated for the semantically preserving subnetwork. It is a norm of 1. It is a 2-norm. The weighting parameters between feature maps, The optical image generated by the VGG-extracted detail reconstruction subnetwork is in the first... l On the layer feature map Features at a spatial location For the first l The height of the layer feature map, For the first l Width of layer feature maps and For the spatial coordinates of the feature mapping, An optical image generated to stop gradient operations on the semantically preserving subnetwork and without backpropagating model parameters. The optical image generated to stop gradient operations on the semantically preserving subnetwork and to prevent model parameters from backpropagating is in the first... l On the layer feature map Features at a spatial location.

Images