SAR (Synthetic Aperture Radar)-to-optical image conversion method based on segmentation semantic guide contrast learning

Abstract

The invention discloses a method for converting an SAR (Synthetic Aperture Radar) into an optical image based on segmentation semantic guide contrast learning, which is characterized in that semantic features of the SAR and the optical image are extracted by introducing a pre-trained segmentation network, explicit alignment of a cross-modal semantic space is realized, and semantic consistency and structural fidelity of a generated image are improved; meanwhile, a block-level contrast loss mechanism based on semantic category constraint is provided, while spatial information integrity is kept, aggregation expression of similar ground feature region features is enhanced, and the modeling ability of multi-instance targets in a complex remote sensing scene is improved. Besides, segmentation cycle loss is proposed, the generated optical image and the original SAR image are input into the semantic segmentation model again, semantic tags of the two images are output, consistency constraint is carried out, a cross-modal closed-loop optimization mechanism is constructed, and therefore semantic separability and structural reversibility of the generated image are further enhanced. According to the method, multi-layer semantic guidance, comparison constraint and segmentation loop constraint are jointly introduced in a training stage, and multi-dimensional perception of a generator on spatial distribution, surface feature boundary and category semantics in a remote sensing image is optimized; in the reasoning stage, an auxiliary module does not need to be introduced, and an optical image with a clear structure and definite semantics can be efficiently output. According to the method, the visual expression ability of the SAR image and the adaptability of a downstream remote sensing task can be remarkably improved, and the method has good generalization ability and engineering practical prospects.

Description

Technical Field

[0001] This invention relates to a SAR-to-optical image conversion method based on segmentation semantic-guided contrastive learning, belonging to multi-source remote sensing image processing technology. Background Technology

[0002] With the rapid development of remote sensing technology, an increasing number of spaceborne and airborne remote sensing sensors are being deployed in surface observation missions, providing rich geospatial information in fields such as urban planning, resource surveys, disaster monitoring, and ecological protection. Among these, optical remote sensing images, due to their intuitive and clear visual representation, have become the mainstream data source for remote sensing analysis and ground feature identification. However, optical sensors rely on natural light and are easily affected by severe weather and diurnal variations, resulting in unstable data acquisition. The acquisition quality is particularly poor under cloudy, foggy, and nighttime conditions, limiting their application scope in continuous monitoring scenarios.

[0003] Synthetic Aperture Radar (SAR), as an active remote sensing system, possesses all-weather, all-time imaging capabilities, can stably acquire surface information under complex climatic conditions, and has strong textural representation capabilities, making it an important supplement to optical imagery. Especially in high-risk scenarios such as extreme weather and wartime reconnaissance, SAR imagery is widely used in tasks such as target detection and surface change analysis due to its advantages of strong penetration and short acquisition cycle. However, due to its microwave scattering-based imaging mechanism, SAR images typically contain a large amount of speckle noise, lack color and texture consistency, and have poor visual readability, resulting in poor performance of traditional computer vision algorithms and manual recognition. Therefore, converting SAR images into optical images with natural visual expressiveness can not only improve their interpretability and visualization effects but also effectively improve the performance of downstream tasks such as target detection, image segmentation, and change detection.

[0004] However, existing methods still face three major challenges: 1. Unstable training process and limited result quality: Although Generative Adversarial Networks (GANs) have been widely used in image transformation, they are prone to training instability and pattern collapse when faced with remote sensing images with complex ground structures and dense semantics. The quality of generated images often exhibits defects such as blurred details and unclear edges. 2. Difficulty in ensuring semantic consistency: Traditional GAN ​​models lack explicit semantic guidance mechanisms, making it difficult to ensure accurate alignment between source and generated images at the semantic level. This is especially true in remote sensing scenes with mixed and densely distributed multiple categories, which can easily lead to semantic drift or misidentification. 3. Lack of structural consistency in contrastive learning strategies: Although contrastive learning has been widely introduced to enhance feature discriminativeness and structural preservation capabilities, current methods mostly construct positive and negative sample relationships based on location matching, ignoring the structural consistency of a large number of similar but displaced instances (such as multiple houses or multiple farmlands) in remote sensing images. This can easily lead to misclassification of semantically consistent regions as negative samples, reducing the model's semantic generalization ability and transformation accuracy. Summary of the Invention

[0005] Purpose of the invention: To overcome the problems of unstable training, poor semantic consistency, and insufficient structure preservation ability in existing remote sensing image conversion methods, this invention provides a SAR to optical image conversion method based on segmentation semantic-guided contrastive learning. It fully integrates segmentation semantic features and contrastive learning mechanisms, optimizes the positive and negative sample construction strategy by introducing semantic segmentation guidance information, enhances the model's ability to model the consistency of similar ground structures, and combines segmentation cyclic loss constraints to effectively improve the semantic accuracy and structural fidelity of cross-modal conversion of remote sensing images under the condition of no paired samples.

[0006] Technical solution: To achieve the above objectives, the technical solution adopted by this invention is as follows:

[0007] A SAR-to-optical image conversion method based on segmentation semantic guided contrastive learning converts real SAR images into generated optical images by constructing a conversion network based on segmentation semantic guided contrastive learning; the conversion network includes a generation network, a trained semantic feature extraction network, and a segmentation contrastive learning network.

[0008] The generative network includes two generators and their respective discriminators. The generators learn cross-modal mapping relationships in an end-to-end training framework, and the discriminators improve the quality of generated image details. The two generators are denoted as Generator I and Generator II, respectively. Generator I takes a SAR image as input and outputs an optical image, while Generator II takes an optical image as input and outputs a SAR image. The two discriminators are denoted as Discriminator I and Discriminator II, respectively. A real SAR image is first processed by Generator I to obtain a generated optical image, and then by Generator II to obtain a reconstructed SAR image. Discriminator I calculates the authenticity score of the generated optical image, which is used to calculate the adversarial loss between the generated optical image and the real optical image, and simultaneously calculates the cyclic error between the real SAR image and the reconstructed SAR image. Similarly, a real optical image is first processed by Generator II to obtain a generated SAR image, and then by Generator I to obtain a reconstructed optical image. Discriminator II calculates the authenticity score of the generated SAR image, which is used to calculate the adversarial loss between the generated SAR image and the real SAR image, and simultaneously calculates the cyclic error between the real optical image and the reconstructed optical image.

[0009] After training, the semantic feature extraction network uses a frozen SAR image segmentation semantic feature extractor and an optical image segmentation semantic feature extractor to participate in the training and testing of the transformation network. The input of the SAR image segmentation semantic feature extractor is a SAR image and the output is SAR semantic features. The input of the optical image segmentation semantic feature extractor is an optical image and the output is optical semantic features.

[0010] The segmentation contrastive learning network includes two multilayer encoders and their respective perceptron networks. The two multilayer encoders correspond to the two generators of the generator network, and are denoted as Multilayer Encoder I and Multilayer Encoder II, respectively. The two perceptron networks are denoted as Perceptron Network I and Perceptron Network II, respectively. Each multilayer encoder consists of M convolutional layers corresponding to M specific feature extraction layers in the feedforward network of the corresponding generator, used to capture the structural and texture information of the input image at multiple levels, obtaining M layers of low-dimensional image features. The perceptron network is a two-layer perceptron structure used to project the M layers of low-dimensional image features extracted by the multilayer encoders into the embedding space required for contrastive learning. A set of feature blocks is obtained; the real SAR image and the generated optical image are first passed through a multi-layer encoder I to obtain low-dimensional real SAR features and low-dimensional generated optical features, respectively, and then passed through a perceptron network I to obtain real SAR feature blocks and generated optical feature blocks, respectively; the real optical image and the generated SAR image are first passed through a multi-layer encoder II to obtain low-dimensional real optical features and low-dimensional generated SAR features, respectively, and then passed through a perceptron network II to obtain real optical feature blocks and generated SAR feature blocks, respectively; the generated optical feature blocks guide the real SAR feature blocks to perform comparative learning, and the generated SAR feature blocks guide the real optical feature blocks to perform comparative learning, where M is a positive integer.

[0011] Specifically, the execution process of the generation network includes: converting real SAR images into real SAR images. sar and real optical images opt The input is fed into the generator network. First, the real SAR image is... sar and real optical images opt The generated optical image is obtained by passing it through generator I and generator II respectively. opt and generating SAR images fake sar Then, generate a fake SAR image. sar and generating optical images fake opt The image is then passed through generator I and generator II again to obtain the reconstructed optical image rec. opt and reconstructed SAR image rec sar The optical image generated using discriminator I is fake. opt The authenticity score is calculated using Discriminator II to generate fake SAR images. sar The score for authenticity.

[0012] Specifically, the execution process of the semantic feature extraction network includes:

[0013] Reconstruct the SAR image rec sar and reconstructing optical images opt The input is fed into a semantic feature extraction network to reconstruct the SAR image rec. sar The SAR features are reconstructed after SAR image segmentation and semantic feature extraction. Reconstructing optical images opt Reconstructed optical features are obtained after optical image segmentation semantic feature extraction.

[0014] Real SAR images sar and generating optical images fake opt The input is fed into the semantic feature extraction network to obtain the real SAR image. sar The true SAR features are obtained by the SAR image segmentation semantic feature extractor. Generating optical images fake opt The generated optical features are obtained through an optical image segmentation semantic feature extractor.

[0015] The generated SAR image is fake. sar and real optical images opt Input into a semantic feature extraction network to generate a fake SAR image. sarSAR features are generated after SAR image segmentation and semantic feature extraction. Real optical images opt The true optical features are obtained through an optical image segmentation semantic feature extractor.

[0016] Specifically, the execution process of the segmentation contrast learning network includes:

[0017] Real SAR images sar Generating SAR images (fake) sar Real optical images opt and generating optical images fake opt Input into the segmentation and contrast learning network; real SAR image sar and generating optical images fake opt First, the low-dimensional true SAR features of feature layer m are obtained by passing the features through multiple encoders I to the corresponding generator I. and low-dimensional generated optical features The actual SAR feature blocks are then obtained through the perceptron network I. and generating optical feature blocks Meanwhile, real optical images opt and generating SAR images fake sar First, the low-dimensional real optical features of the specific feature extraction layer m in the corresponding generator II are obtained by passing the features through a multi-layer encoder II. and low-dimensional generated SAR features The actual optical feature blocks are then obtained through a perceptron network II. and generating SAR feature blocks m = 1, 2, ..., M, where M is the total number of specific feature extraction layers in the generator feedforward network, n = 1, 2, ..., N, where N is the dimension of the low-dimensional feature, and the total number of real SAR feature blocks, generated SAR feature blocks, real optical feature blocks and generated optical feature blocks are all M × N.

[0018] From real SAR characteristics Select the real SAR feature block Regions with the same location are used as real SAR tag blocks Simultaneously generating SAR features Selecting and generating SAR feature blocks Regions with the same location are used as SAR tag blocks for generation. From real optical characteristics Select the real optical feature block Areas with the same location are used as real optical tag blocks. Simultaneously generating optical features Selecting and generating optical feature blocks Regions with the same location are used as the generating optical tag blocks.

[0019] For a specific feature extraction layer m, optical feature blocks are generated. Guided real SAR feature blocks To perform comparative learning, the first step is to select and generate optical feature blocks. As a reference, then zoom in on the actual SAR feature block. For the other N-1 real SAR feature blocks If the real SAR tag block and generating optical tag blocks If the categories are the same, then zoom in to the real SAR feature block. Otherwise, the actual distance of the SAR feature block k∈N,k≠n;

[0020] For a specific feature extraction layer m, SAR feature blocks are generated. Guided real optical feature blocks To conduct comparative learning, the first step is to select and generate SAR feature blocks. As a reference, then zoom in on the actual optical feature block. For the other N-1 real optical feature blocks If a real optical tag block and generating SAR tag blocks If the categories are the same, then zoom in to the actual optical feature blocks. Otherwise, pull the real far optical feature block k∈N,k≠n.

[0021] Specifically, the conversion network also includes a loss value calculation unit to calculate the real SAR image. sar and generating SAR images fake sar The adversarial loss value between them is used to calculate the real optical image. opt and generating optical images fake opt Adversarial loss values ​​between them; calculating real SAR images sar and reconstructed SAR image rec sar The cyclic loss value between them is used to calculate the real optical image. opt and reconstructing optical images opt Cyclic loss values ​​between; calculation of true SAR features and reconstructing SAR features The segmentation cycle loss value between them is used to calculate the true optical features. and reconstruct optical features The segmentation cycle loss value between; calculate the true SAR feature block. The segmentation-guided contrastive learning loss value is used to calculate the true optical feature blocks. The segmentation guides the contrastive learning loss value; the total loss is obtained by weighted summation of all loss values. When the total loss converges, the training of the transformation network is complete.

[0022] Specifically, the semantic feature extraction network includes a SAR image segmentation semantic feature extractor and an optical image segmentation semantic feature extractor with consistent structures. The SAR image segmentation semantic feature extractor uses real SAR images and semantic segmentation labels to extract multi-scale, multi-category SAR semantic features through supervised learning. The optical image segmentation semantic feature extractor uses real optical images and semantic segmentation labels to extract multi-scale, multi-category optical semantic features through supervised learning. Both segmentation semantic feature extractors are trained and constructed based on the DeepLabV3-ResNet network structure, which includes a backbone feature extraction subnetwork and a semantic prediction subnetwork. The backbone feature extraction subnetwork is pre-trained. The ResNet network is used as a feature extractor to extract multi-scale semantic information from the input image. The semantic prediction sub-network includes a Spatial Pyramid Pooling (ASPP) module to enhance the perception of objects at different scales. During the training of the semantic feature extraction network, the input image is first preprocessed through normalization, tensor transformation, and other steps before being fed into the corresponding DeepLabV3-ResNet network for feature extraction. The cross-entropy loss function is used in conjunction with the StepLR scheduler to adjust the learning rate during the training process, thereby achieving stable optimization of the semantic feature extraction network. At the same time, multi-GPU parallel training is used to accelerate the training process, and the convergence training of the semantic feature extraction network is completed on the test set to obtain a segmentation semantic feature extraction capability with generalization ability.

[0023] Specifically, in the segmentation semantic feature extractor of the semantic feature extraction network, the backbone feature extraction subnetwork uses a pre-trained ResNet50 as the feature extractor. ResNet50 has a five-layer structure: the first layer (Conv1) includes a convolutional layer and a pooling layer; the second to fifth layers (Conv2 to Conv5) each include a residual module. These four residual modules achieve step-by-step feature extraction from low-level texture to high-level semantics. This structure effectively preserves edge information and spatial layout in the input image while possessing strong multi-scale structural modeling capabilities. The dilated spatial pyramid pooling module constructs a dilation rate based on the high-level semantic features output by the backbone feature extraction subnetwork. The network employs 3×3 dilated convolution branches with 6, 12, and 18 dilation, as well as a 1×1 convolution branch and an image-level global average pooling branch. The outputs of each branch are concatenated and then fused using 1×1 convolution to generate multi-scale semantic context representations. This enhances the semantic feature extraction network's ability to perceive complex ground objects (such as roads, houses, farmland, and water bodies). During the training of the semantic feature extraction network, the training and test sets are evaluated separately, using the average intersection-union ratio (mIoU) as the evaluation criterion. The parameters of the semantic feature extraction network are saved when the mIoU on the test set is optimal, ultimately resulting in a segmentation semantic feature extractor with good generalization ability in remote sensing image scenarios.

[0024] Specifically, in the generation network, both generators adopt a symmetric ResNetGenerator structure based on the ResNet architecture to realize cross-modal bidirectional mapping between SAR images and optical images; each generator consists of three parts: an encoder, residual blocks, and a decoder. This structure of generator has strong image structure modeling and cross-modal conversion capabilities.

[0025] The encoder first performs multi-scale feature extraction and dimensionality compression on the input image through a set of downsampling convolutional layers to enhance the generator's ability to express spatial features such as edge texture and structural contours. Then, it obtains encoded features through instance normalization and ReLU activation functions to improve the generator's training stability and nonlinear modeling ability.

[0026] The residual block module comprises nine consecutively stacked residual blocks, each consisting of two 3×3 convolutional layers connected using an identity mapping residual connection. The residual module alleviates the gradient vanishing problem in deep networks and enhances the structure preservation and nonlinear representation capabilities during cross-modal mapping. Furthermore, the residual block module enhances the modeling of high-level semantic features of the input image, helping to maintain the geometric layout and ground feature relationships of the input image.

[0027] The decoder gradually restores the image resolution through transposed convolution (upsampling) and guides image generation or image reconstruction by combining encoded features; at the same time, it uses the tanh activation function to output normalized results, thereby achieving high-fidelity restoration of structural and texture information.

[0028] Specifically, in the generator network, both discriminators are implemented using the NLayerDiscriminator based on the PatchGAN architecture, including a 3-layer backbone convolutional module, which aims to perform parallel real and fake discrimination of multiple local regions in the input image. The discriminator divides the input image into local regions through a sliding window mechanism and performs real and fake discrimination prediction for each local region, so as to enhance the generator's ability to model image details, textures and edge regions, and improve the quality performance of the generated image in terms of realism, structural consistency and spatial distribution.

[0029] Specifically, in the segmentation and contrast learning network, the multi-layer encoder consists of five convolutional layers from shallow to deep, corresponding to specific feature extraction layers 0, 4, 8, 12, and 16 in the generator's feedforward network. These layers extract multi-scale features from the input image, including local texture, edge contours, and semantic regions, enabling multi-level capture of the input image's structure and texture information, ultimately yielding low-dimensional image features for each convolutional layer. The perceptron network employs a two-layer perceptron structure, including two linear transformation units (LinearLayer), a normalization layer (e.g., BatchNorm), and a non-linear activation function. (e.g., ReLU) is used to compress and embed the 5 layers of low-dimensional image features extracted by the encoder: First, the first-layer linear transformation unit performs a linear transformation on the input low-dimensional image features, mapping them to an intermediate latent space to initially construct a discriminative embedding; then, the feature distribution of the initially constructed discriminative embedding is normalized by a normalization layer to alleviate the gradient instability problem during training and improve the generalization ability of the segmentation contrastive learning network, obtaining an intermediate representation; next, the second-layer linear transformation unit further maps the intermediate representation to the embedding space required for the final contrastive learning, outputting 5 feature blocks with a dimension of 256.

[0030] Therefore, after inputting the low-dimensional image features into the corresponding perceptron network, a set of feature blocks with a shape of 512×256 will be obtained. The first number 512 is calculated by batchsize×num_patches=2×256, and the second number 256 is the dimension of the low-dimensional image features. The size of each feature block is 1 pixel. Thus, each low-dimensional image feature can be processed into 256 feature blocks through the perceptron network.

[0031] By generating optical feature blocks Guided real SAR feature blocks During comparative learning, all real SAR feature blocks and generated optical feature blocks in a specific feature extraction layer m are first transformed into a 2×256×256 shape. The first '2' represents the batch size, the second '256' represents 256 feature blocks, and the third '256' represents the dimension of each feature block. The selected generated optical feature blocks... This is called an anchor point, and the corresponding real SAR feature block. These are called positive samples. Feature blocks of the same category as the anchor point's location are found from the other 255 real SAR feature blocks, based on the real SAR label blocks. and generating optical tag blocks Whether the values ​​are the same determines whether the categories are the same. Feature blocks of the same category are called positive samples, and feature blocks of different categories are called negative samples. When calculating the segmentation-guided contrastive learning loss value, the anchor point is brought closer to the positive sample and moved further away from the negative sample; k∈N, k≠n.

[0032] By generating SAR feature blocks Guided real optical feature blocks The process of comparative learning is similar.

[0033] Beneficial Effects: The SAR-to-optical image conversion method based on segmentation semantic-guided contrastive learning provided by this invention fully integrates segmentation semantic features and contrastive learning mechanisms. By introducing semantic segmentation guidance information to optimize the positive and negative sample construction strategy, it can enhance the model's ability to model the consistency of similar ground structures. At the same time, combined with segmentation cyclic loss constraints, it can effectively improve the semantic accuracy and structural fidelity of cross-modal conversion of remote sensing images. The method of this invention can accurately capture the semantic and texture correspondence between cross-modal images under the condition of no paired samples, significantly improving the image conversion quality and stability. It has higher accuracy and generalization power for structure preservation and semantic alignment in remote sensing image conversion scenarios. Attached Figure Description

[0034] Figure 1 This is a flowchart illustrating the implementation of the method of the present invention;

[0035] Figure 2 The images consist of pairs of raw SAR images and raw optical images, along with corresponding semantic segmentation labels.

[0036] Figure 3 The conversion effect of using a trained conversion network to convert SAR images into optical images;

[0037] Figure 4 A structural block diagram of the apparatus for implementing the method of the present invention. Detailed Implementation

[0038] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0039] like Figure 1 The diagram shown is a schematic of the implementation process of a SAR to optical image conversion method based on segmentation semantic guided contrastive learning. The method of the present invention will be further explained below with reference to the specific implementation steps.

[0040] Step 1: Acquire multi-source remote sensing images

[0041] Multi-source remote sensing images include raw optical images, raw synthetic aperture radar images (raw SAR images), and semantic segmentation labels. The raw SAR images and raw optical images are cropped and standardized to obtain a real SAR image of a predetermined size. sar and real optical images opt This facilitates subsequent processing. For example... Figure 2 The image shown is the original SAR image, the original optical image, and the semantic segmentation labels.

[0042] Step 2: Divide the training set and the test set

[0043] Real SAR images sar and real optical images opt The dataset was divided into training and testing sets in a 7:3 ratio; during the training phase, real SAR images from the training set were used. sar and real optical images opt Training is performed; after training, real SAR images from the test set are used. sar and real optical images opt Conduct the test.

[0044] Step 3: Construct a transformation network based on segmentation semantic-guided contrastive learning

[0045] A segmentation-based semantic-guided contrastive learning-based conversion network is used to convert real SAR images into generated optical images. It includes a generative network, a trained semantic feature extraction network, and a segmentation-contrast learning network.

[0046] 3.1 Structure of Generative Networks

[0047] The generative network includes two generators and their respective discriminators; the generators are used to learn cross-modal mapping relationships in an end-to-end training framework, and the discriminators are used to improve the quality of image detail generation; the two generators are denoted as Generator I and Generator II, respectively, Generator I takes an optical image as input and outputs an optical image, and Generator II takes an optical image as input and outputs a SAR image; the two discriminators are denoted as Discriminator I and Discriminator II, respectively.

[0048] 3.1.1 Generator Structure

[0049] The generator adopts a symmetric ResNetGenerator structure based on the ResNet architecture to realize cross-modal bidirectional mapping between SAR images and optical images. The generator consists of three parts: an encoder, a residual block module, and a decoder. This generator structure has strong image structure modeling and cross-modal conversion capabilities.

[0050] The encoder first performs multi-scale feature extraction and dimensionality compression on the input image through a set of downsampling convolutional layers to enhance the generator's ability to express spatial features such as edge texture and structural contours. Then, it obtains encoded features through instance normalization layers and ReLU activation functions, thereby improving the generator's training stability and nonlinear modeling ability.

[0051] The residual block module comprises nine consecutively stacked residual blocks, each consisting of two 3×3 convolutional layers connected using an identity mapping residual connection. The residual module alleviates the gradient vanishing problem in deep networks and enhances the structure preservation and nonlinear expression capabilities during cross-modal mapping. Furthermore, the residual block module enhances the modeling of high-level semantic features of the input image, which helps maintain the geometric layout and ground feature relationships of the input image.

[0052] The decoder gradually restores the image resolution through transposed convolution (upsampling) and guides image generation or image reconstruction by combining encoded features; at the same time, it uses the tanh activation function to output normalized results, thereby achieving high-fidelity restoration of structural and texture information.

[0053] 3.1.2 Structure of the discriminator

[0054] The discriminator is implemented using the NLayerDiscriminator based on the PatchGAN architecture, which includes a 3-layer backbone convolution module. It is designed to perform parallel real and fake discrimination on multiple local regions in the input image. The discriminator divides the input image into local regions through a sliding window mechanism and performs real and fake discrimination prediction on each 70×70 local region to enhance the generator's ability to model image details, textures and edge regions.

[0055] The discriminator's input is either a generated SAR image or a generated optical image. The first-layer backbone convolutional module takes a single-channel input and maps it to 64 channels through a 4×4 convolutional kernel with a stride of 2 and padding of 1. It then uses the LeakyReLU activation function for non-linear transformation. The second and third-layer backbone convolutional modules have the same structure, both using a 4×4 convolutional kernel with a stride of 2 or 1 (depending on the network depth). They employ a BatchNorm normalization layer and the LeakyReLU activation function to extract more abstract regional texture and structural information layer by layer, with the number of channels increasing layer by layer (e.g., 128, 256). The discriminator's final output is a two-dimensional discrimination probability map. Its spatial dimension has low consistency with the input image, but each pixel value represents the probability that the corresponding local region (70×70 local region) in the input image is a real image.

[0056] 3.2 Structure of Semantic Feature Extraction Network

[0057] The semantic feature extraction network includes a structurally consistent SAR image segmentation semantic feature extractor and an optical image segmentation semantic feature extractor. The SAR image segmentation semantic feature extractor uses real SAR images and semantic segmentation labels to extract multi-scale, multi-category SAR semantic features through supervised learning. The optical image segmentation semantic feature extractor uses real optical images and semantic segmentation labels to extract multi-scale, multi-category optical semantic features through supervised learning. Both segmentation semantic feature extractors are trained and constructed based on the DeepLabV3-ResNet network structure, which includes a backbone feature extraction subnetwork and a semantic prediction subnetwork.

[0058] The backbone feature extraction subnetwork uses a pre-trained ResNet50 as the feature extractor to extract multi-scale semantic information from the input image. The ResNet50 has a five-layer structure. The first layer, Conv1, includes a convolutional layer and a pooling layer. The second layer, Conv2, to the fifth layer, Conv5, each include a residual module. Through the four residual modules, feature extraction is achieved stepwise from the low-level texture to the high-level semantics. This structure can effectively preserve the edge information and spatial layout in the input image, while also having a strong multi-scale structural modeling capability.

[0059] The semantic prediction subnetwork includes a Spatial Pyramid Pooling (ASPP) module to enhance the perception of objects at different scales. Based on the high-level semantic features output by the backbone feature extraction subnetwork, the SPPP module constructs 3×3 dilated convolution branches with dilation rates of 6, 12, and 18, as well as 1×1 convolution branches and image-level global average pooling branches. The outputs of each branch are concatenated and then fused using 1×1 convolutions to generate multi-scale semantic context representations, thereby enhancing the semantic feature extraction network's perception of complex ground objects (such as roads, houses, farmland, and water bodies).

[0060] During the training of the semantic feature extraction network, the input image and its corresponding semantic segmentation label are first processed by Normalize and ToTensor to obtain a standard tensor, which is then sequentially input into the backbone feature extraction subnetwork and the dilated spatial pyramid pooling module. The cross-entropy loss function is used and ignore_index=255 is set to filter invalid semantic segmentation labels. The Adam optimizer is used in conjunction with the StepLR scheduler to dynamically adjust the learning rate during the training process. At the same time, multi-GPU parallelism is enabled, and DataLoader is used for efficient data loading and batch training. The training set and the test set are evaluated separately, and the average intersection-union ratio (mIoU) is used as the evaluation standard. When the average intersection-union ratio (mIoU) on the test set is optimal, the parameters of the semantic feature extraction network are saved. Finally, a segmentation semantic feature extractor with good generalization ability in remote sensing image scenarios is obtained.

[0061] After the semantic feature extraction network is trained, a frozen SAR image segmentation semantic feature extractor and an optical image segmentation semantic feature extractor are used to participate in the training and testing of the transformation network. The input of the SAR image segmentation semantic feature extractor is the SAR image and the output is the SAR semantic features. The input of the optical image segmentation semantic feature extractor is the optical image and the output is the optical semantic features.

[0062] 3.3 Structure of Segmentation-Contrast Learning Network

[0063] The segmentation contrast learning network includes two multilayer encoders and their respective perceptron networks. The two multilayer encoders correspond to the two generators of the generator network, and are respectively denoted as multilayer encoder I and multilayer encoder II. The two perceptron networks are respectively denoted as perceptron network I and perceptron network II.

[0064] The multilayer encoder consists of M=5 convolutional layers from shallow to deep, corresponding to specific feature extraction layers 0, 4, 8, 12, and 16 in the generator feedforward network, respectively. It can extract multi-scale feature information such as local texture, edge contours, and semantic regions of the input image layer by layer, and is used to capture the structural and texture information of the input image at multiple levels, and finally obtain M layers of low-dimensional image features.

[0065] The perceptron network employs a two-layer perceptron structure, comprising two linear transformation units, a normalization layer, and a nonlinear activation function. This structure is used to compress and embed the M-layer low-dimensional image features extracted by the encoder. First, the first-layer linear transformation unit performs a linear transformation on the input low-dimensional image features, mapping them to an intermediate latent space to initially construct a discriminative embedding. Then, the normalization layer normalizes the feature distribution of the initially constructed discriminative embedding, mitigating gradient instability during training and improving the generalization ability of the segmentation contrastive learning network, resulting in an intermediate representation. Next, the second-layer linear transformation unit further maps the intermediate representation to the embedding space required for the final contrastive learning, outputting five feature blocks with a dimension of 256.

[0066] Therefore, after inputting the low-dimensional image features into the corresponding perceptron network, a set of feature blocks with a shape of 512×256 will be obtained. The first number 512 is calculated by batchsize×num_patches=2×256, and the second number 256 is the dimension of the low-dimensional image features. The size of each feature block is 1 pixel. Therefore, each low-dimensional image feature can obtain 256 feature blocks.

[0067] 3.4 Execution Process of Transformation Network Based on Segmentation Semantic-Guided Contrastive Learning

[0068] Real SAR images sar and real optical images opt The input is fed into the generator network. First, the real SAR image is... sar and real optical images opt The generated optical image is obtained by passing it through generator I and generator II respectively. opt and generating SAR images fake sar Then, generate a fake SAR image. sar and generating optical images fake opt The image is then passed through generator I and generator II again to obtain the reconstructed optical image rec. opt and reconstructed SAR image rec sar The optical image generated using discriminator I is fake. opt The authenticity score is calculated using Discriminator II to generate fake SAR images. sarThe score for authenticity.

[0069] Reconstruct the SAR image rec sar and reconstructing optical images opt The input is fed into a semantic feature extraction network to reconstruct the SAR image rec. sar The SAR features are reconstructed after SAR image segmentation and semantic feature extraction. Reconstructing optical images opt Reconstructed optical features are obtained after optical image segmentation semantic feature extraction.

[0070] Real SAR images sar and generating optical images fake opt The input is fed into the semantic feature extraction network to obtain the real SAR image. sar The true SAR features are obtained by the SAR image segmentation semantic feature extractor. Generating optical images fake opt The generated optical features are obtained through an optical image segmentation semantic feature extractor.

[0071] The generated SAR image is fake. sar and real optical images opt Input into a semantic feature extraction network to generate a fake SAR image. sar SAR features are generated after SAR image segmentation and semantic feature extraction. Real optical images opt The true optical features are obtained through an optical image segmentation semantic feature extractor.

[0072] Real SAR images sar Generating SAR images (fake) sar Real optical images opt and generating optical images fake opt Input into the segmentation and contrast learning network; real SAR image sar and generating optical images fake opt First, the low-dimensional true SAR features of feature layer m are obtained by passing the features through multiple encoders I to the corresponding generator I. and low-dimensional generated optical features The actual SAR feature blocks are then obtained through the perceptron network I. and generating optical feature blocks Meanwhile, real optical images opt and generating SAR images fake sarFirst, the low-dimensional real optical features of the specific feature extraction layer m in the corresponding generator II are obtained by passing the features through a multi-layer encoder II. and low-dimensional generated SAR features The actual optical feature blocks are then obtained through a perceptron network II. and generating SAR feature blocks m = 1, 2, ..., M, where M is the total number of specific feature extraction layers in the generator feedforward network, and n = 1, 2, ..., N, where N is the dimension of the low-dimensional features.

[0073] From real SAR characteristics Select the real SAR feature block Regions with the same location are used as real SAR tag blocks Simultaneously generating SAR features Selecting and generating SAR feature blocks Regions with the same location are used as SAR tag blocks for generation. From real optical characteristics Select the real optical feature block Areas with the same location are used as real optical tag blocks. Simultaneously generating optical features Selecting and generating optical feature blocks Regions with the same location are used as the generating optical tag blocks.

[0074] For a specific feature extraction layer m, optical feature blocks are generated. Guided real SAR feature blocks To perform comparative learning, the first step is to select and generate optical feature blocks. As a reference, then zoom in on the actual SAR feature block. For the other N-1 real SAR feature blocks If the real SAR tag block and generating optical tag blocks If the categories are the same, then zoom in to the real SAR feature block. Otherwise, the actual distance of the SAR feature block k∈N,k≠n.

[0075] For a specific feature extraction layer m, SAR feature blocks are generated. Guided real optical feature blocks To conduct comparative learning, the first step is to select and generate SAR feature blocks. As a reference, then zoom in on the actual optical feature block. For the other N-1 real optical feature blocks If a real optical tag block and generating SAR tag blocks If the categories are the same, then zoom in to the actual optical feature blocks. Otherwise, pull the real far optical feature block k∈N,k≠n.

[0076] Step 4: Supervised training using a loss function

[0077] Calculate the real SAR image sar and generating SAR images fake sar The value of the confrontation between them Calculate real optical images opt and generating optical images fake opt The value of the confrontation between them Adversity loss values ​​encourage the generation of more realistic images.

[0078]

[0079] Where: E(·) represents calculating the expected value, and D(fake) represents... sar ) and D (real sar The output of discriminator II represents the generated SAR image (fake). sar and real SAR images sar True or False Score, ranging from 0 to 1; D(fake) opt ) and D (real opt ) represents the generated optical image generated by the output of discriminator I. opt and real optical images opt The score for true or false is between 0 and 1.

[0080] Calculate the real SAR image sar and reconstructed SAR image rec sar The pixel-level difference between them is used to obtain the cyclic loss value between them. Calculate real optical images opt and reconstructing optical images opt The pixel-level difference between them is used to obtain the cyclic loss value between them.

[0081]

[0082] Where: ||·||1 represents finding the L1 norm.

[0083] Calculate true SAR features and reconstructing SAR features Between the split cycle loss values Calculate real optical features and reconstruct optical features Between the split cycle loss values The degree to which the reconstructed image preserves its semantic structure is measured by the segmentation cycle loss value.

[0084]

[0085] Calculate the true SAR feature block Segmentation-guided contrastive learning loss value Calculate real optical feature blocks Segmentation-guided contrastive learning loss value

[0086]

[0087] Where: L SPatchNCE This represents the segmentation-guided contrastive learning loss value; This indicates the selected anchor point, i.e., the corresponding generated optical feature block. (correspond Or generate SAR feature blocks (correspond ); Indicates anchor point Positive samples, i.e., the corresponding real SAR feature blocks Or real optical feature blocks This indicates that among the other 255 real feature blocks, the anchor point... The set of feature blocks with the same category at corresponding locations, i.e., those corresponding to the anchor point. Real SAR tag blocks of the same category Or real optical feature blocks Indicates anchor point The set of negative samples; v represents the set of all anchor points. P represents the set containing all positive samples.

[0088] N represents the set containing all negative samples. τ is a hyperparameter, and here we set τ = 0.07.

[0089] The total loss is obtained by weighted summation of all the above loss values. The transformation network is then trained under supervision based on the total loss. The training of the transformation network is complete when the total loss converges.

[0090] Step 5: Test using the trained transformation network based on segmentation semantic guided contrastive learning.

[0091] like Figure 3 As shown, the real SAR images in the test set are... sarThe trained conversion network is input, and the output is a cross-modal conversion result. This result is then compared with a real SAR image. sar Comparative analysis reveals that the segmentation semantic-guided contrastive learning strategy employed in this study demonstrates superior conversion performance in complex remote sensing scenes. The conversion results not only maintain clarity at feature boundaries but also exhibit higher consistency in semantic structure. Particularly in scenarios with densely distributed roads, buildings, and water bodies, it accurately reconstructs the target domain image features, effectively avoiding semantic confusion and structural errors. This method possesses strong generalization capabilities, is applicable to various remote sensing image conversion tasks, and has significant practical application value.

[0092] like Figure 4 The diagram shows a structural block diagram of a SAR-to-optical image conversion device based on segmentation semantic-guided contrastive learning for implementing the method of this case. It includes a multi-source remote sensing image acquisition unit, a conversion network based on segmentation semantic-guided contrastive learning, a training unit, and a testing unit. The conversion network based on segmentation semantic-guided contrastive learning includes a generation network, a semantic feature extraction network, and a segmentation contrastive learning network.

[0093] The multi-source remote sensing image acquisition unit is used to acquire the original SAR image, the original optical image, and the semantic segmentation label, and to perform cropping and standardization processing on the original SAR image and the original optical image to obtain a real SAR image and a real optical image of a set size.

[0094] The generative network includes two generators and their respective discriminators; the generators are used to learn cross-modal mapping relationships in an end-to-end training framework, and the discriminators are used to improve the quality of image detail generation; the two generators are denoted as Generator I and Generator II, respectively, Generator I takes an optical image as input and outputs an optical image, and Generator II takes an optical image as input and outputs a SAR image; the two discriminators are denoted as Discriminator I and Discriminator II, respectively.

[0095] After training, the semantic feature extraction network uses a frozen SAR image segmentation semantic feature extractor and an optical image segmentation semantic feature extractor to participate in the training and testing of the transformation network. The input of the SAR image segmentation semantic feature extractor is a SAR image and the output is SAR semantic features. The input of the optical image segmentation semantic feature extractor is an optical image and the output is optical semantic features.

[0096] The segmentation contrastive learning network includes two multilayer encoders and their respective perceptron networks. The two multilayer encoders correspond to the two generators of the generator network and are denoted as Multilayer Encoder I and Multilayer Encoder II, respectively. The two perceptron networks are denoted as Perceptron Network I and Perceptron Network II, respectively. The multilayer encoder consists of M convolutional layers corresponding to M specific feature extraction layers in the feedforward network of the corresponding generator. It is used to capture the structural and texture information of the input image at multiple levels to obtain M layers of low-dimensional image features. The perceptron network is a two-layer perceptron structure, which is used to project the M layers of low-dimensional image features extracted by the multilayer encoders onto the embedding space required for contrastive learning to obtain a set of feature blocks.

[0097] The training unit uses a loss function to train a transformation network based on segmentation semantic guided contrastive learning.

[0098] The test unit uses a trained conversion network based on segmentation semantic guided contrastive learning to perform image conversion on the original SAR image.

[0099] In the aforementioned SAR-to-optical image conversion device based on segmentation semantic-guided contrastive learning, the conversion network based on segmentation semantic-guided contrastive learning is the core, and the segmentation contrastive learning network is an extremely important part of the core. Based on the generated low-dimensional image features, it integrates the semantic category information provided by semantic segmentation labels, and improves the semantic consistency and structure preservation ability of the generated image by explicitly narrowing the feature distance between similar land features and widening the feature distance between dissimilar land features. With the assistance of the semantic segmentation map, the segmentation contrastive learning network can achieve more semantically interpretable and structurally consistent contrastive feature alignment between cross-modal images.

[0100] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the above embodiments do not limit the present invention in any way, and all technical solutions obtained by equivalent substitution or equivalent transformation fall within the protection scope of the present invention.

Claims

1. A SAR-to-optical image conversion method based on segmentation semantic-guided contrastive learning, characterized in that: By constructing a conversion network based on segmentation semantic-guided contrastive learning, real SAR images are converted into generated optical images; the conversion network includes a generation network, a trained semantic feature extraction network, and a segmentation contrastive learning network. The generative network includes two generators and their respective discriminators. The generators are used to learn cross-modal mapping relationships in an end-to-end training framework, and the discriminators are used to improve the quality of generated image details. The two generators are denoted as Generator I and Generator II, respectively. Generator I takes a SAR image as input and outputs an optical image, while Generator II takes an optical image as input and outputs a SAR image. The two discriminators are denoted as Discriminator I and Discriminator II, respectively. After training, the semantic feature extraction network uses a frozen SAR image segmentation semantic feature extractor and an optical image segmentation semantic feature extractor to participate in the training and testing of the transformation network. The input of the SAR image segmentation semantic feature extractor is a SAR image and the output is SAR semantic features. The input of the optical image segmentation semantic feature extractor is an optical image and the output is optical semantic features. The segmentation contrastive learning network includes two multilayer encoders and their respective perceptron networks. The two multilayer encoders correspond to the two generators of the generator network and are denoted as Multilayer Encoder I and Multilayer Encoder II, respectively. The two perceptron networks are denoted as Perceptron Network I and Perceptron Network II, respectively. The multilayer encoder consists of M convolutional layers corresponding to M specific feature extraction layers in the feedforward network of the corresponding generator. It is used to capture the structural and texture information of the input image at multiple levels to obtain M layers of low-dimensional image features. The perceptron network is a two-layer perceptron structure, which is used to project the M layers of low-dimensional image features extracted by the multilayer encoders onto the embedding space required for contrastive learning to obtain a set of feature blocks.

2. The SAR-to-optical image conversion method based on segmentation semantic-guided contrastive learning according to claim 1, characterized in that: The execution process of the generator network includes: converting real SAR images into real SAR images. sar and real optical images opt The input is fed into the generator network. First, the real SAR image is... sar and real optical images opt The generated optical image is obtained by passing it through generator I and generator II respectively. opt and generating SAR images fake sar Then, generate a fake SAR image. sar and generating optical images fake opt The image is then passed through generator I and generator II again to obtain the reconstructed optical image rec. opt and reconstructed SAR image rec sar The optical image generated using discriminator I is fake. opt The authenticity score is calculated using Discriminator II to generate fake SAR images. sar The score for authenticity.

3. The SAR-to-optical image conversion method based on segmentation semantic-guided contrastive learning according to claim 1, characterized in that: The execution process of the semantic feature extraction network includes: Reconstruct the SAR image rec sar and reconstructing optical images opt The input is fed into a semantic feature extraction network to reconstruct the SAR image rec. sar The SAR features are reconstructed after SAR image segmentation and semantic feature extraction. Reconstructing optical images opt Reconstructed optical features are obtained after optical image segmentation semantic feature extraction. Real SAR images sar and generating optical images fake opt The input is fed into the semantic feature extraction network to obtain the real SAR image. sar The true SAR features are obtained by the SAR image segmentation semantic feature extractor. Generating optical images fake opt The generated optical features are obtained through an optical image segmentation semantic feature extractor. The generated SAR image is fake. sar and real optical images opt Input into a semantic feature extraction network to generate a fake SAR image. sar SAR features are generated after SAR image segmentation and semantic feature extraction. Real optical images opt The true optical features are obtained through an optical image segmentation semantic feature extractor.

4. The SAR-to-optical image conversion method based on segmentation semantic-guided contrastive learning according to claim 1, characterized in that: The execution process of the segmentation contrast learning network includes: Real SAR images sar Generating SAR images (fake) sar Real optical images opt and generating optical images fake opt Input into the segmentation and contrast learning network; real SAR image sar and generating optical images fake opt First, the low-dimensional true SAR features of feature layer m are obtained by passing the features through multiple encoders I to the corresponding generator I. and low-dimensional generated optical features The actual SAR feature blocks are then obtained through the perceptron network I. and generating optical feature blocks Meanwhile, real optical images opt and generating SAR images fake sar First, the low-dimensional real optical features of the specific feature extraction layer m in the corresponding generator II are obtained by passing the features through a multi-layer encoder II. and low-dimensional generated SAR features The actual optical feature blocks are then obtained through a perceptron network II. and generating SAR feature blocks m = 1, 2, ..., M, where M is the total number of specific feature extraction layers in the generator feedforward network, and n = 1, 2, ..., N, where N is the dimension of the low-dimensional feature. From real SAR characteristics Select the real SAR feature block Regions with the same location are used as real SAR tag blocks Simultaneously generating SAR features Selecting and generating SAR feature blocks Regions with the same location are used as SAR tag blocks for generation. From real optical characteristics Select the real optical feature block Areas with the same location are used as real optical tag blocks. Simultaneously generating optical features Selecting and generating optical feature blocks Regions with the same location are used as the generating optical tag blocks. For a specific feature extraction layer m, optical feature blocks are generated. Guided real SAR feature blocks To perform comparative learning, the first step is to select and generate optical feature blocks. As a reference, then zoom in on the actual SAR feature block. For the other N-1 real SAR feature blocks If the real SAR tag block and generating optical tag blocks If the categories are the same, then zoom in on the actual SAR feature blocks. Otherwise, the actual distance of the SAR feature block k∈N, k≠n; For a specific feature extraction layer m, SAR feature blocks are generated. Guided real optical feature blocks To conduct comparative learning, the first step is to select and generate SAR feature blocks. As a reference, then zoom in on the actual optical feature block. For the other N-1 real optical feature blocks If a real optical tag block and generating SAR tag blocks If the categories are the same, then zoom in to the actual optical feature blocks. Otherwise, pull the real optical feature block. k∈N, k≠n.

5. The SAR-to-optical image conversion method based on segmentation semantic-guided contrastive learning according to claim 1, characterized in that: The conversion network also includes a loss value calculation unit to calculate the real SAR image. sar and generating SAR images fake sar The adversarial loss value between them is used to calculate the real optical image. opt and generating optical images fake opt Adversarial loss values ​​between them; calculating real SAR images sar and reconstructed SAR image rec sar The cyclic loss value between them is used to calculate the real optical image. opt and reconstructing optical images opt Cyclic loss values ​​between; calculation of true SAR features and reconstructing SAR features The segmentation cycle loss value between them is used to calculate the true optical features. and reconstruct optical features The segmentation cycle loss value between; calculate the true SAR feature block. The segmentation-guided contrastive learning loss value is used to calculate the true optical feature blocks. The segmentation guides the contrastive learning loss value; the total loss is obtained by weighted summation of all loss values. When the total loss converges, the training of the transformation network is complete.

6. The SAR-to-optical image conversion method based on segmentation semantic-guided contrastive learning according to claim 1, characterized in that: The semantic feature extraction network includes a structurally consistent SAR image segmentation semantic feature extractor and an optical image segmentation semantic feature extractor. The SAR image segmentation semantic feature extractor uses real SAR images and semantic segmentation labels to extract multi-scale, multi-category SAR semantic features through supervised learning. The optical image segmentation semantic feature extractor uses real optical images and semantic segmentation labels to extract multi-scale, multi-category optical semantic features through supervised learning. Both segmentation semantic feature extractors are based on Dee. x The DeepLabV3-ResNet network structure is trained and constructed. The DeepLabV3-ResNet network includes a backbone feature extraction subnetwork and a semantic prediction subnetwork. The backbone feature extraction subnetwork uses a pre-trained ResNet as a feature extractor to extract multi-scale semantic information of the input image. The semantic prediction subnetwork includes a dilated spatial pyramid pooling module to enhance the perception of target objects at different scales.

7. The SAR-to-optical image conversion method based on segmentation semantic-guided contrastive learning according to claim 6, characterized in that: In the segmentation semantic feature extractor of the semantic feature extraction network, the backbone feature extraction subnetwork uses a pre-trained ResNet50 as the feature extractor. ResNet50 has a five-layer structure. The first layer includes a convolutional layer and a pooling layer. The second to fifth layers each include a residual module. The four residual modules realize the stepwise feature extraction from the low-level texture to the high-level semantics. The dilated spatial pyramid pooling module constructs a 3×3 dilated convolutional branch with dilation rates of 6, 12, and 18, as well as a 1×1 convolutional branch and an image-level global average pooling branch, based on the high-level semantic features output by the backbone feature extraction subnetwork. After concatenating the outputs of each branch, a multi-scale semantic context representation is generated by fusing them through a 1×1 convolution. During the training process of the semantic feature extraction network, the training set and the test set are evaluated separately. The average intersection-union ratio (mIoU) is used as the evaluation standard. The parameters of the semantic feature extraction network are saved when the average intersection-union ratio (mIoU) on the test set is optimal, and finally the segmentation semantic feature extractor is obtained.

8. The SAR-to-optical image conversion method based on segmentation semantic-guided contrastive learning according to claim 1, characterized in that: In the generative network, both generators employ a symmetric ResNetGenerator structure based on the ResNet architecture to achieve cross-modal bidirectional mapping between SAR and optical images. Each generator consists of an encoder, a residual block module, and a decoder. The encoder first performs multi-scale feature extraction and dimensionality compression on the input image through a set of downsampling convolutional layers to enhance the generator's ability to express spatial features such as edge textures and structural contours. Then, it obtains encoded features through an instance normalization layer and a ReLU activation function, improving the generator's training stability and nonlinear modeling capability. The residual block module includes nine consecutively stacked residual blocks, each consisting of two 3×3 convolutional layers connected using an identity mapping residual connection. The decoder gradually restores the image resolution through transposed convolution and guides image generation or reconstruction by combining encoded features. Simultaneously, it uses a tanh activation function to output normalized results, achieving the restoration of structural and texture information.

9. The SAR-to-optical image conversion method based on segmentation semantic-guided contrastive learning according to claim 1, characterized in that: In the generative network, both discriminators are implemented using the NLayerDiscriminator based on the PatchGAN architecture, which includes a 3-layer backbone convolution module, designed to perform parallel real and fake discrimination of multiple local regions in the input image; the discriminator divides the input image into local regions through a sliding window mechanism and performs real and fake discrimination prediction for each local region.

10. The SAR-to-optical image conversion method based on segmentation semantic-guided contrastive learning according to claim 1, characterized in that: In the segmentation contrastive learning network, the multi-layer encoder consists of five convolutional layers from shallow to deep, corresponding to specific feature extraction layers 0, 4, 8, 12, and 16 in the generator feedforward network, respectively. These layers are used to capture the structural and texture information of the input image at multiple levels, ultimately obtaining low-dimensional image features for each convolutional layer. The perceptron network adopts a two-layer perceptron structure, including two linear transformation units, a normalization layer, and a non-linear activation function. This structure is used to perform feature compression and embedding mapping on the five layers of low-dimensional image features extracted by the encoder. First, the first linear transformation unit performs a linear transformation on the input low-dimensional image features, mapping them to an intermediate latent space to initially construct a discriminative embedding. Then, the feature distribution of the initially constructed discriminative embedding is normalized by the normalization layer to obtain an intermediate representation. Next, the second linear transformation unit further maps the intermediate representation to the embedding space required for the final contrastive learning, outputting five feature block groups with a dimension of 256.

Images