S2o image translation method based on hidden space alignment twin network

Abstract

The application relates to the fields of remote sensing technology and computer vision technology, and particularly discloses an S2O image translation method based on a hidden space alignment twin network, which comprises the following steps: constructing and training a true twin network model, the true twin network model comprising two subnetworks, an SAR-to-optical image generator and an optical-to-SAR image generator, each of the SAR-to-optical image generator and the optical-to-SAR image generator comprising an encoder and a decoder; constructing a training data set, the training data set comprising multiple pairs of matched SAR images and optical images, and inputting a SAR image to be translated into the true twin network model; extracting hidden space features of the SAR image to be translated through the true twin network model and aligning the hidden space features to generate a corresponding optical image; and outputting the converted optical image through the SAR-to-optical image generator.

Description

Technical Field

[0001] This invention relates to the fields of remote sensing technology and computer vision technology, specifically to an S2O image translation method based on latent space aligned twin networks. Background Technology

[0002] Synthetic Aperture Radar (SAR) and optical imaging are two important technologies in the field of remote sensing. SAR has all-weather, all-day operation capabilities and can effectively penetrate clouds and smoke, playing a crucial role in areas such as surface monitoring, disaster assessment, and military reconnaissance. However, SAR images are generally affected by their coherent imaging mechanism, resulting in problems such as speckle noise and geometric distortion. Their grayscale information differs significantly from human visual perception, making visual interpretation difficult and requiring highly specialized interpreters. In contrast, optical images contain rich spectral and textural information, are more consistent with human visual perception, and are easier to understand and analyze.

[0003] To overcome the bottleneck in interpreting SAR images, SAR-to-optical (S2O) image translation technology has emerged. Its core objective is to convert the microwave scattering characteristics of SAR images into simulated optical image representations, thereby improving the intuitiveness and interpretability of the images. In recent years, deep learning-based methods, especially Generative Adversarial Networks (GANs), have become the mainstream solution in this field.

[0004] CycleGAN and its variants (such as S-CycleGAN) achieve unsupervised image translation without paired data by introducing cycle-consistent loss, providing a feasible framework for S2O conversion. Another type of method, such as Pix2Pix, typically generates higher-quality images in a supervised learning setting with paired data. Subsequent research has proposed many improved models, such as WFLM-GAN combining wavelet transform, the semi-supervised framework SemiD2, and the ConditionalDiffusion method based on a diffusion model, attempting to achieve breakthroughs in both generation quality and training stability.

[0005] However, these existing methods still have significant limitations. First, they typically rely on complex network structures (such as multi-scale generators or diffusion models), resulting in a large number of model parameters, high computational costs, and difficulty in stabilizing the training process. Second, and most critically, these methods are insufficient in preserving image detail features when faced with complex scenes (such as urban building complexes or natural terrain), easily leading to texture blurring, structural distortion, or color deviation. The root cause is that these methods lack an effective alignment mechanism for deep features of both SAR and optical modalities, failing to establish a good mapping relationship of detail and structure between the two modalities, resulting in converted images that still lack visual realism and semantic consistency.

[0006] Therefore, the current field urgently needs a new method that can efficiently achieve cross-modal feature alignment and significantly improve the fidelity of image translation details while reducing model complexity. Summary of the Invention

[0007] To achieve the objectives of this invention, this application provides an S2O image translation method based on a latent space aligned Siamese network, comprising: Step S1: Construct and train a true twin network model, which includes two sub-networks: a SAR-to-optical image generator and an optical-to-SAR image generator, each of which includes an encoder and a decoder; Step S2: Construct a training dataset, which includes multiple pairs of paired SAR images and optical images, and input the SAR images to be translated into the true twin network model; Step S3: Extract the latent space features of the SAR image to be translated using the true twin network model and align them to generate the corresponding optical image; Step S4: Output the converted optical image through the SAR to optical image generator.

[0008] In some specific embodiments, in step S1, the two sub-networks of the true twin network model adopt a weight-sharing mechanism to share the parameters of the encoder and decoder.

[0009] In some specific embodiments, step S3 includes: Step S31: Design a loss function based on the true twin network model. The loss function includes adversarial loss, cycle consistency loss, and contrastive loss. The contrastive loss includes latent space-aware contrastive loss function and reconstruction contrastive loss function. Step S32: Calculate the Euclidean distance between the latent space feature maps output by the encoder of the SAR to the optical image generator, and constrain the Euclidean distance by the latent space perceptual contrast loss function; Step S33: Train the true twin network model using the training dataset and optimize the network parameters by minimizing the loss function.

[0010] In some specific embodiments, the latent space-aware contrastive loss function is determined according to the following formula: In the formula, n represents the number of samples in the current training batch, and k represents the dimension of each pixel in the feature map. Let represent the latent space feature map generated by the i-th subnetwork for the j-th sample.

[0011] In some specific embodiments, the reconstruction contrast loss function includes: For the same batch of input samples, reconstructed SAR images are generated through two sub-networks, the difference between the two reconstructed images is calculated, and the difference is added to the total loss function as a loss term.

[0012] In some specific embodiments, the reconstruction contrast loss function is determined according to the following formula: In the formula, rec s1 and rec s2 These represent the reconstructed SAR samples generated through the two branches of the twin network.

[0013] In some specific embodiments, the true twin network model is trained by jointly optimizing adversarial loss, cycle consistent loss, and contrastive loss, wherein the contrastive loss includes a weighted sum of latent space-aware contrastive loss and reconstruction contrastive loss.

[0014] In some specific embodiments, both the SAR-to-optical image generator and the optical-to-SAR image generator employ an encoder-decoder structure.

[0015] In some specific embodiments, step S1 further includes: training the true twin network model by sampling multiple SAR image samples from the same batch and inputting them into two sub-networks respectively.

[0016] In some specific embodiments, the true twin network model uses PatchGAN as the discriminator structure.

[0017] The beneficial effects of the above technical solution are as follows: The proposed S2O image cross-modal conversion method based on latent space aligned Siam networks can efficiently extract common features from SAR and optical images, solving the problem of detail loss during image conversion in complex scenes. Taking advantage of the weight sharing characteristic of SiamS2O, a latent space perceptual contrast loss is proposed to reduce the distance between high-dimensional latent space feature maps in the sub-network encoder. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1A flowchart illustrating an S2O image translation method based on a latent space aligned Siamese network, provided as an embodiment of the present invention; Figure 2 A schematic diagram illustrating the principle of the SiamS2O algorithm for an S2O image translation method based on a latent space aligned Siamese network, provided as an embodiment of the present invention. Detailed Implementation

[0020] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0021] Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar symbols denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the invention, and should not be construed as limiting the invention.

[0022] Example 1 One embodiment of the present invention provides an S2O image translation method based on a latent space aligned Siamese network, referring to... Figure 1 , Figure 2 As shown, it includes: Step S1: Construct and train a true twin network model, which includes two sub-networks: a SAR-to-optical image generator and an optical-to-SAR image generator, each of which includes an encoder and a decoder; In a specific embodiment of the present invention, in step S1, the two sub-networks of the true twin network model adopt a weight sharing mechanism to share the parameters of the encoder and decoder.

[0023] Based on the advantages of model and contrastive loss design, this application proposes a Siam-based S2O image cross-modal conversion algorithm (SiamS2O). Figure 2 As shown, SiamS2O is a true Siamese network architecture, which includes two symmetric recurrent consistent adversarial generative network sub-learning branches. The recurrent consistent loss helps the model retain more image color information; therefore, SiamS2O combines Siamese network strategies and recurrent consistent strategies. Each sub-network consists of two generators (…). and It consists of ) . Among them, the generator The aim is to convert SAR images into optical images, while the generator This is responsible for converting the optical image back into a SAR image. Specifically, it first randomly samples two batches of samples from the SAR domain ( and The data is then input into two sub-networks of the twin network. For a single sub-branch, a cyclically consistent information flow transmission link is constructed: SAR sample-optical-SAR. First, through the generator Converted into an optical image, then processed by a generator The optical image is reconstructed into a SAR image, and the final output is the reconstructed SAR sample. This design not only enables bidirectional translation between SAR and optical images, but also ensures the reliability and stability of the mode conversion process through cyclic consistency constraints.

[0024] Therefore, the output of SiamS2O on each branch is the reconstructed SAR image. For the S2O image cross-modal conversion task, this can be achieved by extracting the intermediate generator G. S The generated results achieve the transformation and generation task. In terms of discriminator design, a discriminator G is constructed using a classic PatchGAN network. S To constrain the quality of generated optical images.

[0025] Step S2: Construct a training dataset, which includes multiple pairs of paired SAR images and optical images, and input the SAR images to be translated into the true twin network model; In one specific embodiment of the present invention, the true twin network model is trained by jointly optimizing adversarial loss, cycle consistent loss and contrastive loss, wherein the contrastive loss includes a weighted sum of latent space perception contrastive loss and reconstruction contrastive loss.

[0026] In one specific embodiment of the present invention, both the SAR to optical image generator and the optical to SAR image generator adopt an encoder-decoder structure.

[0027] In one specific embodiment of the present invention, step S1 further includes: training the true twin network model by sampling multiple SAR image samples from the same batch and inputting them into two sub-networks respectively.

[0028] In one specific embodiment of the present invention, the true twin network model uses PatchGAN as the discriminator structure.

[0029] Step S3: Extract and align the latent space features of the SAR image to be translated using the true twin network model; In a specific embodiment of the present invention, step S3 includes: Step S31: Design a loss function based on the true twin network model. The loss function includes adversarial loss, cycle consistency loss, and contrastive loss. The contrastive loss includes latent space-aware contrastive loss function and reconstruction contrastive loss function. Step S32: Calculate the Euclidean distance between the latent space feature maps output by the encoder of the SAR to the optical image generator, and constrain the Euclidean distance by the latent space perception contrast loss function; Step S33: Train the true twin network model using the training dataset and optimize the network parameters by minimizing the loss function.

[0030] In a specific embodiment of the present invention, the latent space sensing contrast loss function is determined according to the following formula: In the formula, n represents the number of samples in the current training batch, and k represents the dimension of each pixel in the feature map. Let represent the latent space feature map generated by the i-th subnetwork for the j-th sample.

[0031] In one specific embodiment of the present invention, the reconstruction contrast loss function includes: For the same batch of input samples, reconstructed SAR images are generated through two sub-networks, the difference between the two reconstructed images is calculated, and the difference is added to the total loss function as a loss term.

[0032] In a specific embodiment of the present invention, the reconstruction contrast loss function is determined according to the following formula: In the formula, rec s1 and rec s2 These represent the reconstructed SAR samples generated through the two branches of the twin network.

[0033] Specifically, Siamese networks offer the advantage of task-specific contrastive loss design, enabling the model to effectively learn similarity metrics between samples. Through the contrastive loss function, Siamese networks maximize the similarity between positive sample pairs, allowing for efficient model training based solely on the relationship information of sample pairs without explicit classification labels. This makes them particularly suitable for S2O image cross-modal conversion tasks. Furthermore, due to their symmetric architecture and shared weights, Siamese networks maintain strong feature extraction capabilities while reducing the number of parameters, thus mitigating the risk of overfitting.

[0034] (1) Latent space perception contrast loss Typically, the generator in an encoder-decoder transforms image information into a high-dimensional feature space, allowing the feature extractor to extract high-dimensional information from the latent space. Therefore, the encoder's transformation role is crucial, and researchers improve performance by increasing its complexity. In this context, SiamS2O's two sub-networks employ a weight-sharing training method, thus reducing the contrastive loss... It has a natural advantage in terms of design. This application proposes a latent space perception contrast loss. This method optimizes the feature extraction performance of SiamS2O, avoiding techniques that increase network complexity and improving conversion efficiency from the perspective of loss constraints. The following section provides... Formula derivation: Suppose the two twin subnetworks are respectively and The set of input samples is After the encoder, two latent space feature maps are obtained. and ,in Let represent the latent space feature map generated by the i-th subnetwork for the j-th sample.

[0035] The Euclidean distance is used to calculate the difference in the latent space feature maps of the two branches, with the first branch being the first branch. Taking a single sample as an example, the distance between two latent space feature maps for: (1) In the formula, k represents the dimension of each pixel in the feature map. To minimize the difference in latent space feature maps among the same batch of input samples, the average of all samples in the current batch is calculated to obtain... The formula is: (2) The core idea is to align the feature representations of two branches in the latent space, thereby enhancing the discriminativeness and consistency of feature encoding. This process includes two steps: latent space feature extraction and latent space similarity calculation. 1) For latent space feature extraction, after the input sample passes through the encoder in the generator, a latent space feature map (LS) is obtained. These features capture the high-dimensional semantic information of the input sample, such as the geometric structure in SAR images and the spectral characteristics of optical images. 2) Latent space similarity calculation uses Euclidean distance to calculate the difference between the latent space feature maps of the two branches. By minimizing the difference in latent space feature maps of the same batch of input samples in the two branches, the consistency of feature encoding is ensured.

[0036] The three main functions of latent space are feature alignment, improved discriminative power, and accelerated convergence. Specifically, feature alignment in the latent space reduces the modal information differences between SAR and optical images; enhancing the discriminative power of feature encoding provides higher-quality feature representations for subsequent feature extraction modules; and the constraints of the latent space further optimize the network training process, thereby improving learning efficiency.

[0037] (2) Reconstruction contrast loss Reconstruction contrast loss ( This is another core part of the contrastive loss design. The goal of this loss is to quantify the difference between the outputs of two branches, thereby ensuring the consistency and stability of the network during information transmission. Specifically, for two input samples ( and The reconstructed SAR samples are generated through two branches of the twin network, respectively. and ). By calculating the distance between the two branches, the difference in their reconstruction results can be reflected. Therefore, The mechanism by which this loss function works is to regularize the output differences of data from the same batch. This loss function ensures consistency in the constrained reconstruction while also enhancing robustness, thereby reducing reconstruction errors caused by noise and modal differences in the input samples. The mathematical expression is: (3) Therefore, for the contrastive loss in Siamese networks, the total contrastive loss function As shown below: (4) in, These are the weighting coefficients, used to balance the contributions of the two types of losses. represent The loss of hidden space, represent The latent space loss. Overall, The advantages are mainly threefold: 1) Reconstruction contrast loss: Following the calculation method of contrast loss in Siamese networks, the difference in reconstruction results is quantified to ensure the consistency of the two Siamese branches in the reconstruction process, thereby enhancing the robustness of the model and the reconstruction accuracy.

[0038] 2) Latent space perceptual contrastive loss: Aligns feature representations in the latent space, thereby reducing modality differences and improving the discriminativeness and consistency of feature encoding.

[0039] 3) Joint optimization: By using a joint loss function, the reconstruction results and the feature representation of the latent space are optimized simultaneously, which significantly improves the performance and efficiency of the model.

[0040] (3) Objective function SiamS2O's objective function consists of three parts, which are adversarial to loss. Cyclic loss And comparative loss The expression is shown in the following formula: (5) In combating losses, These represent the real SAR and optical images sampled in subnetwork 1, respectively. The images represent real SAR and optical images sampled in sub-network 2. The generator network G aims to improve the realism of generated samples by minimizing adversarial loss, while the discriminator network D strengthens the ability to distinguish between generated and real samples by maximizing adversarial loss, ultimately achieving a dynamic game balance between the two. The cyclic loss constrains the generator style while preventing mode collapse and maintaining transformation stability. This loss establishes a strong alignment consistency constraint between the reconstructed and original images through a forced invertible mapping, acting as a regularizer.

[0041] Step S4: The optical image is converted by the SAR to optical image generator.

[0042] This application proposes a weight-sharing Siamese network to enhance the cross-modal alignment capability of S2O images. The shared weight mechanism of the generator can efficiently extract common features from SAR and optical images, reducing the differences caused by the two imaging mechanisms and thus improving alignment accuracy. A latent space-aware contrast loss is proposed, which aligns high-frequency spatial vectors in the feature extractor by constraining the latent space feature maps of the same encoder position in different sub-networks. This avoids increasing network complexity, thereby improving conversion efficiency. A reconstruction contrast loss is proposed, which reconstructs image feature differences through regularization of the sub-network to ensure consistency and stability during information transmission, enhancing the robustness and reconstruction accuracy of the model.

[0043] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

[0044] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. The embodiments of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing terminal equipment to cause a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1The steps of the functions specified in one or more boxes. Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the invention. Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0045] The methods and apparatus provided by the present invention have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

[0046] In the description of this specification, references to terms such as "an embodiment," "some embodiments," "example," "specific example," or "a specific embodiment" or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0047] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. An S2O image translation method based on latent space aligned Siamese networks, characterized in that, include: Step S1: Construct and train a true twin network model, which includes two sub-networks: a SAR-to-optical image generator and an optical-to-SAR image generator, each of which includes an encoder and a decoder; Step S2: Construct a training dataset, which includes multiple pairs of paired SAR images and optical images, and input the SAR images to be translated into the true twin network model; Step S3: Extract the latent space features of the SAR image to be translated using the true twin network model and align them to generate the corresponding optical image; Step S4: Output the converted optical image through the SAR to optical image generator.

2. The S2O image translation method based on latent space aligned Siamese network according to claim 1, characterized in that, In step S1, the two sub-networks of the true twin network model adopt a weight-sharing mechanism to share the parameters of the encoder and decoder.

3. The S2O image translation method based on latent space aligned Siamese network according to claim 1, characterized in that, Step S3 includes: Step S31: Design a loss function based on the true twin network model. The loss function includes adversarial loss, cycle consistency loss, and contrastive loss. The contrastive loss includes latent space-aware contrastive loss function and reconstruction contrastive loss function. Step S32: Calculate the Euclidean distance between the latent space feature maps output by the encoder of the SAR to the optical image generator, and constrain the Euclidean distance by the latent space perception contrast loss function; Step S33: Train the true twin network model using the training dataset and optimize the network parameters by minimizing the loss function.

4. The S2O image translation method based on latent space aligned Siamese network according to claim 3, characterized in that, The latent space sensing contrast loss function is determined according to the following formula: In the formula, n represents the number of samples in the current training batch, and k represents the dimension of each pixel in the feature map. Let represent the latent space feature map generated by the i-th subnetwork for the j-th sample.

5. The S2O image translation method based on latent space aligned Siamese network according to claim 3, characterized in that, The reconstruction contrast loss function includes: For the same batch of input samples, reconstructed SAR images are generated through two sub-networks, the difference between the two reconstructed images is calculated, and the difference is added to the total loss function as a loss term.

6. The S2O image translation method based on latent space aligned Siamese network according to claim 5, characterized in that, The reconstruction contrast loss function is determined according to the following formula: In the formula, rec s1 and rec s2 These represent the reconstructed SAR samples generated through the two branches of the twin network.

7. The S2O image translation method based on latent space aligned Siamese network according to claim 1, characterized in that, The true twin network model is trained by jointly optimizing adversarial loss, cycle consistent loss, and contrastive loss, wherein the contrastive loss includes a weighted sum of latent space perception contrastive loss and reconstruction contrastive loss.

8. The S2O image translation method based on latent space aligned Siamese network according to claim 1, characterized in that, Both the SAR-to-optical image generator and the optical-to-SAR image generator adopt an encoder-decoder structure.

9. The S2O image translation method based on latent space aligned Siamese network according to claim 1, characterized in that, Step S1 further includes: training the true twin network model by sampling multiple SAR image samples from the same batch and inputting them into the two sub-networks respectively.

10. The S2O image translation method based on latent space aligned Siamese network according to claim 1, characterized in that, The true twin network model uses PatchGAN as the discriminator structure.

Images