A sar image to optical image conversion method based on a multi-modal conditional diffusion model

Abstract

The application provides a SAR image to optical image conversion method based on a multi-modal conditional diffusion model. First, SAR image and optical image data with a paired relationship are collected to ensure that the samples have diversity in scene type, target structure and image style. Second, an image-language multi-modal model is used to generate text descriptions of SAR and optical images respectively, and a unified language description is constructed through semantic analysis and fusion to form a light-SAR-text three-modal training sample set containing structure, semantic and style information. A multi-modal conditional guided denoising diffusion model is designed and trained, and the noise-added and denoised reconstruction process of the original optical image is taken as an optimization target, and SAR images, language descriptions and style images are introduced as multi-modal prompt conditions to comprehensively guide the expression of the generated image in structure restoration, semantic alignment and style presentation.

Description

Technical Field

[0001] This invention belongs to the field of interdisciplinary technology of remote sensing and computer vision, and specifically relates to a method for converting remote sensing SAR images to optical images based on a multimodal conditional diffusion model. Background Technology

[0002] Synthetic Aperture Radar (SAR) imagery, with its advantages of penetrating clouds and fog and providing all-weather, all-day imaging, has been widely used in key areas such as natural disaster monitoring, military reconnaissance, and resource exploration, possessing irreplaceable strategic value. However, because SAR imaging is based on active microwave radar, its images exhibit unique grayscale distribution and scattering texture characteristics, which differ significantly from the natural light imaging results accustomed to by the human visual system. This modal difference poses significant challenges to the intuitive interpretation and rapid analysis of SAR images by humans. Therefore, converting SAR images into optical images that are closer to human visual perception has become an important research direction in the field of intelligent remote sensing image processing.

[0003] In recent years, with breakthroughs in generative diffusion models for image generation and modality conversion tasks, SAR-to-optical image conversion methods based on diffusion models have shown great potential. However, most existing methods use simple channel stitching to input both SAR and optical images into the diffusion network, which has the following significant limitations: First, SAR and optical images differ fundamentally in their imaging physics. For example, SAR is based on slant-range geometric imaging and suffers from speckle noise, while optical images are based on central projection and have natural texture features. Simple stitching easily leads to feature confusion between modes, resulting in structural distortion, semantic misalignment, and color distortion in the generated image. Second, these methods generally lack the ability to explicitly model the unique semantics and geometric structures of SAR images, such as the category information contained in the scattering features of ground objects or the spatial topological changes brought about by radar shadows. Modality fusion is only performed at the input layer, making it difficult to establish an accurate and consistent cross-modal mapping relationship. More importantly, existing end-to-end training schemes usually require parameter tuning of the entire diffusion model. In the context of scarce SAR-optical paired samples, the training cost is high, and the model is prone to overfitting and has insufficient generalization ability.

[0004] The core issue is that existing technologies have failed to effectively establish a collaborative control mechanism between the spatial structure of SAR images and the semantics of optical images, lacking sufficient multimodal semantic guidance and spatial constraints. Therefore, it is urgent to propose a multimodal conditional diffusion model that integrates linguistic and visual cues. By explicitly injecting semantic and structural prior information, this model can improve the generated images' performance in terms of structural consistency, visual realism, and semantic expressiveness, achieving high-fidelity and highly semantically consistent SAR image visualization generation. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies in semantic guidance and imprecise image style control during remote sensing image generation, this invention proposes a SAR image-to-optical image conversion method based on a multimodal conditional diffusion model. By introducing a multimodal guidance mechanism that includes textual language prompts, image content prompts, and image style prompts, the method significantly enhances the structural modeling and semantic alignment capabilities of the diffusion model in remote sensing scenarios, thereby improving the overall performance of the generated image in terms of controllability, realism, and semantic consistency.

[0006] A method for converting SAR images to optical images based on a multimodal conditional diffusion model, comprising the following steps:

[0007] Step 1: Collect remote sensing image datasets. Collect datasets containing SAR images and their corresponding optical images, ensuring the samples exhibit scene diversity, target type diversity, and stylistic differences. Data sources include publicly available remote sensing image datasets, satellite image databases, or paired images acquired independently. Additionally, collect optical-SAR semantic datasets containing semantic segmentation labels, utilizing the semantic categories and spatial structure information within them as linguistic descriptions or auxiliary reference information for image content.

[0008] Step 2: Construct a multimodal training dataset of SAR image-optical image-text, aiming to build a trimodal training dataset that integrates structural information (SAR image), semantic information (text description), and style information (optical image). A large-scale remote sensing image-language multimodal model is used to generate text descriptions for both SAR and optical images, extracting their semantic features. Subsequently, semantic parsing and fusion techniques are combined to semantically align and integrate the generated text, forming a unified and complete language description, thereby enhancing semantic consistency between different modalities. Simultaneously, an optical image with a similar style but different content from the original optical image is selected as a style guide image for each set of samples. Finally, each set of training samples includes: the original optical image, the corresponding SAR image, the fused text description, and the style image.

[0009] Step 3: Construction and Training of the Multimodal Conditional Diffusion Model (MM-DM). A multimodal conditional diffusion generation model, MM-DM, was designed and built. This model uses the denoising and denoising reconstruction process of the original optical image as the model optimization objective, combining multimodal conditional information such as SAR imagery, textual descriptions, and style images to guide the generation of images in terms of content and style. The main body of the model uses a stable diffusion model, introducing parallel ControlNet branches and a decoupled cross-attention mechanism to process structural guidance, textual semantics, image content, and image style cues, injecting multimodal conditional information into the denoising UNet structure. End-to-end iterative training is performed under the joint constraints of noise prediction loss and overall perception loss. Iterative training is conducted using the multimodal training dataset constructed in Step 2, jointly optimizing model parameters by combining noise prediction loss, reconstruction loss, and perception loss to improve the image generation quality of the model in complex remote sensing scenarios.

[0010] Step 4: SAR to Optical Image Inference Generation. In the inference phase, a textual description is first generated for the SAR image to be processed, and a target style image is selected. Then, using a trained MM-DM model, with the SAR image, textual description, and style image as conditions, random noise is initialized, and a diffusion inversion process of a set number of steps is executed to generate an optical image output that conforms to semantic and style constraints.

[0011] Specifically, the steps for collecting remote sensing image datasets in step 1 are as follows: obtain paired datasets of diverse SAR images and corresponding optical images from public datasets, satellite image databases, or through self-collection. In addition to datasets for conversion or fusion purposes, optical SAR semantic segmentation datasets are also collected simultaneously. The semantic segmentation labels in these datasets contain semantic categories and spatial structure information, which can also serve as reference information for image language description.

[0012] Step 2, which describes constructing a multimodal training dataset of SAR images, optical images, and text, specifically includes the following steps:

[0013] The first step is to generate image text descriptions. Using a multimodal large model of remote sensing image language, corresponding image text descriptions are generated for SAR images and optical images respectively, capturing their respective scene semantics and target information.

[0014] The second step is structured semantic parsing. The image description text above is then parsed in a structured manner. Language understanding tools are used to transform the description into semantic fragments containing geographic entities, attributes, and spatial relationships, thus constructing a semantic structure.

[0015] The third step is semantic alignment and conflict detection. Semantic fragments from SAR and optical images are analyzed for alignment. If the descriptions contain the same ground features or spatial relationships, they are considered semantically consistent; if one mentions something while the other does not, it is recorded as modal difference information; if there are obvious semantic conflicts, weighted judgment and conflict resolution can be performed by combining auxiliary features in the images or model confidence. This step constructs a semantic lookup table, providing a basis for generating fused descriptions.

[0016] The fourth step is to integrate and generate a unified language description. Based on alignment and difference analysis, the shared and complementary information from the two modal descriptions is integrated, and a language model (such as GPT) is used to guide the generation of a natural, fluent, and semantically complete integrated language description. Structured semantic information can be input to guide the language model to generate unified text, ensuring the consistency and expressiveness of the language.

[0017] Step 5: Sample Construction. Based on the obtained optical image, corresponding SAR image, and fused text description, an additional optical image similar in color and scene style but different in content is selected as a style guide image. This image is then combined with the aforementioned three-modal data to form a complete set of training samples.

[0018] Step 3 proposes a design and training method based on a multimodal conditional denoising diffusion model (MM-DM). The specific steps are as follows:

[0019] The first step is latent space noise addition and modeling. The original optical image is input into a pre-trained VAE encoder and compressed into a latent representation. Subsequently, Gaussian noise is gradually injected into the latent variables through a Markov chain with fixed steps to obtain a noisy latent representation that approximates a normal distribution, which serves as the initial input for the diffusion process.

[0020] The second step is structure control guidance. SAR images are used as prior structural information input to the ControlNet branch to provide overall spatial structure and contour information of the image to be generated, enhancing the ability to control image layout and ground feature morphology.

[0021] The third step involves multimodal semantic and style-guided injection. SAR images are processed by a semantic encoder to extract structural semantic features, text descriptions by a text encoder to extract linguistic semantic features, and optical style images by a style encoder to extract image style features. These three types of features are then uniformly embedded into tokens and injected into multiple cross-attention layers of UNet. By decoupling the cross-attention mechanism, various conditions can function at different semantic levels, achieving fine-grained control over image content and style.

[0022] The fourth step is noise prediction and image reconstruction. During denoising, the UNet network predicts the noise components in the latent representation based on the aforementioned multimodal conditional information and compares them with the injected known noise to calculate the noise prediction error and construct the noise prediction loss. Simultaneously, the denoised latent representation is reconstructed into an image using a VAE decoder, and the reconstruction error and perceptual error are jointly calculated with the original optical image. This process ensures the accuracy of content restoration in the generated image and also provides effective optimization in semantic consistency and style matching, further improving the realism, detail fidelity, and overall expressiveness of the generated image.

[0023] Step 4, SAR-to-optical image inference and generation. This step aims to utilize a trained multimodal conditional diffusion model (MM-DM) to generate optical images with specified style features from SAR images. The specific process includes the following three sub-steps:

[0024] The first step is SAR image language description generation. The SAR image to be processed is analyzed to generate a corresponding language text description, which serves as one of the semantic guidance conditions.

[0025] The second step is model loading and initialization. The MM-DM model trained in step 3 is loaded, the number of denoising steps in the diffusion process is set, and random noise is initialized as the initial input to the denoising UNet network.

[0026] The third step involves introducing conditions and performing generation. Representative target-style optical images are selected as style-guided conditions, while SAR images and their linguistic descriptions are input into the structure and semantic guidance channels, respectively. Multi-step denoising inference is performed using the MM-DM model to ultimately generate an optical image that is consistent with the target conditions in terms of structure, semantics, and style.

[0027] Compared to existing technologies, this invention offers the following advantages: It proposes an innovative SAR image-to-optical image conversion method based on multimodal conditional guidance. By fusing SAR images, optical images, and their corresponding linguistic descriptions, a three-dimensional collaborative multimodal information guidance mechanism integrating structure, semantics, and style is constructed. This method decouples and models multimodal cue information, dynamically injecting it in each denoising iteration of the diffusion generation process, comprehensively guiding the image generation process from three levels: semantic understanding, spatial structure reconstruction, and style presentation. This mechanism significantly overcomes the bottlenecks of traditional SAR image conversion methods in semantic expression and structural restoration, enhancing the model's structural modeling, semantic alignment, and style control capabilities in remote sensing image generation tasks. It effectively improves the controllability, realism, and semantic consistency of generated images, breaking through the information expression bottleneck of traditional methods in the modality conversion process. The technical solution of this invention has broad application prospects, particularly suitable for tasks such as military reconnaissance, disaster response, urban planning, remote sensing reconstruction, and multi-source image fusion, and is of great significance in improving the level of intelligent remote sensing processing and image usability. This invention proposes a SAR image-to-optical image conversion method based on a multimodal conditional diffusion model. First, paired SAR and optical image data are collected to ensure diversity in scene type, target structure, and image style. Second, an image-language multimodal model is used to generate text descriptions for SAR and optical images respectively. Through semantic parsing and fusion, a unified language description is constructed, forming a three-modal training sample set of light-SAR-text containing structural, semantic, and stylistic information. This invention designs and trains a multimodal condition-guided denoising diffusion model (MM-DM), using the denoising-denoising reconstruction process of the original optical image as the optimization objective. SAR images, language descriptions, and style images are introduced as multimodal cue conditions to comprehensively guide the expression of generated images in terms of structural restoration, semantic alignment, and style presentation. This model combines noise prediction loss, reconstruction loss, and perceptual loss for joint training, effectively improving the realism, detail fidelity, and semantic consistency of the generated images. In the inference phase, the trained MM-DM model is used as input, along with the SAR image, its generated text description, and the target style image. Through a diffusion sampling process, an optical image with specified style features is generated. This method significantly enhances the model's structural modeling and semantic control capabilities in remote sensing image generation by injecting multimodal semantic and structural information into each denoising iteration, thereby improving the controllability and realism of the generated results. This invention is applicable to tasks such as the fusion, reconstruction, and style transfer of multi-source remote sensing images, and is particularly suitable for achieving high-quality, controllable SAR-to-optical image conversion in remote sensing scenarios. Attached Figure Description

[0028] Figure 1 This is the overall flowchart of this method;

[0029] Figure 2 This is a schematic diagram of the training process of the multimodal conditional diffusion model of the present invention;

[0030] Figure 3 This is a schematic diagram of the ControlNet conditional control mechanism provided in an embodiment of the present invention;

[0031] Figure 4 This is a schematic diagram of the inference process based on the multimodal conditional diffusion model of the present invention. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the present invention will be further described below with reference to the accompanying drawings and embodiments. The present invention includes, but is not limited to, the following embodiments.

[0033] A method for converting SAR images to optical images based on a multimodal conditional diffusion model includes the following steps:

[0034] Step 1: Acquire several sets of SAR images and corresponding raw optical images of the same scene as initial remote sensing image training data, ensuring that the samples possess scene diversity, target type diversity, and style differences. Image data can be sourced from public datasets, satellite image databases, or self-collected channels. SAR images and optical images must be registered to ensure spatial consistency. The selected images should cover various terrain features, land cover types, and different imaging conditions to enhance the model's adaptability and generalization performance to complex remote sensing scenes.

[0035] Step 2: Based on the initial remote sensing image training data formed in Step 1, generate scene text descriptions for each group of SAR images and their corresponding original optical images, resulting in a diffusion model training dataset that includes SAR images, original optical images, text descriptions, and target style images.

[0036] Because SAR images and optical images differ significantly in imaging mechanisms and visual characteristics, directly using a diffusion model for conversion can easily lead to errors in land cover type identification. For example, roads and rivers have similar shapes in SAR images, making conversion prone to errors. Therefore, conditional prior information about scene content needs to be introduced during the diffusion model training process to guide the reasonable generation of image content. In addition to using the SAR image itself as a prior to provide spatial structure and image content information, known prior information such as geographical location and coordinates can be used to generate scene descriptions through natural language, serving as textual conditional input to the model to further enhance the model's understanding of image semantics and the accuracy of image generation.

[0037] During training, to reduce the workload of manually writing text descriptions, a multimodal language model can be used to generate scene descriptions in batches from images. To improve the accuracy and consistency of the text descriptions, initial text descriptions can be generated for the original optical images and SAR images respectively. Then, through semantic parsing, content comparison, and information fusion, more accurate text descriptions can be obtained, which serve as prior information for the scene in the training of the multimodal diffusion model.

[0038] In some applications, SAR image conversion requires generating optical images with a specific style. In such cases, a target style image can be introduced, and its style prior information can be extracted for style guidance during image generation. For applications that do not require a specified style, the target style image can be omitted to achieve a more general generation effect. During the training phase, the target style image can be selected from the training dataset as an optical image with a similar style but different content from the original optical image, and style features can be extracted using the style encoder described in step 3. During the inference phase, the target style image can be flexibly selected according to specific application requirements to control the style attributes of the generated image.

[0039] Step 3: Establish a multimodal conditional diffusion model. The multimodal conditional diffusion model uses the noise addition-denoising reconstruction process of optical images as the optimization objective. The core of the multimodal conditional diffusion model is a stable diffusion model, which introduces a parallel ControlNet branch and a decoupled cross-attention adapter.

[0040] The first step is to input the original optical image into the pre-trained VAE encoder and compress it into a latent representation. Then, Gaussian noise is gradually injected into the latent variables through a Markov chain with fixed steps to obtain a noisy latent representation with an approximate normal distribution, which serves as the initial input for the diffusion process.

[0041] The second step is to input the SAR image as structural prior information into the ControlNet branch to provide information on the overall spatial layout and contour conditions of the optical image to be generated.

[0042] The third step involves extracting structural semantic features from SAR images using a pre-trained image semantic encoder, extracting scene prior features from text descriptions using a pre-trained text encoder, and extracting image style features from target style images using a style encoder. The structural semantic features, scene prior features, and image style features are then uniformly embedded into the form of a token and injected into the cross-attention layer of the denoising UNet through a decoupled cross-attention adapter.

[0043] The fourth step is to use the latent representation output by the denoised Unet to generate a denoised optical image through the VAE decoder.

[0044] Step 4: Use the diffusion model training dataset obtained in Step 2 to train the multimodal conditional diffusion model established in Step 3, resulting in the trained multimodal conditional diffusion model. During training, the pre-trained parameters of the VAE encoder, denoising UNet, SAR image semantic encoder, and text encoder remain frozen and do not participate in updates. The ControlNet branch, style encoder, decoupled cross-attention module, and VAE decoder participate in parameter updates. The model predicts the noise components added to the latent representation of the original optical image and compares them with the injected known noise, calculating the noise prediction error to construct the noise prediction loss. Then, the latent features denoised by UNet are restored to the optical image through the VAE decoder and compared with the original optical image to calculate the image reconstruction error.

[0045] Step 5: Input the SAR image to be converted into the trained multimodal conditional diffusion model for inference. A text description of the scene prior information of the SAR image to be converted is generated. The text description of the SAR image to be converted, the SAR image to be converted, and the target style image are input into the trained multimodal conditional diffusion model obtained in Step 4 to obtain the converted optical image, such as... Figure 4 As shown.

[0046] The first step is to extract the scene prior information of the SAR image to be converted to obtain the text description of the SAR image to be converted.

[0047] The second step is model loading and initialization. The multimodal conditional diffusion model trained in step 4 is loaded, the number of denoising steps in the diffusion process is set, and random noise is initialized as the initial input to the denoising UNet network.

[0048] The third step involves introducing conditions and performing generation. The SAR image is first input into the ControlNet conditional control branch, and simultaneously, the SAR image is input into the SAR image semantic encoder to extract structural semantic features. The text description is used to extract scene prior features through the text encoder, and the target style image is selected and input into the style encoder to extract style features. Subsequently, the structural semantic features, scene prior features, and image style features are uniformly embedded into the form of tokens, and then injected into the cross-attention layer of the diffusion model's UNet through a decoupled cross-attention adapter.

[0049] The fourth step is to restore the denoised latent representation output by Unet to an optical image through a VAE decoder, which is then used as the optical image generated by SAR image conversion.

[0050] Example

[0051] This invention proposes a SAR image-to-optical image conversion method based on a multimodal conditional diffusion model. By introducing a multimodal guidance mechanism that includes textual language prompts, image content prompts, and image style prompts, the method significantly enhances the structural modeling and semantic alignment capabilities of the diffusion model in remote sensing scenarios, thereby improving the overall performance of the generated image in terms of controllability, realism, and semantic consistency. Figure 1 As shown, the specific implementation process is as follows:

[0052] Step 1: Acquire several sets of SAR images and corresponding raw optical images of the same scene to form a remote sensing image dataset. The acquired images must be registered to achieve pixel-by-pixel spatial correspondence. Simultaneously, the dataset should cover diverse geographical scenes, rich land cover types, and significant differences in imaging styles to enhance its representativeness and improve the model's generalization ability in complex remote sensing tasks.

[0053] In this embodiment, the following publicly available remote sensing image datasets can be used as typical representatives to meet the above-mentioned diversity requirements:

[0054] The SAR images in the SEN12 dataset were acquired from Sentinel-1 dual-polarization data, and the optical images were acquired from Sentinel-2 multispectral images. The dataset contains a total of 282,384 pairs of SAR and optical images, each pair being 256×256 pixels in size, covering the four seasons of spring, summer, autumn, and winter, and exhibiting good temporal and regional diversity.

[0055] The SAR images in the QXS-SAROPT dataset were acquired by the Gaofen-3 satellite in single-polarization mode, while the optical images were sourced from the Google Earth platform. The dataset contains 20,000 pairs of SAR and optical images, each pair measuring 256×256 pixels, covering several typical port cities.

[0056] The SAR2OPT dataset contains SAR images from the TerraSAR-X satellite and optical images from Google Earth. The dataset includes 2,076 pairs of SAR and optical images, each pair measuring 600×600 pixels, covering multiple urban areas in Asia. In this embodiment, each image is cropped into 256×256 tiles using a sliding window with a size of 256×256 and a step size of 256.

[0057] WHU-OPT-SAR is a semantically annotated optical SAR multimodal semantic segmentation dataset. The SAR images are from the Gaofen-3 satellite, and the optical images are from the Gaofen-1 satellite. This dataset focuses on Hubei Province, China, and contains 100 paired SAR and optical images, each with a size of 5536×3704 pixels. The semantic annotations in this dataset include seven categories: farmland, city, village, water, forest, road, and others. In this embodiment, each 5536×3704-pixel image is cropped into 256×256 slices using a sliding window with a size of 256×256 and a stride of 256.

[0058] Step 2: Based on the initial remote sensing image training data formed in Step 1, generate scene text descriptions for each group of SAR images and their corresponding original optical images, resulting in a diffusion model training dataset that includes SAR images, original optical images, text descriptions, and target style images.

[0059] In this embodiment, to generate scene text descriptions for images in batches, a publicly available remote sensing multimodal language model is used to form text descriptions from the original optical images and SAR image pairs. Taking the QXS-SAROPT dataset as an example, the specific steps include the following:

[0060] The first step is to use the SARChat model to generate a scene text description of the SAR image. The SAR image is used as input, and a prompt is set to ask a question.

[0061] Question: What do you observe in this SAR image?

[0062] Example of model output: In this image, we can observe the following: Many smallships are visible near the shore in the lower part of the image. A road can also be seen on the land.

[0063] (Translation: As you can see in the image below, there are many small boats on the shore. There is a road on land.)

[0064] The second step involves using the LHRS-Bot model to generate a text description of the original optical image. The original optical image is used as input, and a prompt is set to ask a question.

[0065] Enter an image and ask: What do you see from this optical image?

[0066] Model output answer: From this image, I can see a waterfront port area with the dock stretching along the upper edge of the water body. In the lower part of the image, many small boats or mooring structures are densely lined up near the shoreline. Several large buildings are located along the land side,accompanied by visible roads and supporting infrastructure in the surrounding area.

[0067] (Translation: This image shows a waterfront harbor area with piers extending along the upper edge of the water. The lower half of the shoreline is densely packed with small boats and mooring structures, while several large buildings stand on the land side, surrounded by roads and supporting infrastructure.)

[0068] The third step is to merge the text descriptions from the SAR and the original optical images to generate a unified scene description.

[0069] First, structured semantic parsing. Using language tools such as spaCy, semantic units are extracted from the text descriptions of the original optical and SAR images. For example, the SAR description "Many small ships...land" can be parsed as: ships, shore, lower part, road, land; the optical description "waterfront port area...infrastructure" can be parsed as: port, dock, water body, boats, shoreline, buildings, roads.

[0070] Subsequently, multimodal semantic collaborative processing is performed. To unify potential semantic ambiguities or conflicts across different modalities, the descriptions of SAR images are proofread and supplemented. This mainly includes the following three aspects.

[0071] Semantic consistency merging: When two modalities contain geographical entities with different expressions but similar semantics (such as "ship" and "boat"), they can be considered to have semantic consistency and be directly merged.

[0072] Modal difference record: The optically unique “buildings” are not included in the final description because they are not detected by SAR. They can be transferred to the input model as guidance information for the semantic coding features of SAR images.

[0073] Semantic conflict resolution: If the optical description "port" conflicts with the SAR description "land", the geographic truth is verified by semantic segmentation mask or OpenStreetMap, and the "port" that conforms to the scene attributes is retained first.

[0074] Finally, a unified description is generated. By integrating shared information from both modalities, logically coherent scene text is generated.

[0075] Fusion example:

[0076] Retained elements: boat, shore, lower part, road, port

[0077] Final description: In the lower half of the picture, you can see several smallboats docked on the shore, and there is a road extending from the pier to the waterfront port area.

[0078] (Translation: The lower half of the image shows several small boats moored on the shore, with a road extending from the dock to the seaside harbor area.)

[0079] After obtaining the optical image, the corresponding SAR image, and their unified language description, an optical image that is similar in color style and scene atmosphere but different in content is further selected as a style guidance map, forming a four-tuple data set: SAR image, original optical image, text description, and target style image. This structure can support image generation training tasks under multimodal guidance conditions and improve the model's ability to comprehensively perceive and control structural, semantic, and style information.

[0080] Step 3: Establish a multimodal conditional diffusion model. In this step, SAR images, image text descriptions, and target-style optical images are used as input information to the multimodal conditional guidance model. These are systematically decoupled into multiple dimensions of conditional constraints, including overall spatial layout and structural semantic features of the image, prior information about the image scene, and image style attributes. Through the synergistic guidance of these multi-dimensional conditions, the model's ability to model spatial structure and semantic alignment in remote sensing scenes is enhanced, thereby improving the overall performance of the final generated image in terms of controllability, realism, and semantic consistency. Figure 2As shown, the proposed model uses a stable diffusion generation model as its main structure. Building upon the existing text-based conditional prompts, it further introduces a ControlNet-based control branch and a decoupled attention adapter. The model primarily consists of a SAR image semantic encoder, an optical image style encoder, a text encoder, the stable diffusion model body, a ControlNet image conditional control branch, and a decoupled cross-attention adapter.

[0081] The first step is latent space noise addition and modeling. The basic principle of the diffusion model is to gradually reconstruct the target image from Gaussian noise through an iterative denoising process. This process includes two stages: forward diffusion and backward denoising. In the forward diffusion process, Gaussian noise is gradually injected into the original image using a Markov chain with fixed steps, perturbing the image to a representation close to the labeled Gaussian distribution. In the backward denoising process, a learnable neural network model (such as Unet) is used to gradually denoise the noisy latent variables, restoring the data distribution, thereby generating a clear image from the Gaussian noise. Compared with traditional generative adversarial networks (GANs), the diffusion model training process is more standardized and effectively avoids problems such as model collapse.

[0082] To avoid the enormous computational overhead of performing the diffusion process in high-resolution pixel space and to improve image generation efficiency, this invention employs a stable diffusion generation model as its main structure. This method utilizes a pre-trained variational autoencoder (VAE) to compress high-dimensional images into a low-dimensional latent space. While significantly reducing computational complexity, it effectively preserves the structural and semantic details of the image, achieving a superior balance between efficiency and quality in image generation.

[0083] In this embodiment, the task of converting SAR images to optical images involves inputting the original optical image x0 into the pre-trained VAE encoder E in the stable diffusion model, compressing it into a latent representation z0. Subsequently, Gaussian noise is gradually injected into the latent variables through a Markov chain with a fixed step T, obtaining a noisy latent representation z0 that approximates a normal distribution. T , which serves as the input for the denoising Unet.

[0084] The second step is structural semantic condition guidance. The SAR image is input into the ControlNet branch as prior structural semantic information to provide the overall spatial layout and contour information of the image to be generated. For example... Figure 3 As shown, the noisy latent representation is input to the denoising Unet. Suppose F(·θ) is a sub-module of the pre-trained denoising Unet network with parameter θ, which transforms the input feature map x into another feature map y.

[0085] To add the ControlNet branch to the denoising UNet network, the parameters θ of the original module are frozen, and the module is copied as a trainable copy with parameters θ. c The trainable copy takes a SAR image C as input. The trainable copy is connected to the original frozen model structure via zero-convolutional layers, denoted as L(·;·). The two zero-convolutional layers are 1×1 convolutional layers with weights and biases initialized to zero, and their parameters are denoted as θ. l1 and θ l2 .

[0086] y c =F(x;θ)+L(F(x+L(C;θ)) l1 );θ c );θ l2 )

[0087] The frozen parameters of the stable diffusion model retain the pre-trained parameters trained on billions of images. Trainable copies can be reused to build deep, robust, and powerful backbone networks to handle different input conditions. However, ControlNet lacks the ability to model semantic information, which can easily lead to semantic errors or color deviations in the generated images.

[0088] The third step is multimodal conditional guided injection. In traditional diffusion models, a cross-attention mechanism is typically used to inject textual description information into the denoising UNet to guide image content generation. However, relying solely on textual conditions often fails to adequately control complex attributes such as spatial layout, object pose, and structural morphology of the image. Therefore, this invention introduces a multimodal conditional guided mechanism, injecting the structural semantic information of the SAR image, the scene information of the textual description, and the style information of the target style image into the denoising UNet network through a decoupled cross-attention adapter, thereby enhancing the model's structural awareness and style consistency expression capabilities.

[0089] The specific operating procedure is as follows:

[0090] First, the SAR image x c The input is the pre-trained SAR image semantic encoder CWSAM, denoted as τ. c Then through a trainable linear layer l c Extracting structural semantic features c c This process is represented as: c c =l c (τ c (x c )). Describe the text x t Input the pre-trained text encoder CLIP, denoted as τ. t Then through a trainable linear layer l tExtracting prior features c of the scene t This process is represented as: c t =l t (τ t (x t )). Target style image x s The input style encoder is VGG16, denoted as τ. s Then through a trainable linear layer l s Extracted visual style features c s This process is represented as: c s =l s (τ s (x s )).

[0091] Then, the above three types of features are converted into token representations and cross-attention is calculated with the features of the denoising UNet network respectively, ensuring that different modal information is guided independently at different semantic levels, thereby achieving fine-grained collaborative control of structure, semantics and style.

[0092] Given the query features Z and the prior features c of the text scene in the denoised Unet network. t Calculate the cross-attention output of text features Where Q = ZW q , These are the query, key, and value matrix for the attention operation, respectively, W. q , It is the weight matrix of the trainable linear projection layer in the cross-attention module.

[0093] Given the query features Z in the denoising Unet network and the SAR image structural semantic features c c Calculate the cross-attention output. Where Q = ZW q , These are the query for attention operations, the key and value matrix, and W. q , It is the weight matrix of the trainable linear projection layer in the cross-attention module.

[0094] Given the query features Z and style features τ in a denoised Unet network s Calculate the image style cross-attention output Where Q = ZW q , These are the query for attention operations, the key and value matrix, and W. q , It is the weight matrix of the trainable linear projection layer in the cross-attention module.

[0095] Finally, the fusion output of decoupled cross-attention can be expressed as:

[0096] Z new =Z c +Z t +Z s

[0097] Through the aforementioned multimodal conditional injection mechanism, this model can significantly improve the overall performance of generated images in terms of structural fidelity, semantic consistency, and style naturalness, making it suitable for high-quality remote sensing image generation tasks in complex scenarios.

[0098] The fourth step involves analyzing the latent variable z′ after Unet prediction and noise removal. T Then, the optical image is restored from the original image using the VAE decoder D, resulting in...

[0099] Step 4: Training process of the multimodal conditional diffusion model. For example... Figure 2 As shown, during the denoising process, the UNet network predicts the noise components in the latent representation based on the aforementioned multimodal conditional information, compares them with the injected known noise, calculates the noise prediction error, and constructs the noise prediction loss. Simultaneously, the denoised latent representation is restored to an image using a VAE decoder, and the reconstruction error is calculated by comparing it with the original optical image. This process ensures the accuracy of content restoration in the generated image and also provides effective optimization in semantic consistency and style matching, further improving the realism, detail fidelity, and overall expressiveness of the generated image.

[0100] In this embodiment:

[0101] Given a latent representation z0 of an input image, the image diffusion algorithm gradually adds noise to the image and generates a noisy representation z. T , where T represents the number of times noise is added. Given a set of conditions, time step T, SAR image spatial structure control C, and text prompt c. t SAR image semantic c c Style image conditions c s Training model ε θ Prediction is added to the noise characterization z T The noise in the data. The noise prediction function can be expressed as:

[0102]

[0103] The latent variable z′ after Unet prediction and noise removal T Then, the image is restored to its final optical image using the VAE decoder D. The loss calculated using the L2 loss function and the input image can be expressed as:

[0104]

[0105] Ultimately, the overall loss function can be expressed as L = L n +L p

[0106] Step 5: Input the SAR image to be converted into the trained multimodal conditional diffusion model for inference. For example... Figure 4 As shown, a scene prior text description is generated for the SAR image to be converted. The text description of the SAR image to be converted, the SAR image to be converted, and the target style image are input into the multimodal conditional diffusion model trained in step 4 to obtain the converted optical image, as shown. Figure 4 As shown. The specific process includes the following sub-steps:

[0107] The first step is to extract the scene prior information of the SAR image to be converted to obtain the text description of the SAR image to be converted.

[0108] The second step is model loading and initialization. The multimodal conditional diffusion model trained in step 4 is loaded, the number of denoising steps in the diffusion process is set, and random noise is initialized as the initial input to the denoising UNet network.

[0109] The third step involves introducing conditions and performing denoising. The SAR image is first input into the ControlNet conditional control branch, and simultaneously input into the SAR image semantic encoder to extract structural semantic features. The text description is then used to extract scene prior features through the text encoder, and the target style image is selected and input into the style encoder to extract style features. Subsequently, the structural semantic features, scene prior features, and image style features are uniformly embedded into the form of tokens, and then injected into the cross-attention layer of the UNet network of the diffusion model through a decoupled cross-attention adapter.

[0110] The fourth step involves using a VAE decoder to reconstruct the optical image from the denoised latent representation output by the Unet network, which is then used as the optical image generated by SAR image conversion. Several improvements can be made without departing from the principles of this invention, and these should also be considered within the scope of protection of this invention.

[0111] In summary, the above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for converting SAR images to optical images based on a multimodal conditional diffusion model, characterized in that... The steps are as follows: Step 1: Acquire several SAR images and raw optical images of the same scene to form initial remote sensing image training data; Step 2: Based on the initial remote sensing image training data formed in Step 1, text descriptions are generated for the scene prior information of each group of SAR images and original optical images, resulting in a diffusion model training dataset including SAR images, original optical images, text descriptions, and target style images. Step 3: Establish a multimodal conditional diffusion model; Step 4: Use the diffusion model training dataset obtained in Step 2 to train the multimodal conditional diffusion model established in Step 3, and obtain the trained multimodal conditional diffusion model. Step 5: Create a text description of the scene prior information for the SAR image to be converted. Input the text description of the SAR image to be converted, the SAR image to be converted, and the target style image into the multimodal conditional diffusion model trained in Step 4 to obtain the converted optical image.

2. The SAR image to optical image conversion method based on a multimodal conditional diffusion model according to claim 1, characterized in that: In step 1, the data sources for SAR images and raw optical images of the same scene include publicly available remote sensing image datasets, satellite image databases, or paired images acquired independently.

3. The SAR image to optical image conversion method based on a multimodal conditional diffusion model according to claim 1, characterized in that: In step 2, the text descriptions in the diffusion model training dataset are used to provide prior information about the scene, the SAR images are used to provide overall spatial layout and structural semantic information, and the target style images are used to provide style information.

4. The SAR image to optical image conversion method based on a multimodal conditional diffusion model according to claim 1, characterized in that: In step 3, the established multimodal conditional diffusion model takes the noise addition-denoising reconstruction process of optical images as the optimization objective. The core of the multimodal conditional diffusion model is a stable diffusion model, which introduces parallel ControlNet branches and a decoupled cross-attention mechanism.

5. The SAR image to optical image conversion method based on a multimodal conditional diffusion model according to claim 4, characterized in that: The working process of the multimodal conditional diffusion model is as follows: The first step is to input the original optical image into the pre-trained VAE encoder and compress it into a latent representation. Then, Gaussian noise is gradually injected into the latent variables through a Markov chain with fixed steps to obtain a noisy latent representation with an approximate normal distribution, which serves as the initial input for the diffusion process. The second step is to input the SAR image as structural prior information into the ControlNet branch to provide information on the overall spatial layout and contour conditions of the optical image to be generated. Third, structural semantic features are extracted from SAR images through a semantic encoder, scene prior features are extracted from text descriptions through a text encoder, and image style features are extracted from target style images through a style encoder. The structural semantic features, scene prior features, and image style features are uniformly embedded into the form of a token, and then injected into the cross attention layer in the denoising UNet through a decoupled cross attention adapter. The fourth step involves predicting the noise component in the latent representation and comparing it with the injected known noise to calculate the noise prediction error. Simultaneously, the predicted noise is subtracted from the noisy latent representation to obtain the denoised latent representation, which is then restored to an optical image using a VAE decoder. The reconstruction error is then calculated by comparing the denoised latent representation with the original optical image.

Images