SAR image-to-optical image conversion method based on multi-mode conditional diffusion model

Abstract

The invention provides a method for converting an SAR image into an optical image based on a multi-mode conditional diffusion model. The method comprises the following steps: firstly, collecting SAR image and optical image data with a pairing relationship, and ensuring that a sample has diversity in the aspects of scene types, target structures and image styles; secondly, an image-language multi-modal model is adopted to generate text descriptions of SAR and optical images respectively, unified language descriptions are constructed through semantic analysis and fusion, and a light-SAR-text three-modal training sample set containing structure, semantic and style information is formed; according to the method, a denoising diffusion model guided by a multi-modal condition is designed and trained, a noise adding-denoising reconstruction process of an original optical image is taken as an optimization target, an SAR image, language description and a style image are introduced as multi-modal prompt conditions, and the expression of the generated image in the aspects of structure reduction, semantic alignment and style presentation is comprehensively guided.

Description

Technical Field

[0001] The present invention belongs to the technical field of intersection of remote sensing and computer vision, and particularly relates to a method for converting remote sensing SAR images into optical images based on a multimodal conditional diffusion model. Background Art

[0002] Synthetic Aperture Radar (SAR) imagery, with its ability to penetrate clouds and fog and provide all-day, all-weather imaging, has been widely used in key areas such as natural disaster monitoring, military reconnaissance, and resource exploration, possessing irreplaceable strategic value. However, because SAR imaging is based on active microwave radar, its images exhibit unique grayscale distribution and scattering texture characteristics, significantly different from the natural light imaging results to which the human visual system is accustomed. This modal difference poses significant challenges to the intuitive interpretation and rapid deciphering of SAR images. Therefore, converting SAR images into optical images that are more consistent with human perception has become an important research direction in the field of intelligent remote sensing image processing.

[0003] In recent years, with breakthroughs in image generation and modality conversion using generative diffusion models, SAR-to-optical image conversion methods based on diffusion models have shown great potential. However, most existing methods employ a simple channel-wise splicing approach to jointly input SAR and optical images into a diffusion network. This approach suffers from significant limitations: First, SAR and optical images exhibit fundamental differences in their physical imaging mechanisms. For example, SAR uses slant-range geometry and is subject to speckle noise, while optical images are based on central projection and possess natural texture features. Simple splicing can easily lead to inter-modal feature confusion, resulting in structural distortion, semantic misalignment, and color distortion in the generated images. Second, such methods generally lack the ability to explicitly model the unique semantic and geometric structure of SAR images, such as the categorical information embedded in ground object scattering characteristics or the spatial topological changes introduced by radar shadows. Performing modality fusion solely at the input layer makes it difficult to establish an accurate and consistent cross-modal mapping. Furthermore, existing end-to-end training schemes typically require parameter tuning of the entire diffusion model. Given the scarcity of SAR-optical paired samples, training is expensive, and the models are prone to overfitting and lack generalization.

[0004] The core of the aforementioned problem lies in the fact that existing technologies fail to effectively establish a collaborative control mechanism between the spatial structure of SAR images and the semantics of optical images, lacking sufficient multimodal semantic guidance and spatial constraints. Therefore, it is urgent to propose a multimodal conditional diffusion model that integrates language cues and image cues. By explicitly injecting semantic and structural prior information, it can improve the structural consistency, visual realism, and semantic expressiveness of the generated images, thereby achieving high-fidelity and highly semantically consistent SAR image visualization. Summary of the Invention

[0005] To overcome the shortcomings of existing remote sensing image generation techniques, such as insufficient semantic guidance and imprecise image style control, this paper proposes a SAR image-to-optical image conversion method based on a multimodal conditional diffusion model. By introducing a multimodal guidance mechanism that includes textual language cues, image content cues, and image style cues, the diffusion model's structural modeling and semantic alignment capabilities in remote sensing scenarios are significantly enhanced, thereby improving the overall performance of the generated images in terms of controllability, realism, and semantic consistency.

[0006] A SAR image to optical image conversion method based on a multimodal conditional diffusion model, the steps are as follows:

[0007] Step 1: Collect a remote sensing image dataset. Collect a dataset containing SAR images and their corresponding optical images, ensuring that the samples have diverse scenes, diverse object types, and different styles. Data sources include public remote sensing image datasets, satellite image databases, or paired images collected independently. Additionally, collect a light-SAR semantic dataset containing semantic segmentation labels, using the semantic categories and spatial structure information as a linguistic description of the image content or as reference auxiliary information.

[0008] Step 2: Construct a multimodal training dataset of SAR images, optical images, and text. This aims to create a trimodal training dataset that fuses structural information (SAR images), semantic information (text descriptions), and style information (optical images). A large remote sensing image-language multimodal model is used to generate text descriptions for both SAR and optical images, extracting their semantic features. Subsequently, semantic parsing and fusion techniques are combined to semantically align and integrate the generated texts, forming a unified and comprehensive language description, thereby enhancing semantic consistency across modalities. Furthermore, for each sample group, an optical image with a similar style but different content to the original optical image is selected as a style guide. Ultimately, each training sample group consists of: the original optical image, the corresponding SAR image, the fused text description, and the style image.

[0009] Step 3: Construction and Training of a Multimodal Conditional Diffusion Model (MM-DM). A multimodal conditional diffusion generative model (MM-DM) is designed and constructed. This model optimizes the denoising-denoising reconstruction process of the original optical image. It combines multimodal conditional information, such as SAR images, textual descriptions, and style images, to guide the expression of the generated image at the content and style levels. The model uses a stable diffusion model (Stable Diffusion) as the main framework, introducing parallel ControlNet branches and a decoupled cross-attention mechanism to process structural guidance, textual semantics, image content, and image style cues, respectively, and inject multimodal conditional information into the denoising UNet structure. End-to-end iterative training is performed under the joint constraints of a noise prediction loss and an overall perceptual loss. Iterative training is performed using the multimodal training dataset constructed in Step 2. Model parameters are optimized by combining noise prediction, reconstruction, and perceptual losses to improve the model's image generation quality in complex remote sensing scenarios.

[0010] Step 4: SAR to Optical Image Inference Generation. During the inference phase, a textual description of the SAR image is generated and a target style image is selected. Subsequently, the trained MM-DM model is used, conditional on the SAR image, the textual description, and the style image. Random noise is initialized and a diffusion inversion process is performed for a set number of steps to generate an optical image output that conforms to the semantic and style constraints.

[0011] Specifically, the steps for collecting remote sensing image datasets in step 1 are as follows: A diverse dataset of paired SAR images and corresponding optical images is obtained from public datasets, satellite image databases, or self-collected datasets. In addition to datasets for conversion or fusion purposes, a dataset of optical SAR semantic segmentation is also collected. The semantic segmentation labels contain semantic categories and spatial structure information, which can also serve as a reference for image language descriptions.

[0012] The construction of the multimodal training dataset of SAR image, optical image and text described in step 2 specifically includes the following steps:

[0013] The first step is to generate image language descriptions. Using a large multimodal model of remote sensing image language, we generate corresponding image text descriptions for SAR and optical images, capturing their respective scene semantics and target information.

[0014] The second step is structured semantic parsing. The image description text is structured and parsed. Language understanding tools are used to convert the description into semantic fragments containing feature entities, attributes, and spatial relationships, thus constructing a semantic structure.

[0015] The third step is semantic alignment and conflict detection. Semantic segments from the SAR and optical images are aligned and analyzed. If the descriptions share common ground objects or spatial relationships, they are considered semantically consistent. If one description mentions a feature but not the other, this is recorded as modality discrepancy. If there is a significant semantic conflict, a weighted judgment and conflict resolution can be performed using auxiliary image features or model confidence. This step constructs a semantic comparison table, providing a basis for generating fused descriptions.

[0016] The fourth step is to generate a unified linguistic description. Based on alignment and difference analysis, the shared and complementary information between the two modal descriptions is integrated, and a language model (such as GPT) is used to guide the generation of a natural, fluent, and semantically complete fusion language description. Structured semantic information can be input to guide the language model to generate unified text, ensuring language consistency and expressiveness.

[0017] The fifth step is sample construction. Based on the optical image, the corresponding SAR image, and the fused text description, an additional optical image with similar color and scene style but different content is selected as a style guide. This is combined with the aforementioned three-modal data to form a complete set of training samples.

[0018] Step 3: Propose a design and training method based on the multimodal conditional denoising diffusion model (MM-DM). The specific steps are:

[0019] The first step is to add noise to the latent space and model it. The original optical image is fed into a pre-trained VAE encoder and compressed into a latent representation. Subsequently, Gaussian noise is gradually injected into the latent variable through a fixed-step Markov chain, obtaining a noisy latent representation with a near-normal distribution, which serves as the initial input to the diffusion process.

[0020] The second step is structural control guidance. The SAR image is input into the ControlNet branch as structural prior information to provide the overall spatial structure and contour information of the image to be generated, enhancing the control over image layout and ground feature morphology.

[0021] The third step involves multimodal semantic and style-guided injection. A semantic encoder extracts structural semantic features from SAR images, a text encoder extracts linguistic semantic features from text descriptions, and a style encoder extracts stylistic features from optical style images. These three types of features are uniformly embedded in token form and injected into multiple cross-attention layers of the UNet. By decoupling the cross-attention mechanism, various conditions are applied at different semantic levels, achieving fine-grained control of image content and style.

[0022] The fourth step is noise prediction and image reconstruction. During the denoising process, the UNet network predicts the noise component in the latent representation based on the aforementioned multimodal conditional information, compares it with the injected known noise, calculates the noise prediction error, and constructs the noise prediction loss. Simultaneously, the denoised latent representation is restored to an image through the VAE decoder, and the reconstruction error and perceptual error are jointly calculated with the original optical image. This process ensures the accuracy of the generated image's content restoration and provides effective optimization in terms of semantic consistency and style matching, further improving the realism, detail fidelity, and overall expressiveness of the generated image.

[0023] Step 4: SAR to optical image inference generation. This step aims to use the trained multimodal conditional diffusion model (MM-DM) to generate optical images with specified style features from SAR images. The specific process includes the following three sub-steps:

[0024] The first step is to generate a SAR image language description. The SAR image to be processed is analyzed and a corresponding language text description is generated as one of the semantic guidance conditions.

[0025] Step 2: Model loading and initialization. Load the MM-DM model trained in step 3, set the number of denoising steps in the diffusion process, and initialize random noise as the initial input of the denoising UNet network.

[0026] The third step is to introduce the conditions and perform generation. A representative target style optical image is selected as the style guidance condition. The SAR image and its language description are simultaneously input into the structural and semantic guidance channels, respectively. Multi-step denoising inference is performed using the MM-DM model, ultimately generating an optical image that is consistent with the target condition in terms of structure, semantics, and style.

[0027] Compared to existing technologies, the present invention offers the following advantages: It proposes an innovative SAR image-to-optical image conversion method based on multimodal conditional guidance. By fusing SAR images, optical images, and their corresponding language descriptions, it constructs a multimodal information guidance mechanism that integrates structure, semantics, and style. This method decouples and models multimodal cues and dynamically injects them into each denoising iteration of the diffusion generation process, comprehensively guiding the image generation process from three perspectives: semantic understanding, spatial structure reconstruction, and style presentation. This mechanism significantly overcomes the bottlenecks of traditional SAR image conversion methods in semantic expression and structural restoration, enhancing the model's structural modeling, semantic alignment, and style control capabilities for remote sensing image generation tasks. This effectively improves the controllability, realism, and semantic consistency of the generated images, breaking through the information expression bottleneck of traditional methods during modal conversion. The technical solution presented in this paper has broad application prospects, particularly in tasks such as military reconnaissance, disaster response, urban planning, remote sensing reconstruction, and multi-source image fusion. It is of great significance in improving remote sensing intelligent processing and image usability. The present invention proposes a SAR image-to-optical image conversion method based on a multimodal conditional diffusion model. First, paired SAR and optical image data are collected to ensure diversity in scene type, target structure, and image style. Second, an image-language multimodal model is used to generate textual descriptions of the SAR and optical images, respectively. Through semantic parsing and fusion, a unified language description is constructed, forming a trimodal training sample set of light-SAR-text that contains structural, semantic, and style information. This paper designs and trains a multimodal condition-guided denoising diffusion model (MM-DM). Using the denoising-denoising reconstruction process of the original optical image as the optimization objective, the SAR image, language description, and style image are introduced as multimodal cues to comprehensively guide the generated image's expression in terms of structural restoration, semantic alignment, and style presentation. This model is jointly trained using a noise prediction loss, a reconstruction loss, and a perceptual loss, effectively improving the realism, detail fidelity, and semantic consistency of the generated image. In the inference phase, the trained MM-DM model is fed with the SAR image, its generated textual description, and the target style image. An optical image with the specified style features is generated through a diffusion sampling process. By infusing multimodal semantic and structural information into each denoising iteration, this method significantly enhances the model's structural modeling and semantic control capabilities in remote sensing image generation, improving the controllability and realism of the generated results. This method is applicable to tasks such as multi-source remote sensing image fusion, reconstruction, and style transfer, and is particularly well-suited for achieving high-quality, controllable SAR-to-optical image conversion in remote sensing scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is the overall flow chart of this method;

[0029] Figure 2 Schematic diagram of the training process based on the multimodal conditional diffusion model of the present invention;

[0030] Figure 3 Schematic diagram of the ControlNet conditional control mechanism provided by an embodiment of the present invention;

[0031] Figure 4 Schematic diagram of the reasoning process based on the multimodal conditional diffusion model of the present invention. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the present invention will be further described below in conjunction with the drawings and embodiments in the embodiments of the present application. The present invention includes but is not limited to the following embodiments.

[0033] A method for converting SAR images to optical images based on a multimodal conditional diffusion model includes the following steps:

[0034] Step 1: Obtain several sets of SAR images and corresponding raw optical images of the same scene as initial remote sensing image training data, ensuring that the samples exhibit scene diversity, object type diversity, and style differences. Image data can come from public datasets, satellite image databases, or self-collected sources. SAR and optical images must be registered to ensure spatial alignment. The selected images should cover a variety of terrain, landforms, object types, and imaging conditions to enhance the model's adaptability and generalization performance to complex remote sensing scenarios.

[0035] Step 2: Based on the initial remote sensing image training data formed in step 1, generate scene text descriptions for each set of SAR images and the corresponding original optical images, and obtain a diffusion model training dataset including SAR images, original optical images, text descriptions, and target style images.

[0036] Due to the significant differences in imaging mechanisms and visual properties between SAR and optical images, direct conversion using a diffusion model can easily lead to misidentification of feature types. For example, roads and rivers have similar shapes in SAR images, making conversion errors prone to errors. Therefore, conditional prior information about scene content must be incorporated into diffusion model training to guide the proper generation of image content. In addition to using the SAR image itself as a prior to provide spatial structure and image content, known prior information such as geographic location and coordinates can be used to generate scene descriptions using natural language and serve as textual input to the model, further enhancing the model's understanding of image semantics and generation accuracy.

[0037] During training, to reduce the workload of manually writing text descriptions, a large multimodal language model can be used to generate batch scene descriptions for images. To improve the accuracy and consistency of text descriptions, initial text descriptions can be generated for the original optical and SAR images. Then, through semantic parsing, content comparison, and information fusion, more accurate text descriptions can be obtained, which serve as scene prior information in multimodal diffusion model training.

[0038] In some application scenarios, the conversion of SAR images requires the generation of optical images with a specific style. In this case, a target style image can be introduced to extract its style prior information for style guidance during the image generation process. For application scenarios that do not require a specific style, the target style image can be omitted to achieve a more versatile generation effect. During the training phase, the target style image can be selected from the training dataset, using an optical image with similar style but different content to the original optical image. The style encoder described in step 3 is used to extract style features. During the inference phase, the target style image can be flexibly selected based on the specific application requirements to control the style properties of the generated image.

[0039] Step 3: Establish a multimodal conditional diffusion model. This model optimizes the denoising and denoising reconstruction of optical images. The model is based on a stable diffusion model, which introduces a parallel ControlNet branch and a decoupled cross-attention adapter.

[0040] In the first step, the original optical image is input into the pre-trained VAE encoder and compressed into a latent representation. Then, Gaussian noise is gradually injected into the latent variable through a fixed-step Markov chain to obtain a noisy latent representation with an approximately normal distribution, which serves as the initial input of the diffusion process.

[0041] In the second step, the SAR image is input into the ControlNet branch as structural prior information to provide the overall spatial layout and contour condition information of the optical image to be generated;

[0042] In the third step, the SAR image is decoded using a pre-trained image semantic encoder to extract structural semantic features, the text description is decoded using a pre-trained text encoder to extract scene prior features, and the target style image is decoded using a style encoder to extract image style features. The structural semantic features, scene prior features, and image style features are then uniformly embedded into a token form and injected into the cross-attention layer of the denoising UNet via a decoupled cross-attention adapter.

[0043] In the fourth step, the potential representation output by the denoising Unet is passed through the VAE decoder to generate the denoised optical image.

[0044] Step 4: Use the diffusion model training dataset obtained in step 2 to train the multimodal conditional diffusion model established in step 3 to obtain a trained multimodal conditional diffusion model. During the training process, the pre-trained parameters of the VAE encoder, denoising Unet, SAR image semantic encoder, and text encoder remain frozen and do not participate in the update. The ControlNet branch, style encoder, decoupled cross-attention module, and VAE decoder participate in the parameter update. The model predicts the noise component added to the latent representation loaded in the original optical image, compares it with the injected known noise, and calculates the noise prediction error to construct the noise prediction loss. Then, the latent features denoised by UNet are restored to the optical image through the VAE decoder and compared with the original optical image to calculate the image reconstruction error.

[0045] Step 5: Input the trained multimodal conditional diffusion model to the SAR image to be converted for inference. A text description of the scene prior information of the SAR image to be converted is formed. The text description of the SAR image to be converted, the SAR image to be converted, and the target style image are input into the trained multimodal conditional diffusion model obtained in step 4 to obtain the converted optical image, such as Figure 4 shown.

[0046] In the first step, the scene prior information of the SAR image to be converted is extracted to obtain a text description of the SAR image to be converted.

[0047] The second step is model loading and initialization. Load the multimodal conditional diffusion model trained in step 4, set the number of denoising steps in the diffusion process, and initialize random noise as the initial input of the denoising UNet network.

[0048] In the third step, conditions are introduced and generation is performed. The SAR image is first fed into the ControlNet conditional control branch. The SAR image is also fed into the SAR image semantic encoder to extract structural semantic features. The text description is fed into the text encoder to extract scene prior features. The target style image is then fed into the style encoder to extract style features. The structural semantic features, scene prior features, and image style features are then uniformly embedded into tokens and injected into the cross-attention layer of the diffusion model's UNet via a decoupled cross-attention adapter.

[0049] In the fourth step, the denoised potential representation output by Unet is restored to an optical image through the VAE decoder, which is used as the optical image generated by SAR image conversion.

[0050] Example

[0051] This paper proposes a SAR image to optical image conversion method based on a multimodal conditional diffusion model. By introducing a multimodal guidance mechanism of text language prompts, image content prompts, and image style prompts, the structural modeling and semantic alignment capabilities of the diffusion model in remote sensing scenarios are significantly enhanced, thereby improving the comprehensive performance of the generated image in terms of controllability, realism, and semantic consistency. Figure 1 As shown, the specific implementation process is as follows:

[0052] Step 1: Acquire several sets of SAR images and corresponding raw optical images of the same scene to form a remote sensing image dataset. The acquired images must be registered to ensure pixel-by-pixel spatial correspondence. Furthermore, the dataset should cover diverse geographic scenes, a rich variety of ground object types, and significant differences in imaging styles to enhance the representativeness of the dataset and improve the model's generalization capabilities for complex remote sensing tasks.

[0053] In this embodiment, the following public remote sensing image datasets can be used as typical representatives to meet the above diversity requirements:

[0054] The SEN12 dataset uses Sentinel-1 dual-polarization SAR images and Sentinel-2 multispectral optical images. The dataset contains 282,384 pairs of SAR and optical images, each measuring 256×256 pixels. The images cover spring, summer, autumn, and winter, demonstrating good temporal and regional diversity.

[0055] The QXS-SAROPT dataset uses SAR images acquired by the Gaofen-3 satellite in single-polarization mode, while optical images are collected from Google Earth. The dataset contains 20,000 pairs of SAR and optical images, each measuring 256 × 256 pixels, covering several typical port cities.

[0056] The SAR2OPT dataset contains SAR images from the TerraSAR-X satellite, and optical images from Google Earth. The dataset includes 2,076 pairs of SAR and optical images, each measuring 600×600 pixels, covering multiple urban areas in Asia. In this example, each image was cropped into 256×256 slices using a sliding window with a step size of 256.

[0057] WHU-OPT-SAR is a semantically annotated multimodal semantic segmentation dataset for optical SAR (SAR) images. The SAR images are from the Gaofen-3 satellite, and the optical images are from the Gaofen-1 satellite. This dataset, based on Hubei Province, China, contains 100 paired SAR and optical images with an image size of 5536 × 3704 pixels. The semantic annotations in this dataset cover seven categories: farmland, urban, village, water, forest, road, and other. In this example, each 5536 × 3704 image is cropped into 256 × 256 slices using a sliding window with a step size of 256.

[0058] Step 2: Based on the initial remote sensing image training data formed in step 1, generate scene text descriptions for each set of SAR images and the corresponding original optical images, and obtain a diffusion model training dataset including SAR images, original optical images, text descriptions, and target style images.

[0059] In this example, to batch generate scene text descriptions for images, a publicly available remote sensing multimodal language model is used to generate text descriptions for pairs of raw optical and SAR images. Using the QXS-SAROPT dataset as an example, the following sub-steps are included:

[0060] The first step is to use the SARChat model to generate a text description of the SAR image scene. The SAR image is used as input and prompts are set to ask questions.

[0061] Input question: What do you observe in this SAR image?

[0062] Example model output: In this image, we can observe the following: Many smallships are visible near the shore in the lower part of the image. A road can also be seen on the land.

[0063] (Chinese translation: In the picture below, you can see many small boats on the shore. There is a road on the land.)

[0064] The second step is to use the LHRS-Bot model to generate a text description of the original optical image. The original optical image is used as input and prompts are set to ask questions.

[0065] Input an image and ask: What do you see from this optical image?

[0066] Model output answer: From this image, I can see a waterfront port area with the dock stretching along the upper edge of the water body. In the lower part of the image, many small boats or mooring structures are densely lined up near the shoreline. Several large buildings are located along the land side,accompanied by visible roads and supporting infrastructure in the surrounding area.

[0067] (Chinese translation: From this image, I can see a waterfront port area with a pier extending along the upper edge of the water. Many small boats or mooring structures are densely packed near the lower coastline. There are several large buildings on the land side, and roads and supporting infrastructure can be seen around them.)

[0068] The third step is to fuse the text descriptions of the collated SAR and original optical images to generate a unified scene description.

[0069] First, structured semantic parsing is performed. Using language tools such as spaCy, semantic units are extracted from the text descriptions of the original optical and SAR images. For example, the SAR description "Many small ships...land" can be parsed into: ships, shore, lower part, road, land; while the optical description "waterfront port area...infrastructure" can be parsed into: port, dock, water body, boats, shoreline, buildings, roads.

[0070] Subsequently, multimodal semantics are collaboratively processed. To unify any semantic ambiguities or conflicts that may exist between different modalities, the SAR image descriptions are proofread and supplemented. This primarily involves the following three aspects.

[0071] Semantic consistency merging: When two modal descriptions contain entities with different expressions but similar semantics (such as "ship" and "boat"), they can be considered semantically consistent and directly merged;

[0072] Modal difference recording: The optically unique “buildings” are not detected by SAR and are not included in the final description for the time being. However, they can be transferred to the input model to guide the semantic encoding features of the SAR image.

[0073] Semantic conflict resolution: If the optical description "port" conflicts with the SAR description "land", the geographic truth is verified through semantic segmentation masks or OpenStreetMap, and the "port" that matches the scene attributes is retained first.

[0074] Finally, unified description generation is achieved, fusing the shared information of the two modalities to generate a logically coherent scene text.

[0075] Fusion example:

[0076] Retained elements: boat, shore, lower part, road, port

[0077] Final description: In the lower half of the picture, you can see several smallboats docked on the shore, and there is a road extending from the pier to the waterfront port area.

[0078] (Chinese translation: In the lower half of the picture, you can see several small boats docked at the shore, and there is a road extending from the dock to the seaside port area.)

[0079] After obtaining an optical image, its corresponding SAR image, and its unified language description, a style-guided image with similar color style and scene atmosphere but different content is selected. This constructs a four-tuple of data: the SAR image, the original optical image, the text description, and the target style image. This structure supports image generation training tasks under multimodal guidance and enhances the model's ability to comprehensively perceive and control structural, semantic, and style information.

[0080] Step 3: Establish a multimodal conditional diffusion model. In this step, SAR images, image text descriptions, and target style optical images are input into the model as multimodal conditional guidance information, and are systematically decoupled into conditional constraints in multiple dimensions, such as the overall spatial layout and structural semantic features of the image, image scene prior information, and image style attributes. Through the collaborative guidance of multi-dimensional conditions, the model's spatial structure modeling ability and semantic alignment ability in remote sensing scenes are enhanced, thereby improving the comprehensive performance of the final generated image in terms of controllability, realism, and semantic consistency. Figure 2As shown in the figure, the proposed model uses the Stable Diffusion generative model as its main structure. Based on the existing text-conditional prompt, it further introduces the ControlNet structure control branch and the decoupled attention adapter. The model mainly consists of a SAR image semantic encoder, an optical image style encoder, a text encoder, the Stable Diffusion model main body, the ControlNet image conditional control branch, and the decoupled cross attention adapter.

[0081] The first step is to add noise to the latent space and model it. The basic principle of the diffusion model is to gradually reconstruct the target image from Gaussian noise through an iterative denoising process. The process consists of two stages: the forward diffusion process and the reverse denoising process. In the forward diffusion process, Gaussian noise is gradually injected into the original image using a Markov chain with a fixed step, and the image is perturbed into a representation close to the annotated Gaussian distribution; in the reverse denoising process, a learnable neural network model (such as Unet) is used to gradually denoise the noisy latent variables and restore the data distribution, thereby generating a clear image in Gaussian noise. Compared with the traditional generative adversarial network (GAN), the diffusion model training process is more documented and effectively avoids problems such as model collapse.

[0082] To avoid the enormous computational overhead of performing diffusion in a high-resolution pixel space and improve image generation efficiency, this paper employs a stable diffusion generative model as its core architecture. This method utilizes a pretrained variational autoencoder (VAE) to compress high-dimensional images into a low-dimensional latent space. This significantly reduces computational complexity while effectively preserving the image's structural and semantic details, achieving an optimal balance between efficiency and quality in image generation.

[0083] In this embodiment, the task of converting SAR images into optical images is to input the original optical image x0 into the VAE encoder E pre-trained in the stable diffusion model and compress it into a latent representation z0. Then, through a Markov chain with a fixed step T, Gaussian noise is gradually injected into the latent variable to obtain a noisy latent representation z with an approximate normal distribution. T , as the input of the denoising Unet.

[0084] The second step is structural semantic condition guidance. The SAR image is input into the ControlNet branch as structural semantic prior condition information to provide the overall spatial layout and contour information of the image to be generated. Figure 3 As shown, the noisy latent representation is input to the denoising Unet. Assume that F(·;θ) is a submodule of the pre-trained denoising Unet network with parameter θ, which transforms the input feature map x into another feature map y.

[0085] To add the ControlNet branch to the denoising Unet network, we freeze the parameters θ of the original module and simultaneously copy the module as a trainable copy with parameters θ c The trainable replica takes the SAR image C as input. The trainable replica is connected to the original frozen model structure through the zero convolution layer, denoted as L(·;·). The two zero convolution layers are 1×1 convolution layers with weights and biases initialized to zero, and the parameters are denoted as θ l1 and θ l2 .

[0086] y c =F(x;θ)+L(F(x+L(C;θ) l1 );θ c );θ l2 )

[0087] The frozen parameters of the stable diffusion model retain the pre-trained parameters trained on billions of images. Trainable replicas can reuse this large-scale pre-trained model to build a deep, robust, and powerful backbone network that can handle diverse input conditions. However, ControlNet lacks the ability to model semantic information, which can easily lead to semantic errors or color deviations in the generated images.

[0088] The third step is multimodal condition guidance injection. In traditional diffusion models, a cross-attention mechanism is usually used to inject text description information into the denoising UNet to guide image content generation. However, relying solely on text conditions is often difficult to fully control complex attributes such as the spatial layout, object posture and structural morphology of the image. To this end, the present invention introduces a multimodal condition guidance mechanism, which injects the structural semantic information of the SAR image, the scene information of the text description, and the style information of the target style image into the denoising UNet network through a decoupled cross-attention adapter, thereby enhancing the model's structural perception ability and style consistency expression ability.

[0089] The specific operation process is as follows:

[0090] First, the SAR image x c Input the pre-trained SAR image semantic encoder CWSAM, denoted as τ c , and then through the trainable linear layer l c Extracting structural semantic features c c , the process is expressed as: c c =l c (τ c (x c )). Write the text description x t Input the pre-trained text encoder CLIP, denoted as τ t , and then through the trainable linear layer l tExtract scene prior features c t , the process is expressed as: c t =l t (τ t (x t )). The target style image x s Input style encoder VGG16, denoted as τ s , and then through the trainable linear layer l s Extracted visual style features c s , the process is expressed as: c s =l s (τ s (x s )).

[0091] Then, the above three types of features are converted into Token representation and cross-attention is calculated with the features of the denoising UNet network to ensure that different modal information is independently guided at different semantic levels, thereby achieving fine-grained collaborative control of structure, semantics and style.

[0092] Given the query feature Z and text scene prior feature c in the denoising Unet network t , calculate the text feature cross attention output Where Q = ZW q , are the query, key, and value matrices of the attention operation, respectively, and W q 、 is the weight matrix of the trainable linear projection layer in the crisscross attention module.

[0093] Given the query feature Z in the denoising Unet network and the SAR image structural semantic feature c c , calculate the cross attention output, Where Q = ZW q , are the query, key, and value matrices of the attention operation, W q 、 is the weight matrix of the trainable linear projection layer in the crisscross attention module.

[0094] Given the query feature Z and style feature τ in the denoising Unet network s , calculate the image style cross attention output Where Q = ZW q , are the query, key, and value matrices of the attention operation, W q 、 is the weight matrix of the trainable linear projection layer in the crisscross attention module.

[0095] Finally, the fusion output of the disentangled cross attention can be expressed as:

[0096] Z new =Z c +Z t +Z s

[0097] Through the above-mentioned multimodal condition-guided injection mechanism, this model can significantly improve the comprehensive performance of generated images in terms of structural fidelity, semantic consistency and style naturalness, and is suitable for high-quality remote sensing image generation tasks in complex scenes.

[0098] The fourth step is to use the latent variable z′ after Unet prediction and noise removal. T , and then restored to the generated optical image through the VAE decoder D, we get

[0099] Step 4: Training process of multimodal conditional diffusion model. Figure 2 As shown in the figure, during the denoising process, the UNet network predicts the noise component in the latent representation based on the aforementioned multimodal conditional information, compares it with the injected known noise, calculates the noise prediction error, and constructs the noise prediction loss. Simultaneously, the denoised latent representation is restored to an image through the VAE decoder, and the reconstruction error is calculated compared with the original optical image. This process ensures the accuracy of the generated image's content restoration and effectively optimizes semantic consistency and style matching, further improving the realism, detail fidelity, and overall expressiveness of the generated image.

[0100] In this embodiment:

[0101] Given the input image latent representation z0, the image diffusion algorithm gradually adds noise to the image and produces a noisy representation z T , where T represents the number of times noise is added. Given a set of conditions, time step T, SAR image spatial structure control C, text prompt c t , SAR image semantic cue condition c c , style image condition c s , training model ε θ , the prediction is added to the noise representation z T The noise in . The noise prediction function can be expressed as:

[0102]

[0103] The latent variable z′ after Unet prediction and noise removal T , and then restored to the final optical image through the VAE decoder D, we get The loss is calculated using the L2 loss function and the input image, which can be expressed as:

[0104]

[0105] Finally, the overall loss function can be expressed as L = L n +L p

[0106] Step 5: The SAR image to be converted is input into the trained multimodal conditional diffusion model for inference. Figure 4 As shown in , a scene priori text description is formed for the SAR image to be converted, and the text description of the SAR image to be converted, the SAR image to be converted, and the target style image are input into the trained multimodal conditional diffusion model obtained in step 4 to obtain the converted optical image, as shown in Figure 4 The specific process includes the following sub-steps:

[0107] In the first step, the scene prior information of the SAR image to be converted is extracted to obtain a text description of the SAR image to be converted.

[0108] The second step is model loading and initialization. Load the multimodal conditional diffusion model trained in step 4, set the number of denoising steps in the diffusion process, and initialize random noise as the initial input of the denoising UNet network.

[0109] In the third step, conditions are introduced and denoising is performed. The SAR image is first fed into the ControlNet conditional control branch. The SAR image is also fed into the SAR image semantic encoder to extract structural semantic features. The text description is passed through the text encoder to extract scene prior features. The target style image is then fed into the style encoder to extract style features. The structural semantic features, scene prior features, and image style features are then uniformly embedded into tokens and injected into the cross-attention layer of the diffusion model's UNet network via a decoupled cross-attention adapter.

[0110] In the fourth step, the denoised latent representation output by the Unet network is restored to an optical image through a VAE decoder, which serves as the optical image generated by the SAR image conversion. Several improvements can be made without departing from the principles of the present invention, and these should also be considered within the scope of protection of the present invention.

[0111] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A SAR image to optical image conversion method based on a multimodal conditional diffusion model, characterized in that Here are the steps: Step 1: Acquire several SAR images and original optical images of the same scene to form initial remote sensing image training data; Step 2: Based on the initial remote sensing image training data generated in step 1, a text description is generated for the scene prior information of each set of SAR images and original optical images, thereby obtaining a diffusion model training dataset including SAR images, original optical images, text descriptions, and target style images. Step 3, establish a multimodal conditional diffusion model; Step 4: Use the diffusion model training data set obtained in step 2 to train the multimodal conditional diffusion model established in step 3 to obtain a trained multimodal conditional diffusion model; In step 5, a text description of the scene prior information is formed for the SAR image to be converted, and the text description of the SAR image to be converted, the SAR image to be converted, and the target style image are input into the trained multimodal conditional diffusion model obtained in step 4 to obtain the converted optical image.

2. The method for converting SAR images to optical images based on a multimodal conditional diffusion model according to claim 1, characterized in that: In step 1, the data sources of the SAR image and the original optical image of the same scene include public remote sensing image datasets, satellite image databases, or paired images obtained through independent collection.

3. The method for converting SAR images to optical images based on a multimodal conditional diffusion model according to claim 1, characterized in that: In step 2, the text description in the diffusion model training dataset is used to provide scene prior information, the SAR image is used to provide overall spatial layout and structural semantic information, and the target style image is used to provide style information.

4. The method for converting SAR images to optical images based on a multimodal conditional diffusion model according to claim 1, wherein: In step 3, the established multimodal conditional diffusion model takes the denoising-denoising reconstruction process of the optical image as the optimization target; The main body of the multimodal conditional diffusion model is the stable diffusion model, which introduces parallel ControlNet branches and decoupled cross-attention mechanism.

5. The method for converting SAR images to optical images based on a multimodal conditional diffusion model according to claim 4, characterized in that: The working process of the multimodal conditional diffusion model is: In the first step, the original optical image is input into the pre-trained VAE encoder and compressed into a latent representation. Then, Gaussian noise is gradually injected into the latent variable through a fixed-step Markov chain to obtain a noisy latent representation with an approximately normal distribution, which serves as the initial input of the diffusion process. In the second step, the SAR image is input into the ControlNet branch as structural prior information to provide the overall spatial layout and contour condition information of the optical image to be generated; In the third step, the SAR image is processed through a semantic encoder to extract structural semantic features, the text description is processed through a text encoder to extract scene prior features, and the target style image is processed through a style encoder to extract image style features. The structural semantic features, scene prior features, and image style features are uniformly embedded in a token form and injected into the cross-attention layer of the denoising UNet through a decoupled cross-attention adapter. In the fourth step, the noise component in the latent representation is predicted and compared with the injected known noise to calculate the noise prediction error. At the same time, the predicted noise is subtracted from the noisy latent representation to obtain the denoised latent representation, which is then restored to an optical image through the VAE decoder, and the reconstruction error is calculated compared with the original optical image.

Images