Sar to optical image conversion method based on multiscale structure perception diffusion bridge model

Abstract

The application discloses a SAR-to-optical image conversion method based on a multi-scale structure perception diffusion bridge model, and belongs to the technical field of image processing, and comprises the following steps: S1, acquiring a real SAR image, and performing structure extraction on the real SAR image to enhance the structure information of the real SAR image, so that an edge-enhanced SAR image is obtained; S2, embedding a convolution attention fusion module and a multi-scale fusion module based on a U-shaped network to construct a network framework capable of capturing local details and global correlations; S3, inputting the edge-enhanced SAR image and a noise-added optical image into the constructed network framework, and outputting a trained model through a forward diffusion process of a diffusion bridge model and loss function optimization; and S4, generating a target optical image by using the trained model. By using the method, the generated optical image is clear in structure, high in color restoration degree, free of obvious distortion, and has a high similarity to a real optical image.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a SAR-to-optical image conversion method based on a multi-scale structural sensing diffusion bridge model. Background Technology

[0002] In recent years, SAR (Synthetic Aperture Radar) imaging has been widely used in important fields such as disaster monitoring, topographic mapping, resource exploration, and military reconnaissance. Its all-weather, all-day imaging capabilities offer advantages for accurate scene analysis. However, due to its unique imaging mechanism, SAR images lack the rich color information found in optical images and are accompanied by geometric distortion and speckle noise, posing numerous challenges to SAR image interpretation. Therefore, converting SAR images into corresponding optical images can effectively leverage the complementary advantages of both modes. Thanks to the rapid development of generative models, SAR-to-optical image conversion has become a research hotspot and urgently needs to be advanced.

[0003] Under supervised learning with paired SAR and optical image data, researchers proposed a bidirectional reversible generative adversarial network (GAN) structure to enable generated images to possess both low-frequency and high-frequency features. Meanwhile, another study proposed a multi-domain cycle-consistent GAN, fusing the edge-preserving capabilities of CycleGAN with the pixel-level high accuracy of Pix2Pix to improve the quality of generated optical images. Furthermore, due to the difficulty in obtaining strictly spatiotemporally paired SAR and optical image datasets, researchers explored unsupervised training methods for CycleGAN on unpaired data. However, existing GAN-based methods are generally constrained by two major problems: structural ambiguity and pattern collapse, making it difficult to stably generate high-resolution images in specific scenarios.

[0004] Unlike generative adversarial networks (GANs), diffusion models iteratively add Gaussian noise to an image through a forward process and reconstruct the image based on a reverse denoising process. Existing SAR-to-optical image translation methods based on diffusion models, with DDPM as the mainstream framework, can achieve stable distribution approximation while significantly reducing the risk of mode collapse and effectively reducing structural blur. However, such methods still suffer from color distortion, which can easily lead to scene confusion. Summary of the Invention

[0005] The purpose of this invention is to provide a SAR-to-optical image conversion method based on a multi-scale structure-aware diffusion bridge model, which aims to solve the problems of color shift and structural blur and improve the performance of SAR-to-optical image translation tasks.

[0006] To achieve the above objectives, this invention provides a SAR-to-optical image conversion method based on a multi-scale structural sensing diffusion bridge model, comprising the following steps: S1. Obtain a real SAR image and extract its structure to enhance the structural information of the real SAR image, thus obtaining an edge-enhanced SAR image; S2. Based on the U-shaped network, embedding a convolutional attention fusion module and a multi-scale fusion module to construct a network framework that can capture local details and global relationships; S3. Input the edge-enhanced SAR image and the noisy optical image into the built network framework, and output the trained model through the forward diffusion process of the diffusion bridge model and the loss function optimization. S4. Generate the target optical image using the trained model.

[0007] Preferably, step S1 specifically involves: fusing the real SAR image with the Canny edge map through the SAR structure perception module, extracting structural features through multi-layer convolution, and generating an edge-enhanced SAR image.

[0008] Preferably, the SAR structure sensing module uses real SAR images. and edge maps extracted using the Canny operator As input, the system learns structural features through four convolutional layers, generating four sets of feature maps and a set of adjustable weights to preserve and enhance the edge information of the original image. The edge-enhanced SAR image output by the SAR structure sensing module is represented as follows: ; in, This represents the SAR image after edge enhancement. Represents a true SAR image. This represents the edge map extracted using the Canny operator. , , and All represent weights.

[0009] Preferably, step S2 includes: S21. Replace some residual blocks of the U-shaped network with convolutional attention fusion modules; S22. Multi-scale feature fusion is performed during the downsampling stage through a multi-scale fusion module, and used as input for the intermediate layer of the network.

[0010] Preferably, the convolutional attention fusion module is a plug-and-play structure consisting of convolutional layers and multi-head self-attention layers; The convolutional attention fusion module extracts local detail features through convolution; calculates global dependencies using a multi-head self-attention mechanism with positional encoding; fuses the attention output with the input features through residual connections; and optimizes the feature representation through convolution.

[0011] Preferably, step S22 specifically includes: The multi-scale fusion module extracts features from three downsampling layers. The system unifies the spatial resolution of features through bilinear interpolation and aligns channel dimensions using convolution operations. It uses a compression activation module to calculate channel dimension weights with global average pooling and two fully connected layers to highlight key features. The system recalibrates features with a single convolution layer to preserve global semantics while retaining local details, thus achieving better feature representation.

[0012] Preferably, the forward diffusion process in step S3 specifically includes: In the diffusion bridge model, the SAR image is denoted as... Optical images are denoted as Using a series of time series variables Characterizing the diffusion process, explicit modeling and The connection between them; following the principle of noise reduction diffusion bridge, optical images Gaussian noise is gradually added to obtain a transfer kernel that follows a Gaussian distribution. By learning the inverse transformation, the diffusion bridge model can generate optical pseudo-domain images with optical features from SAR images; the diffusion process is characterized using a Doob'sh-transform-based approach. and The implicit variable domain between: ; in, Represents the differential of the distribution at the current time step. Indicates the drift term. The coefficient representing noise. This indicates the steepest direction in which the current time step distribution transitions towards the target time step distribution. Indicates Gaussian noise. Indicates in The distribution of time steps must strictly conform to the distribution of the optical image. express The distribution of time steps; during the training of the diffusion bridge model, the score matching method is used to constrain the transformation network. This makes it output by the network. The sequence of representation The prediction score can approximate the generation process, and thus obtain .

[0013] Preferably, the loss function in step S3 Represented as: ; in, Represents the mathematical expectation. The weights represent the weights of the loss function. Represents a fractional function. This represents the data distribution at time step t. This represents the transition probability distribution conditioned on optical and SAR images during the reverse denoising process.

[0014] Preferably, step S4 is the sampling phase of the diffusion process, from the known joint distribution. Mid-sampling ; Using the form of probability flow ordinary differential equations, the reverse stage of the diffusion process can be represented as: ; ; For any , ; Represents a fractional function. Indicates Gaussian noise. This represents the transition probability distribution conditioned on the SAR image during the reverse denoising process.

[0015] Preferably, step S4 specifically involves: inputting the SAR image to be converted into the trained model, performing denoising iterations through reverse diffusion, and outputting a clear, color-accurate, and structurally complete target optical image, thereby completing the accurate conversion from SAR to optical image.

[0016] Therefore, the SAR to optical image conversion method based on the multi-scale structural sensing diffusion bridge model described above has the following beneficial effects: applying the bidirectional diffusion bridge model to the study of SAR to optical image conversion, clearly constructing the conversion path from SAR image to optical image, and making the converted image consistent with the real optical image to the greatest extent in terms of color and structure.

[0017] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0018] Figure 1 This is a flowchart of the SAR to optical image conversion method based on a multi-scale structure sensing diffusion bridge model according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of the convolutional attention fusion module according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of the multi-scale fusion module according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the SAR structure sensing module according to an embodiment of the present invention; Figure 5 The image shows the visualization results of five comparison methods and the method proposed in this invention on the SEN1-2 dataset in this embodiment of the invention. Figure 6 The image shows the visualization results of five comparison methods on the SAR2Opt dataset and the method proposed in this invention. Figure 7 The images show the detailed performance of different models in this invention on the SAR2Opt dataset. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0020] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0021] Example like Figure 1 As shown, this invention provides a SAR-to-optical image conversion method based on a multi-scale structure-sensing diffusion bridge model, comprising the following steps: S1. Acquire a real SAR image and extract its structure to enhance the structural information of the real SAR image, thus obtaining an edge-enhanced SAR image.

[0022] In this embodiment, step S1 specifically involves fusing the real SAR image with the Canny edge map using the SAR structure perception module, extracting structural features through multi-layer convolution, and finally generating an edge-enhanced SAR image. By perceiving the structure of the SAR image and integrating it into the conditional information for image conversion, the SAR structure perception module enables the model to retain the overall structure of the original image, thereby improving the image translation quality and reducing the edge structure differences between synthetic aperture radar images and optical images.

[0023] The SAR structure sensing module uses real SAR images and edge maps extracted using the Canny operator As input, the image learns structural features through four convolutional layers, generating four sets of feature maps and a set of adjustable weights to preserve and enhance the edge information of the original image. The overall structure is as follows: Figure 4 As shown. The SAR structure perception module outputs an edge-enhanced SAR image to assist subsequent networks in maintaining and highlighting the overall structure of the SAR image, which can be represented as: ; in, This represents the SAR image after edge enhancement. Represents a true SAR image. This represents the edge map extracted using the Canny operator. , , and All represent weights.

[0024] S2. Based on the U-shaped network, a network framework that can capture local details and global relationships is constructed by embedding a convolutional attention fusion module and a multi-scale fusion module. This enables the key global relationships and local details in the feature map to be effectively transmitted, thereby improving the quality of the generated image.

[0025] In this embodiment, step S2 includes: S21. Replace some residual blocks of the U-shaped network with a convolutional attention fusion module. The convolutional attention fusion module first extracts local detail features through convolution, then uses a multi-head self-attention mechanism with positional encoding to calculate global dependencies, then fuses the attention output with the input features through residual connections, and further optimizes the feature representation through convolution.

[0026] The convolutional attention fusion module is a plug-and-play structure consisting of convolutional layers and multi-head self-attention layers, such as... Figure 2 As shown, by combining the advantages of convolutional layers in capturing local spatial features with the strengths of multi-head self-attention mechanisms in modeling long-distance dependencies and global semantic connections, the convolutional attention fusion module can effectively capture cross-regional feature associations.

[0027] S22. Multi-scale feature fusion is performed during the downsampling stage using a multi-scale fusion module, and this fusion is used as input to the intermediate layer of the network to extract richer SAR image distribution information. Specifically: The multi-scale fusion module extracts features from three downsampling layers. The system unifies the spatial resolution of features through bilinear interpolation and aligns channel dimensions using convolutional operations. It then uses a compression activation module with global average pooling and two fully connected layers to calculate channel dimension weights, highlighting key features. Finally, a single convolutional layer recalibrates the features, preserving both local details and global semantics, resulting in superior feature representation. The overall structure is as follows: Figure 3 As shown.

[0028] S3. Input the edge-enhanced SAR image and the noisy optical image into the constructed network framework. Through the forward diffusion process of the diffusion bridge model and the loss function optimization, output the trained model that has learned the transformation law.

[0029] The forward diffusion process specifically includes: In the diffusion bridge model, the SAR image is denoted as... Optical images are denoted as Using a series of time series variables Characterizing the diffusion process, explicit modeling and The connection between them; following the principle of noise reduction diffusion bridge, optical images Gaussian noise is gradually added to obtain a transfer kernel that follows a Gaussian distribution. By learning the inverse transformation, the diffusion bridge model can generate optical pseudo-domain images with optical features from SAR images; the diffusion process is characterized using a Doob'sh-transform-based approach. and The implicit variable domain between: ; in, Represents the differential of the distribution at the current time step. Represents the differential of the distribution at the current time step. Indicates the drift term. This indicates the steepest direction in which the current time step distribution transitions towards the target time step distribution. Indicates Gaussian noise. Indicates in The distribution of time steps must strictly conform to the distribution of the optical image. express Distribution of time steps. During the training of the diffusion bridge model, the score matching method is used to constrain the transformation network. This makes it output by the network. The sequence of representation The prediction score can approximate the generation process, and thus obtain .

[0030] loss function Represented as: ; in, Represents the mathematical expectation. The weights represent the weights of the loss function. Represents a fractional function. This represents the data distribution at time step t. This represents the transition probability distribution conditioned on optical and SAR images during the reverse denoising process.

[0031] S4. Generate the target optical image using the trained model. Specifically, input the SAR image to be converted into the trained model, perform denoising iteration through reverse diffusion, and output a clear, color-accurate, and structurally complete target optical image, thus completing the accurate conversion from SAR to optical image.

[0032] During the sampling phase of the diffusion process, from the known joint distribution Mid-sampling .

[0033] Using the form of probability flow ordinary differential equations, the reverse stage of the diffusion process can be represented as: ; ; For any , ; Represents a fractional function. Indicates Gaussian noise. This represents the transition probability distribution conditioned on the SAR image during the reverse denoising process.

[0034] like Figure 5 As shown, each column, from left to right, represents the following methods: (a) SAR image, (b) real optical image, (c) Pix2Pix, (d) CycleGAN, (e) S-CycleGAN, (f) ParallelGAN, (g) Palette, and (h) the method proposed in this invention. The SEN1-2 dataset contains 282,384 pairs of SAR-optical images from Sentinel-1 and Sentinel-2 satellites. The SAR images have a spatial resolution of 10 meters and cover diverse global scenes across all four seasons. 3,200 image pairs were selected for training, and 800 SAR images were used for testing to ensure a balanced data volume across seasons for efficient training. The method of this invention significantly improves conversion accuracy through more accurate color perception, especially in scenes such as rivers, farmland, and roads. In the pairing method, Figure 5 (c) Pix2Pix and Figure 5 (f) ParallelGAN generates structures that are blurry, lack detail, or contain errors; furthermore,Figure 5 (g) Palette exhibits significant color distortion.

[0035] like Figure 6 As shown, each column, from left to right, represents the following methods: (a) SAR image, (b) real optical image, (c) Pix2Pix, (d) CycleGAN, (e) S-CycleGAN, (f) ParallelGAN, (g) Palette, and (h) the method proposed in this invention. The SAR2Opt dataset contains 2077 pairs of SAR-optical images with a SAR image resolution of 1 meter, covering 10 cities in Asia, North America, Oceania, and Europe. We randomly cropped the original 600×600 images to 512×512 for training, with 1247 image pairs used for training and 415 SAR images used for testing. There is no overlap between the training and test sets. The method of this invention performs excellently in terms of structural fidelity and visual quality, generating images that are highly similar to real optical images. In terms of visual results, Figure 6 (c) Pix2Pix Figure 6 (d) CycleGAN and Figure 6 (e) S-CycleGAN performs poorly in perceiving structural details, such as... Figure 6 As shown in the first row, they generate blurred areas in urban areas, making it impossible to correctly distinguish between areas of trees and grass. Furthermore, as... Figure 6 As shown in the last line, these methods often misidentify farmland as grassland or forest. In contrast, the method of this invention can accurately generate well-structured urban tree areas and clearly distinguish farmland boundaries.

[0036] like Figure 7 As shown in the yellow box and its enlarged area, the method of this invention performs better in terms of structure preservation and color accuracy compared to other models. Similar to its performance on the SEN1-2 dataset, the method proposed in this invention also achieves state-of-the-art results overall.

[0037] Therefore, this invention adopts the above-mentioned SAR to optical image conversion method based on the multi-scale structure sensing diffusion bridge model. Based on the diffusion bridge model, the conversion path from SAR image to optical image is explicitly constructed, maximizing the consistency between the generated image and the optical image in terms of both color and structure.

[0038] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A SAR-to-optical image conversion method based on a multi-scale structural sensing diffusion bridge model, characterized in that, Includes the following steps: S1. Obtain a real SAR image and extract its structure to enhance the structural information of the real SAR image, thus obtaining an edge-enhanced SAR image; S2. Based on the U-shaped network, embedding a convolutional attention fusion module and a multi-scale fusion module to construct a network framework that can capture local details and global relationships; S3. Input the edge-enhanced SAR image and the noisy optical image into the built network framework, and output the trained model through the forward diffusion process of the diffusion bridge model and the loss function optimization. S4. Generate the target optical image using the trained model.

2. The SAR-to-optical image conversion method based on a multi-scale structural sensing diffusion bridge model according to claim 1, characterized in that, Step S1 specifically involves fusing the real SAR image with the Canny edge map using the SAR structure perception module, extracting structural features through multi-layer convolution, and generating an edge-enhanced SAR image.

3. The SAR-to-optical image conversion method based on a multi-scale structural sensing diffusion bridge model according to claim 2, characterized in that: The SAR structure sensing module uses real SAR images and edge maps extracted using the Canny operator As input, the system learns structural features through four convolutional layers, generating four sets of feature maps and a set of adjustable weights to preserve and enhance the edge information of the original image. The edge-enhanced SAR image output by the SAR structure sensing module is represented as follows: ； in, This represents the SAR image after edge enhancement. Represents a true SAR image. This represents the edge map extracted using the Canny operator. , , and All represent weights.

4. The SAR-to-optical image conversion method based on a multi-scale structural sensing diffusion bridge model according to claim 1, characterized in that, Step S2 includes: S21. Replace some residual blocks of the U-shaped network with convolutional attention fusion modules; S22. Multi-scale feature fusion is performed during the downsampling stage through a multi-scale fusion module, and used as input for the intermediate layer of the network.

5. The SAR-to-optical image conversion method based on a multi-scale structural sensing diffusion bridge model according to claim 4, characterized in that: The convolutional attention fusion module is a plug-and-play structure consisting of convolutional layers and multi-head self-attention layers; The convolutional attention fusion module extracts local detail features through convolution; calculates global dependencies using a multi-head self-attention mechanism with positional encoding; fuses the attention output with the input features through residual connections; and optimizes the feature representation through convolution.

6. The SAR-to-optical image conversion method based on a multi-scale structural sensing diffusion bridge model according to claim 4, characterized in that, Step S22 is as follows: The multi-scale fusion module extracts features from three downsampling layers. The system unifies the spatial resolution of features through bilinear interpolation and aligns channel dimensions using convolution operations. It uses a compression activation module to calculate channel dimension weights with global average pooling and two fully connected layers to highlight key features. The system recalibrates features with a single convolution layer to preserve global semantics while retaining local details, thus achieving better feature representation.

7. The SAR-to-optical image conversion method based on a multi-scale structural sensing diffusion bridge model according to claim 1, characterized in that, The forward diffusion process in step S3 specifically includes: In the diffusion bridge model, the SAR image is denoted as... Optical images are denoted as Using a series of time series variables Characterizing the diffusion process, explicit modeling and The connection between them; following the principle of noise reduction diffusion bridge, optical images Gaussian noise is gradually added to obtain a transfer kernel that follows a Gaussian distribution. By learning the inverse transform, the diffusion bridge model can generate optical pseudo-domain images with optical features from SAR images; the diffusion process is characterized using Doob's h-transform. and The implicit variable domain between: ； in, The derivative of the distribution at the current time step is represented. Indicates the drift term. This represents the coefficient used to remove noise. This indicates the steepest direction in which the current time step distribution transitions towards the target time step distribution. Indicates Gaussian noise. Indicates in The distribution of time steps must strictly conform to the distribution of the optical image. express Distribution of time steps; During the training of the diffusion bridge model, a score matching method is used to constrain the transformation network. This makes it output by the network. The sequence of representation The prediction score can approximate the generation process, and thus obtain .

8. The SAR-to-optical image conversion method based on a multi-scale structural sensing diffusion bridge model according to claim 1, characterized in that, Loss function in step S3 Represented as: ； in, Represents the mathematical expectation. The weights represent the weights of the loss function. Represents a fractional function. This represents the data distribution at time step t. This represents the transition probability distribution conditioned on optical and SAR images during the reverse denoising process.

9. The SAR-to-optical image conversion method based on a multi-scale structural sensing diffusion bridge model according to claim 1, characterized in that: Step S4 is the sampling phase of the diffusion process, from the known joint distribution. Mid-sampling ; Using the form of probability flow ordinary differential equations, the reverse stage of the diffusion process can be represented as: ； ； For any , ; Represents a fractional function. Indicates Gaussian noise. This represents the transition probability distribution conditioned on the SAR image during the reverse denoising process.

10. The SAR-to-optical image conversion method based on a multi-scale structural sensing diffusion bridge model according to claim 1, characterized in that, Step S4 specifically involves inputting the SAR image to be converted into the trained model, performing denoising iterations through reverse diffusion, and outputting a clear, color-accurate, and structurally complete target optical image, thus completing the accurate conversion from SAR to optical image.

Images