Sar-to-optical image conversion method and system based on spatial-frequency steering

Abstract

This invention discloses a space-frequency guided SAR-to-optical image conversion method and system, belonging to the field of image conversion technology. The method includes: extracting image features from the SAR image; extracting noise from the optical image to obtain a target noise image; extracting features from the target noise image to obtain noise features; segmenting the noise features along the channel dimension into processing path features and shortcut path features; performing residual concatenation between the image features and the processing path features to obtain residual fusion features; enhancing the residual fusion features to obtain enhanced features; concatenating and fusing the enhanced features with the shortcut path features and adding them to the noise features to obtain injected features; decoding the injected features to obtain a predicted noise image; and subtracting the predicted noise image from the target noise image to obtain a virtual optical image. This method improves the generation quality and computational speed of virtual optical images and solves the problem of poor virtual image quality generated by current methods.

Description

Technical Field

[0001] This invention relates to the fields of remote sensing image processing and computer vision technology, and in particular to a method and system for converting SAR to optical images based on space-frequency guidance. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] With the rapid development of remote sensing technology, synthetic aperture radar (SAR) and optical remote sensing images are increasingly widely used in environmental monitoring, disaster assessment, and military reconnaissance. Optical images are favored for their rich texture information and good visual interpretability, but their acquisition is affected by factors such as weather and lighting, making continuous all-weather observation difficult. In contrast, SAR has all-weather, all-day imaging capabilities, effectively compensating for the lack of optical data. Therefore, SAR-to-optical (S2O) image conversion technology has become a research hotspot in the field of remote sensing, aiming to convert SAR images into high-quality optical images to support more accurate ground feature identification and environmental understanding.

[0004] However, SAR images are severely affected by multiplicative speckle noise, resulting in a low signal-to-noise ratio. The noise is spatially coupled with the real signal, which makes it difficult for traditional methods to extract features and perform cross-modal alignment, thus affecting the structural integrity and detail fidelity of the converted images.

[0005] In recent years, diffusion models have shown significant potential in image generation and translation tasks due to their powerful generative capabilities and training stability. However, existing diffusion model-based methods for converting SAR images to optical images simply stitch and inject SAR images with noisy images, making it difficult to achieve effective cross-modal alignment. This can easily lead to structural distortion and texture confusion, ultimately resulting in poor quality virtual optical images. Summary of the Invention

[0006] To address the aforementioned problems, this invention proposes a SAR-to-optical image conversion method and system based on space-frequency guidance, which improves the quality of the final obtained virtual optical image.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] In a first aspect, the present invention proposes a SAR-to-optical image conversion method based on space-frequency guided methods, including:

[0009] Acquire SAR and optical images;

[0010] Extracting image features from SAR images;

[0011] Noise is extracted from the optical image to obtain the target noise image;

[0012] Feature extraction is performed on the target noisy image to obtain noise features;

[0013] The noise features are segmented along the channel dimension into processing path features and shortcut path features;

[0014] Residual fusion features are obtained by performing residual concatenation between image features and processing path features.

[0015] Enhance the residual fusion features to obtain the enhanced features;

[0016] The enhanced features are concatenated and fused with the shortcut path features, and then added to the noise features to obtain the injected features;

[0017] Decode the injected features to obtain the predicted noise image;

[0018] The virtual optical image is obtained by subtracting the predicted noise image from the target noise image.

[0019] Furthermore, the image features are upsampled and then weighted and averaged after projection through a lightweight linear layer to obtain the fusion conditional features;

[0020] The residual fusion features are obtained by adding the fusion condition features and the processing path features.

[0021] Furthermore, channel attention and spatial attention modulation are applied to the residual fusion features to obtain enhanced features.

[0022] Furthermore, multi-scale spatial features and multi-scale frequency domain features of SAR images are extracted;

[0023] Multi-scale spatial features and multi-scale frequency domain features are fused to obtain the image features of SAR images.

[0024] Furthermore, the SAR image is converted into a single-channel grayscale image;

[0025] Obtain the complex spectrum of a single-channel grayscale image at the frequency coordinates;

[0026] Calculate the amplitude spectrum of the complex spectrum and perform a logarithmic transformation on the amplitude spectrum to obtain the logarithmic amplitude spectrum;

[0027] Based on the logarithmic amplitude spectrum, obtain the frequency image of the three channels;

[0028] Multi-scale feature extraction is performed on the frequency images of the three channels to obtain the multi-scale frequency domain features of the SAR images.

[0029] Furthermore, the target noisy image is encoded to obtain encoded features;

[0030] Further feature extraction is performed on the encoded features to obtain noise features.

[0031] Secondly, this invention also proposes a space-frequency guided SAR-to-optical image conversion system, comprising:

[0032] Image acquisition unit, used to acquire SAR images and optical images;

[0033] The image feature extraction unit is used to extract image features from SAR images;

[0034] The noise feature extraction unit is used to extract noise from the optical image to obtain a target noise image; and to extract noise features from the target noise image.

[0035] The diffusion unit is used to segment noise features along the channel dimension into processing path features and shortcut path features; perform residual concatenation between image features and processing path features to obtain residual fusion features; enhance the residual fusion features to obtain enhanced features; concatenate and fuse the enhanced features with the shortcut path features and add them to the noise features to obtain injected features; decode the injected features to obtain the predicted noise image; and subtract the predicted noise image from the target noise image to obtain the virtual optical image.

[0036] Thirdly, a computer device is proposed, the device comprising:

[0037] A processor, adapted to execute computer programs;

[0038] A computer-readable storage medium storing a computer program, which, when executed by the processor, implements the space-frequency guided SAR-to-optical image conversion method proposed in the first aspect.

[0039] Fourthly, a computer-readable storage medium is proposed, which stores a computer program adapted for loading and execution by a processor of the space-frequency guided SAR-to-optical image conversion method proposed in the first aspect.

[0040] Fifthly, a computer program product is proposed, which includes a computer program that, when executed by a processor, implements the space-frequency guided SAR to optical image conversion method proposed in the first aspect.

[0041] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0042] This invention proposes a space-frequency guided SAR-to-optical image conversion method and system. When injecting image features from a SAR image into a target noise image, the method extracts features from the target noise image to obtain noise features. These noise features are then segmented along the channel dimension into processing path features and shortcut path features. The processing path features serve as the main channel influencing the generation process through conditional information, while the shortcut path features retain the original information passed from the previous layer and are directly passed to the output through subsequent operations. The image features and processing path features are then residually concatenated to obtain residual fusion features. These residual fusion features are then enhanced to obtain enhanced features. The enhanced features are then concatenated and fused with the shortcut path features, added to the noise features, and decoded to obtain a predicted noise image. Finally, the predicted noise image is subtracted from the target noise image to obtain a virtual optical image. By setting shortcut path features, the method ensures that even when conditional information is not entirely accurate or relevant, the forward propagation signal and the backward propagation gradient can proceed smoothly, greatly improving the model's stability, effectively controlling the number of feature channels involved in complex conditional fusion calculations, and reducing computational complexity.

[0043] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0044] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments of this application and their descriptions are used to explain this application and do not constitute an undue limitation of this application.

[0045] Figure 1 This is a flowchart of the SAR to optical image conversion method based on space-frequency guidance proposed in an embodiment of the present invention;

[0046] Figure 2 This is a diagram of the image conversion model architecture proposed in an embodiment of the present invention;

[0047] Figure 3 This is a diagram of the multi-scale integrated encoder architecture proposed in an embodiment of the present invention;

[0048] Figure 4 This is a flowchart of the conditional injection process proposed in an embodiment of the present invention. Detailed Implementation

[0049] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0050] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0051] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0052] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0053] This invention proposes a space-frequency guided SAR-to-optical image conversion method, such as... Figures 1-4 As shown, it includes:

[0054] Acquire SAR and optical images;

[0055] Extracting image features from SAR images;

[0056] Noise is extracted from the optical image to obtain the target noise image;

[0057] Feature extraction is performed on the target noisy image to obtain noise features;

[0058] The noise features are segmented along the channel dimension into processing path features and shortcut path features;

[0059] Residual fusion features are obtained by performing residual concatenation between image features and processing path features.

[0060] Enhance the residual fusion features to obtain the enhanced features;

[0061] The enhanced features are concatenated and fused with the shortcut path features, and then added to the noise features to obtain the injected features;

[0062] Decode the injected features to obtain the predicted noise image;

[0063] The virtual optical image is obtained by subtracting the predicted noise image from the target noise image.

[0064] SAR images are synthetic aperture radar images, denoted as Where H and W are the height and width of the image, and the number of channels is 3. If the original SAR image is a single-channel or dual-channel image, it needs to be expanded into three-channel data through channel copying or interpolation methods to maintain consistency with the input dimensions in subsequent operations.

[0065] In this embodiment of the invention, image features of SAR images are extracted through an image conversion model, and noise features are fused with SAR images to obtain virtual optical images.

[0066] Among them, by acquiring optical images , ;

[0067] Noise is extracted from the optical image to obtain the target noise image.

[0068] SAR images can come from publicly available remote sensing datasets (such as SEN12 and QXS-SAROPT) or be acquired in real time via satellite or airborne platforms. SAR images can be collected through web scraping, API calls, or direct access to databases.

[0069] Optical images The image conversion model can call the target noise image to synthesize the SAR image into a virtual optical image, which is the optical ground truth image corresponding to the SAR image.

[0070] After acquiring SAR and optical images, this embodiment of the invention first performs preprocessing on the input SAR and optical images. The preprocessing includes size unification and numerical standardization. The images are uniformly adjusted to a preset resolution (256×256) through preprocessing.

[0071] Size unification refers to uniformly cropping or scaling SAR images and corresponding optical images (used for training) to a resolution of 256×256 pixels to ensure consistent size during batch processing.

[0072] Numerical standardization: Normalize the pixel values ​​of SAR images so that they are distributed in the range of [-1, 1] or [0, 1] to improve the stability of model training;

[0073] Data augmentation: During the training phase, random horizontal / vertical flipping, rotation, and other augmentation operations are performed on image pairs to improve the model's generalization ability.

[0074] The image conversion model in this embodiment of the invention includes a multi-scale integrated encoder (MIE) and a diffusion model. The MIE is used to extract multi-scale spatial and frequency domain features from SAR images, and feature fusion is performed. The process of extracting image features from SAR images includes:

[0075] Extracting multi-scale spatial and frequency domain features from SAR images;

[0076] Multi-scale spatial features and multi-scale frequency domain features are fused to obtain the image features of SAR images.

[0077] like Figure 3As shown, the multi-scale integrated encoder (MIE) includes a spatial branch and a frequency branch. The spatial branch extracts multi-scale spatial features from the SAR image through convolution and stacking residual blocks. The frequency branch converts the SAR image into a grayscale image, performs a two-dimensional Fourier transform and calculates the logarithmic amplitude spectrum to extract global structural information that is insensitive to noise, thereby obtaining the multi-scale frequency domain features of the SAR image.

[0078] Specifically: Set up a spatial encoder in the spatial branch. The spatial encoder consists of five cascaded convolutional downsampling blocks. Each block contains a convolutional layer, a group normalization layer, a Swish activation function, and two cascaded residual blocks. Each residual block contains convolution, normalization, and activation operations. By downsampling the SAR image step by step through these five convolutional downsampling blocks, spatial features at five scales are obtained. .

[0079] In this block, the first convolutional downsampling block contains a 3×3 convolutional layer with a stride of 1, and the remaining convolutional downsampling blocks contain 3×3 convolutional layers with a stride of 2. The convolutional downsampling blocks are used to downsample and expand the number of channels, and the number of groups in the group normalization layer is set to 32.

[0080] Input SAR image First, the image passes through a first convolutional downsampling block, expanding the number of channels to 64 while maintaining the same spatial size (256×256). Then, it passes through four more convolutional downsampling blocks, each halving the spatial size of the input image (resolutions change sequentially to 128×128, 64×64, 32×32, and 16×16), while progressively increasing the number of channels (to 128, 256, 512, and 1024 respectively). Finally, the encoder outputs five spatial features at different scales, denoted as follows:

[0081] ；

[0082] in , , =1, 2, 3, 4 or 5, number of channels It increases with increasing downsampling depth, specifically as follows: .feature arrive Each contains spatial image information at different levels, from fine to coarse. It retains the richest local details and edge information, while This encodes the highest-level semantics and global contextual information. Spatial features It mainly includes local texture, edge information and detail features. It is sensitive to speckle noise, but it is crucial for detail restoration.

[0083] The frequency branch is responsible for extracting multi-scale features from SAR images in the frequency domain. This branch aims to extract global structural features inherently robust to multiplicative speckle noise from the frequency domain representation of SAR images. The theoretical basis is that speckle noise in SAR images mainly manifests as high-frequency random scattering, while the semantic structure and contour information of ground features are primarily distributed in the mid-to-low frequency bands. By performing feature extraction in the frequency domain, noise interference can be reduced at its source, resulting in cleaner and more stable structural priors. The process includes:

[0084] Convert SAR images to single-channel grayscale images;

[0085] Obtain the complex spectrum of a single-channel grayscale image at the frequency coordinates;

[0086] Calculate the amplitude spectrum of the complex spectrum and perform a logarithmic transformation on the amplitude spectrum to obtain the logarithmic amplitude spectrum;

[0087] Based on the logarithmic amplitude spectrum, obtain the frequency image of the three channels;

[0088] Multi-scale feature extraction is performed on the frequency images of the three channels to obtain the multi-scale frequency domain features of the SAR images.

[0089] Specifically:

[0090] (1) Perform a spatial-domain to frequency-domain conversion on the SAR image, converting the SAR image into a single-channel grayscale image, denoted as . This process is achieved through a standard luminance-weighted formula;

[0091] ;

[0092] The conversion uses the standard grayscale formula:

[0093] ;

[0094] in SAR image The pixel coordinates, R, G, B represent the image respectively. The image uses three color channels: red, green, and blue. Grayscale processing eliminates interference that color information might introduce, allowing subsequent frequency domain analysis to focus on the structural intensity distribution of the image.

[0095] (2) The obtained grayscale image The two-dimensional Discrete Fourier Transform (DFT) is applied to transform it from the spatial domain to the frequency domain. This transformation process is mathematically represented as:

[0096] ;

[0097] in, In frequency coordinates The complex spectrum at that point, It is the imaginary unit. This represents the Fourier transform operation. This transform decomposes an image into the sum of sinusoidal wave components of different directions and frequencies, thus revealing the frequency composition of the image.

[0098] (3) Due to the complex spectrum Phase information is sensitive to noise and difficult to extract stably. This invention focuses primarily on amplitude information, which represents the energy intensity of each frequency component. The amplitude spectrum (i.e., complex modulus) is calculated and a logarithmic transformation is performed to enhance numerical stability.

[0099] ;

[0100] In the formula, The operation of taking the modulus of a complex number and the logarithmic operation have two important functions: first, they compress the dynamic range of the spectral values, avoiding numerical imbalances caused by individual strong frequency components; second, they conform to the logarithmic perceptual characteristic of the human visual system for intensity changes. The final result... Known as the logarithmic amplitude spectrum, it visually represents a SAR image in the form of a two-dimensional image. The relative energy distribution of each spatial frequency component in the spectrum. In this spectrum, speckle noise typically appears as randomly distributed high-frequency bright spots, while the edges, textures, and other structures of ground features appear as organized low- or mid-frequency banded patterns.

[0101] (4) To input this single-channel frequency representation into a convolutional neural network with the same spatial branch architecture, it needs to be expanded to three channels through a simple channel duplication operation. Subsequently, the three-channel frequency image... Feed into frequency encoder It is particularly important to emphasize that the frequency encoder Network structure and spatial encoder The design maintains complete consistency, including network depth, the number and size of filters in each layer, downsampling factors, and the normalization and activation functions used. This symmetrical design ensures that features extracted from two completely different domains (spatial and frequency domains) correspond precisely in abstract hierarchy and scale, creating the necessary conditions for subsequent feature-level fusion. The frequency encoder outputs frequency domain feature maps with five different abstraction levels and spatial resolutions:

[0102] ;

[0103] Each frequency domain feature In terms of both spatial dimensions and passageway dimensions, it is similar to spatial features of the same scale. Strict matching. These frequency domain features primarily encode the global shape contour, main structural orientation, and low-to-mid-frequency semantic information of SAR images that are relatively insensitive to noise.

[0104] At each scale of the spatial and frequency branches The extracted spatial features Frequency domain characteristics Channel concatenation is performed, which simply connects two feature tensors along the channel dimension, so that the fused features simultaneously contain dual information at that scale regarding "how the texture details are" (from the spatial branch) and "what the structural contours are" (from the frequency branch). However, doubling the number of channels leads to a significant increase in subsequent computation. To achieve efficient information interaction and compression in a lightweight manner, a fusion projection layer is introduced. This layer typically consists of a 1×1 convolutional layer. accomplish:

[0105] ;

[0106] The 1×1 convolution here plays two key roles: First, it performs cross-channel information fusion and feature recombination, learning how to most effectively combine complementary information from space and frequency, rather than simply concatenating them; second, it reduces the channel dimension from 2... Compress back to the original This makes the output image features It can directly adapt to the input channel requirements of subsequent U-Net layers without requiring an additional adaptation layer. Typically, It is followed by a set of normalization layers and Swish activation functions to stabilize training and introduce non-linearity.

[0107] The features obtained from fusing five different scales are collected to form the final multi-scale, cross-domain image feature set. :

[0108] ;

[0109] This set It is a hierarchical feature pyramid, in which (Highest resolution) primarily preserves the fine edges and texture information after fusion, while The lowest resolution encodes the fused high-level semantics and global scene context. This rich set of features provides comprehensive and robust prior guidance for the subsequent diffusion generation process.

[0110] Subsequently, in this embodiment of the invention, a virtual optical image is generated using a diffusion model based on the image features of the SAR image, resulting in an image feature set. This will be passed as a key conditional input to the diffusion model. During the diffusion model's inverse denoising process to reconstruct the optical image, each feature in the set... The conditions are precisely and efficiently integrated into the intermediate layers of U-Net through a lightweight dynamic conditional injection mechanism. This multi-scale, adaptive conditional injection method allows the generation process to be guided by spatial details and frequency structure at different levels of abstraction, thereby ensuring that the generated virtual optical image has both realistic local textures and maintains the correct overall structure and contours.

[0111] The Diffusion Model for Image Reduction (DDIM) extracts the target noise image from the optical image and injects the image features extracted from the SAR image as conditional features into the target noise image to obtain a virtual optical image. The process includes:

[0112] The diffusion model incorporates U-Net and Dynamic Conditional Partial Injection (DCPI) mechanisms. The process of virtual optical image generation using the diffusion model includes a "forward diffusion process" and a "denoising process," as detailed below:

[0113] Forward diffusion is mainly used in the training phase of image conversion models. It is used to process optical images. According to linear noise scheduling (from linearly increase to Gradually add Gaussian noise After T=1000 steps, an approximately pure noise target noise image is obtained. Arbitrary time step The image can be calculated using a closed-form expression:

[0114] ;in, .

[0115] In the formula, Indicates time step The cumulative noise variance at a given point is used to control the noise addition process.

[0116] The denoising process is implemented using a diffusion model. This model encodes and decodes the target noisy image using U-Net, and injects image features extracted from the SAR image into each intermediate layer of the U-Net. The process includes: encoding the target noisy image to obtain encoded features; further extracting features from the encoded features to obtain noise features; injecting the image features into the noise features to obtain injected features; and decoding the injected features to obtain the predicted noisy image. Specifically, the input to the U-Net includes the target noisy image. and image features The output of U-Net is the predicted noisy image. U-Net generates time-step embeddings during input processing. Time step embedding It is generated by time step t through sinusoidal position encoding mapping.

[0117] U-Net consists of an encoding layer, an intermediate layer, and a decoding layer. The encoding layer encodes the target noisy image to obtain encoded features. The intermediate layer further extracts features from the encoded features to obtain noise features. In each intermediate layer of U-Net, the image features are injected into the noise features using the Dynamic Conditional Partial Injection (DCPI) mode to obtain injected features. Finally, the decoding layer decodes the injected features to obtain the predicted noisy image.

[0118] The Dynamic Conditional Partial Injection (DCPI) mode injects image feature information into noise features. The specific operations for obtaining the injected features include:

[0119] The noise characteristics output by the intermediate layer Along the channel dimension =0.5 segmentation into processing path features and shortcut path features ;

[0120] Image features at various scales Perform upsampling to make it consistent with The spatial dimensions are the same, and after projection through a lightweight linear layer, a weighted average is obtained to obtain the fusion conditional features. ;

[0121] Fusion condition features and The residual fusion features are obtained by summing the features. Channel attention and spatial attention modulation are then applied to the residual fusion features to obtain the enhanced features. ;

[0122] Will and By concatenating the features and adding residual connections through a fusion convolutional layer, injected features are obtained. , as the output feature of the intermediate layer.

[0123] Among them, bilinear interpolation is used to extract image features at various scales. Perform upsampling.

[0124] Inject features from the intermediate layers of U-Net The input is decoded in the decoding layer to obtain the predicted noise image. , target noisy image Subtracting the predicted noise image yields the noise image for the next step. Noisy image The virtual optical image is obtained by re-entering the diffusion model and performing T-step prediction and subtracting the predicted noise image.

[0125] The DCPI mode, serving as the core bridge connecting conditional feature extraction and image generation in this embodiment of the invention, functions to efficiently and accurately integrate the rich conditional information (image features) extracted by the multi-scale ensemble encoder into the inverse denoising process of the diffusion model. This achieves cross-modal semantic alignment and feature modulation while significantly reducing the computational redundancy of traditional conditional injection methods. Figure 4 As shown.

[0126] Traditional conditional diffusion models typically employ simple feature concatenation or injecting conditional features into all channels. These methods suffer from two main problems: first, they are computationally expensive, as all feature channels participate in conditional fusion, increasing the number of model parameters and computational load; second, the utilization of conditional information is not refined enough, potentially leading to interference from irrelevant information or dilution of crucial information. To address these issues, the DCPI model proposed in this embodiment is based on two core innovations:

[0127] Partial channel injection: Only some feature channels are selected to interact with the conditional information, while the remaining channels are used as "shortcuts" to retain the original information. This achieves both conditional guidance and ensures stable gradient propagation.

[0128] Dynamic adaptive modulation: By using a series of channel attention and spatial attention mechanisms, the importance of feature maps in different channels and spatial locations is dynamically recalibrated based on the content of conditional features, thereby achieving refined feature fusion.

[0129] The specific implementation steps are as follows:

[0130] Step S11: Initialization and Input Reception

[0131] This step initializes the DCPI module and receives input data from two sources, which is a prerequisite for the module to start working.

[0132] First, noise features from the current intermediate layer output of U-Net are received. These noise features are referred to as "Query Features" in this embodiment of the invention, denoted as... .in This represents the batch size. H represents the number of channels in the feature set of this layer, and H and W represent the height and width of the feature, respectively. For example, with an input image resolution of 256×256, the typical size of the output feature of a U-Net intermediate layer might be... =10, =256, H=16, W=16. This feature map carries a partial implicit representation of the image at the current denoising stage.

[0133] Simultaneously, it receives the output from the multi-scale integrated encoder, namely the image feature set. Each image feature With varying spatial dimensions and channel numbers, they collectively constitute a hierarchical, cross-domain description of the input SAR image. The core task of the DCPI module is to process the currently generated noise features. and image features To achieve efficient integration.

[0134] Step S12: Channel splitting operation

[0135] This step is a key step in achieving lightweighting of DCPI. Its purpose is to divide the complete query feature stream into two parts: one part is used for deep modulation of receiving conditional information, and the other part serves as a shortcut to retain the original information.

[0136] The specific operation is along the channel dimension (i.e.) Dimensions) will query features According to a preset fixed ratio Segmentation is performed. In this embodiment of the invention, the experiment is set to default settings. A value of 0.5 strikes a good balance between performance and efficiency. The partitioning process can be mathematically represented as:

[0137] ;

[0138] in, These are referred to as processing path features. These features will be deeply modulated by subsequent fusion and attention mechanisms, and are the main channel through which conditional information influences the generation process; These are known as shortcut path features. These features do not directly participate in conditional fusion; instead, they retain their original information passed from the previous network layer and are directly transmitted to the output through subsequent concatenation operations. The shortcut path design has two important functions: first, it acts as an information superhighway, ensuring that even when the conditional information is not entirely accurate or relevant, the forward propagation signal and the backward propagation gradient can flow smoothly, greatly improving training stability; second, it effectively controls the number of feature channels participating in complex conditional fusion calculations and reduces computational complexity.

[0139] Step S13: Multi-scale conditional feature fusion

[0140] The goal of this step is to combine image features at five different scales. Integrate into a processing path feature Single fusion condition feature with perfectly matched dimensions .

[0141] Due to image features Since they have different spatial dimensions, they first need to be uniformly upsampled to match the processing path features. The same spatial resolution. This embodiment uses a bilinear interpolation algorithm for upsampling, which is computationally efficient and maintains good smoothness. After upsampling, the features at each scale... The number of its channels still needs to be increased. From projection to Consistent This is achieved through a set of lightweight linear projection layers. Implementation. Each Essentially, it's a 1×1 convolutional layer followed by group normalization and the Swish activation function. To comprehensively utilize all scale information from local details to global context, a simple average fusion is performed on all five projected features, i.e. =5. The averaging operation assumes that information at each scale is equally important, and is an efficient and stable fusion strategy.

[0142] ;

[0143] The final result It is a feature tensor that integrates multi-scale, cross-domain conditional information, and its spatial size and number of channels are related to the processing path features. Full alignment prepares the ground for subsequent feature interactions.

[0144] Step S14: Dual Attention Modulation

[0145] This step corresponds to Figure 4 The "channel attention" and "spatial attention" mechanisms in DCPI are the core of its accurate conditional alignment. It sequentially applies channel attention and spatial attention mechanisms to adaptively recalibrate features that incorporate conditional information, allowing the network to automatically focus on the semantic channels and spatial regions most relevant to the current generation task.

[0146] First, conditional features are fused through a residual connection. Add to processing path features In this process, preliminary conditional information injection is achieved. This residual connection ensures that conditional information is incorporated in an additive perturbation manner, avoiding potential information overload and facilitating gradient flow.

[0147] Subsequently, channel attention was performed. Modulation. The goal of the channel attention mechanism is to learn the importance weights of different feature channels. It first compresses each channel of the feature across the entire spatial range, uses global average pooling to obtain a channel-level statistical vector, and feeds it into a perceptron to learn the nonlinear interactions between channels and generate channel weights. This operation amplifies feature channels that are relevant to important structures in SAR images (such as building outlines, roads, and water boundaries) and suppresses irrelevant or noise-dominated channels.

[0148] Next, spatial attention is performed. Modulation. The goal of spatial attention mechanisms is to learn the importance of feature maps at different locations in space. It takes channel-modulated features as input and performs average pooling and max pooling operations along the channel dimension, concatenates the results, and processes them with convolutional layers to obtain spatial weights.

[0149] This yields the enhanced features after modulation. :

[0150] ;

[0151] at this time, It is already an enhanced feature that has been deeply modulated by multi-scale conditional information, which emphasizes both key semantic channels and focuses on spatial regions related to the structure of the input SAR image.

[0152] Step S15: Feature Recombination and Final Output

[0153] This step is responsible for recombinating the finely modulated enhanced features with the original shortcut path features to form a complete output feature.

[0154] First, the modulated enhanced features and shortcut path features Reassemble along the channel dimension to restore the original input. Same number of channels Next, a fusion convolutional layer is used. Lightweight post-processing is performed on the spliced ​​features. Typically, it's also a 1×1 convolutional layer, its role being to facilitate information exchange between the processing path and the shortcut path, and to fine-tune the fused feature distribution. Finally, a global residual connection is introduced, along with the query features from the original input, to generate the injected features output by the DCPI module. :

[0155] ;

[0156] Global residual connections are crucial for ensuring module learning identity mapping. They guarantee that network performance does not degrade when conditional information is weak or module parameters are initialized near zero, greatly simplifying the training of deep networks. (Feature injection) It is then fed into the next layer of U-Net (the decoding layer) to continue participating in the image denoising and reconstruction process.

[0157] The diffusion model in this embodiment of the invention implements a two-stage training strategy:

[0158] The first phase of training uses a simplified mean squared error function:

[0159] ;

[0160] In the formula, The first-stage loss is used; the Adam optimizer is employed, and the initial learning rate is set to... The training process involved 100,000 steps using a cosine decay strategy with a batch size of 10. Validation was performed using a 50-step DDIM sampler during this phase.

[0161] The second stage of training introduces a joint loss function:

[0162] ;

[0163] in, For the losses in the second phase, To perceive the loss, a pre-trained VGG network is used to compute feature differences; To account for the focal frequency loss, the spectral difference is calculated in the frequency domain using a weighted average. To counteract the loss, a visual aid discriminator is introduced for adversarial training; the weights are set to a default value. .

[0164] VGG networks are a classic deep convolutional neural network widely used in image recognition. In LPIPS loss, they are typically used as feature extractors.

[0165] The feature differences calculated here are not the original pixel differences, but rather the differences between feature maps extracted from multiple intermediate convolutional layers as the image propagates forward through the VGG network. These feature maps encode multi-level visual information, ranging from low-level edge textures to high-level semantic structures. Specifically, the feature maps of each layer are channel-normalized, and then the L2 distance between the feature maps of two images is calculated layer by layer. Finally, a weighted sum is obtained to obtain the total perceptual difference, thus simulating the human visual system's perception and judgment of image similarity.

[0166] This phase continues training for 200,000 steps, with the same learning rate scheduling as the first phase, but a 1-step DDIM sampler is used for validation.

[0167] This invention uses SAR images of known optical images as training data, extracts noise from the optical images in the training data to obtain training noise images, and extracts image features from the SAR images in the training data as training image features. The diffusion model is then trained in the two-stage process using the training noise images and training image features. After training, the optimal model parameters are saved and loaded into the inference environment. The model supports export in multiple formats to adapt to different deployment platforms.

[0168] For newly input SAR and optical images, image feature extraction and noise extraction are performed after preprocessing. The extracted image features and target noise image are directly input into the trained diffusion model. The diffusion model uses U-Net and 1-step DDIM to predict from noise to a clear image in one step, and outputs the corresponding virtual optical image. .

[0169] The virtual optical image output by the model is denormalized to the pixel value range of [0, 255] and saved as a standard image format (such as PNG, JPEG). The system supports batch processing and can continuously process multiple SAR images and output the corresponding optical image sequence.

[0170] The effectiveness of the space-frequency guided SAR-to-optical image conversion method proposed in this invention is verified and illustrated using the SEN12 and QXS-SAROPT datasets:

[0171] (1) Quantitative evaluation: Five metrics were calculated between the virtual optical image generated using the method provided in this embodiment of the invention and the real optical image: peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), learned perceptual patch similarity (LPIPS), Fraser initiation distance (FID), and mean squared error (L2). Experiments show that the method proposed in this embodiment of the invention achieves optimal performance on both datasets;

[0172] (2) Visual comparison: The virtual optical image generated by the method provided in the embodiments of the present invention is compared with baseline methods such as Pix2Pix, SPADE, SR3, and ControlNet. The method proposed in the embodiments of the present invention performs better in terms of edge sharpness, structural integrity and texture detail.

[0173] (3) Efficiency analysis: The average inference time for a single 256×256 image using the method proposed in this embodiment is 0.21 seconds, which is comparable to the GAN method and meets the real-time requirements.

[0174] Through the above steps, the embodiments of the present invention fully realize the SAR to optical image conversion based on space-frequency integration, which significantly improves the computational efficiency while ensuring the generation quality, and provides a practical and feasible technical solution for remote sensing image processing.

[0175] This invention proposes a space-frequency guided SAR-to-optical image conversion method. Through forward noise addition and reverse denoising processes, combined with multi-scale space-frequency conditional guidance, it achieves high-quality, high-efficiency one-step image conversion. This solves problems such as severe speckle noise interference, low cross-modal alignment efficiency, and high computational complexity in SAR images. It overcomes the limitations of low signal-to-noise ratio and multiplicative noise in SAR images on feature extraction, achieving multiple benefits including high-fidelity image generation, high computational efficiency, strong generalization ability, and ease of deployment. A multi-scale integrated encoder is proposed to extract robust dual-domain features, effectively suppressing SAR speckle noise, maintaining image structural integrity and detail richness, generating high-quality optical images, and solving the problem of structural information degradation in noisy environments. A lightweight dynamic conditional partial injection mechanism is designed, significantly reducing inference time and computational resource requirements, improving computational efficiency, and solving the problems of computational redundancy and insufficient semantic alignment in traditional conditional injection methods. This method achieves a significant improvement in computational efficiency while maintaining generation quality. Furthermore, thanks to its lower parameter count and computational complexity, the model is suitable for deployment on edge devices and remote sensing satellite platforms, providing a feasible real-time image conversion solution for resource-constrained remote sensing platforms.

[0176] This invention also proposes a space-frequency guided SAR-to-optical image conversion system, comprising:

[0177] Image acquisition unit, used to acquire SAR images and optical images;

[0178] The image feature extraction unit is used to extract image features from SAR images;

[0179] The noise feature extraction unit is used to extract noise from the optical image to obtain a target noise image; and to extract noise features from the target noise image.

[0180] The diffusion unit is used to segment noise features along the channel dimension into processing path features and shortcut path features; perform residual concatenation between image features and processing path features to obtain residual fusion features; enhance the residual fusion features to obtain enhanced features; concatenate and fuse the enhanced features with the shortcut path features and add them to the noise features to obtain injected features; decode the injected features to obtain the predicted noise image; and subtract the predicted noise image from the target noise image to obtain the virtual optical image.

[0181] It should be noted that the SAR-to-optical image conversion system based on space-frequency guidance provided in the above embodiments is only illustrated by the division of the functional modules described above when converting SAR images into virtual optical images. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the SAR-to-optical image conversion system based on space-frequency guidance provided in the above embodiments and the SAR-to-optical image conversion method embodiments based on space-frequency guidance belong to the same concept, and the specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0182] The present invention also discloses a computer device, the device comprising:

[0183] A processor, adapted to execute computer programs;

[0184] A computer-readable storage medium storing a computer program, which, when executed by the processor, implements the space-frequency guided SAR-to-optical image conversion method proposed in this embodiment of the invention.

[0185] The present invention also discloses a computer-readable storage medium storing a computer program adapted for loading and execution by a processor of the space-frequency guided SAR to optical image conversion method proposed in the embodiments of the present invention.

[0186] The present invention also discloses a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the space-frequency guided SAR to optical image conversion method proposed in the embodiments of the present invention.

[0187] The method proposed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor. The software modules can reside in readily available storage media in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, detailed descriptions are omitted here.

[0188] Those skilled in the art will recognize that the units and algorithm steps described in conjunction with the embodiments herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0189] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A SAR-to-optical image conversion method based on space-frequency guidance, characterized in that, include: Acquire SAR and optical images; Extracting image features from SAR images; Noise is extracted from the optical image to obtain the target noise image; Feature extraction is performed on the target noisy image to obtain noise features; The noise features are segmented along the channel dimension into processing path features and shortcut path features; Residual fusion features are obtained by performing residual concatenation between image features and processing path features. Enhance the residual fusion features to obtain the enhanced features; The enhanced features are concatenated and fused with the shortcut path features, and then added to the noise features to obtain the injected features; Decode the injected features to obtain the predicted noise image; The virtual optical image is obtained by subtracting the predicted noise image from the target noise image.

2. The SAR-to-optical image conversion method based on space-frequency guidance as described in claim 1, characterized in that, Image features are upsampled and then projected through a lightweight linear layer and weighted to obtain fusion conditional features; The residual fusion features are obtained by adding the fusion condition features and the processing path features.

3. The SAR-to-optical image conversion method based on space-frequency guidance as described in claim 1, characterized in that, Channel attention and spatial attention modulation are applied to the residual fusion features to obtain the enhanced features.

4. The SAR-to-optical image conversion method based on space-frequency guidance as described in claim 1, characterized in that, Extracting multi-scale spatial and frequency domain features from SAR images; Multi-scale spatial features and multi-scale frequency domain features are fused to obtain the image features of SAR images.

5. The SAR-to-optical image conversion method based on space-frequency guidance as described in claim 4, characterized in that, Convert SAR images to single-channel grayscale images; Obtain the complex spectrum of a single-channel grayscale image at the frequency coordinates; Calculate the amplitude spectrum of the complex spectrum and perform a logarithmic transformation on the amplitude spectrum to obtain the logarithmic amplitude spectrum; Based on the logarithmic amplitude spectrum, obtain the frequency image of the three channels; Multi-scale feature extraction is performed on the frequency images of the three channels to obtain the multi-scale frequency domain features of the SAR images.

6. The SAR-to-optical image conversion method based on space-frequency guidance as described in claim 1, characterized in that, Encode the target noisy image to obtain encoded features; Further feature extraction is performed on the encoded features to obtain noise features.

7. A SAR-to-optical image conversion system based on space-frequency guidance, characterized in that, include: Image acquisition unit, used to acquire SAR images and optical images; The image feature extraction unit is used to extract image features from SAR images; The noise feature extraction unit is used to extract noise from the optical image to obtain a target noise image; and to extract noise features from the target noise image. The diffusion unit is used to segment noise features along the channel dimension into processing path features and shortcut path features; the image features and processing path features are residually concatenated to obtain residual fusion features; the residual fusion features are enhanced to obtain enhanced features; The enhanced features are concatenated and fused with the shortcut path features, and then added to the noise features to obtain the injected features; The injected features are decoded to obtain the predicted noise image; the predicted noise image is subtracted from the target noise image to obtain the virtual optical image.

8. An electronic device, characterized in that, The device includes: A processor, adapted to execute computer programs; A computer-readable storage medium storing a computer program, which, when executed by the processor, implements the space-frequency guided SAR-to-optical image conversion method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted to be loaded by a processor and executed by the space-frequency guided SAR to optical image conversion method according to any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the space-frequency guided SAR to optical image conversion method according to any one of claims 1-6.

Images