A weakly supervised SAR to visible light image translation method that incorporates scene information

Abstract

This invention discloses a weakly supervised SAR-to-visible light image translation method that integrates scene information, comprising the following steps: designing a noise suppression module; updating the parameters of the noise reduction module according to the results of the entire network; constructing a scene information fusion generator; combining local and global features through self-attention and cross-attention; and using a combined discriminator with two modules, Patch-GAN and a scene discriminator, to achieve detailed discrimination of the authenticity of the generated image and the scene, thereby guiding the generator to learn the scene features of the image and generate a more realistic image that conforms to the scene. Because this invention integrates scene auxiliary information and combines a backbone network with a scene attention mechanism, it effectively extracts effective features of the local and global scene of the image. Simultaneously, it constructs a noise suppression module to remove speckle noise, and combines it with the prediction network to update parameters, accelerating network convergence and realizing the generation of SAR-to-visible light images of complex scenes.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a method for weakly supervised SAR to visible light image translation that integrates scene information. Background Technology

[0002] Current image translation methods based on generative adversarial networks (GANs) typically require paired datasets. However, in practical applications, it is difficult to simultaneously acquire paired datasets for SAR and visible light images. Even if paired datasets are available, significant manual labor is required. Compared to visible light images, SAR images offer a wider field of view, higher viewing angle, richer content, and clearer scene distinctions. Directly applying generators suitable for natural image translation to SAR-to-visible light image translation results in a simplistic network structure and insufficient learning ability. In unsupervised tasks, differences in imaging mechanisms may lead to incorrect translation of SAR images by the network, affecting subsequent interpretation tasks.

[0003] The paper "Guo J, Li J, Fu H, et al. Alleviating semantics distortion in unsupervised low-level image-to-image translation via structure consistency constraint[C]. Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2022:18249-18259." proposes a method to mitigate semantic distortion in unsupervised low-level image-to-image translation through structure consistency constraints. This method introduces structure consistency constraints to ensure structural consistency between the input and generated images, thereby reducing semantic distortion. Simultaneously, a novel metric is proposed to evaluate the quality of image translation and better capture semantic information. However, this method has the following drawbacks: First, when dealing with certain complex image scenes or diverse datasets, structure consistency constraints may lead to the loss of details in image translation, resulting in local and global color errors, affecting translation quality. Second, introducing structure consistency constraints increases the computational complexity of the model, thus extending training and inference time. This limitation restricts the applicability of this method on large-scale datasets. Summary of the Invention

[0004] To overcome the shortcomings of existing technologies, this invention provides a weakly supervised SAR-to-visible light image translation method that integrates scene information, comprising the following steps: A noise suppression module is designed, combining wavelet decomposition, threshold denoising, and deep learning methods to effectively reduce speckle noise in SAR images; the parameters of the denoising module are updated following the results of the entire network, making the network more focused on effective information in the image, improving network performance and stability; a scene information fusion generator is constructed, and a scene-guided feature extraction module is rebuilt on the Cycle-GAN network structure, possessing stronger image feature extraction capabilities, reducing translation scene errors, and improving the ability to process complex scene information; a scene filtering residual block and a scene fusion attention block are employed. The scene filtering residual block achieves the fusion of image features and global scene features, while the scene fusion attention block combines local and global features through self-attention and cross-attention, better integrating the relationship between the image and the scene. A combined discriminator, through two modules—Patch-GAN and a scene discriminator—achieves detailed discrimination of the authenticity of the generated image and the scene, thereby guiding the generator to learn the scene features of the image and generate more realistic images that conform to the scene. Because this invention integrates scene-aided information and combines a backbone network with a scene attention mechanism, it effectively extracts effective features of local and global scenes in the image. At the same time, it constructs a noise suppression module to remove speckle noise and combines it with the prediction network to update parameters, thereby accelerating network convergence and realizing the generation of SAR to visible light images of complex scenes.

[0005] The technical solution adopted by this invention to solve its technical problem is as follows:

[0006] Step 1: Building the noise suppression module;

[0007] The input SAR image is transformed and decomposed into four sub-band images. Each sub-band contains information of different frequencies in the original image. The sub-band images are then input into the prediction network to predict the filtering threshold of each sub-band and perform threshold filtering. Finally, the sub-band images are inversely transformed to obtain the denoised SAR image.

[0008] Step 2: Constructing the scene information fusion generator;

[0009] A scene information fusion generator was constructed using a Cycle-GAN-based network architecture; Table 1 shows the structure of the scene information fusion generator.

[0010] Table 1 Network Structure of Scene Information Fusion Generator

[0011]

[0012] The input image undergoes initial feature extraction via a downsampling module, reducing the size of the feature map while increasing its dimensionality. The extracted features are then used by the input scene to guide the feature extraction module, which includes a scene filtering residual block and a scene fusion attention block.

[0013] The scene filtering residual block consists of two 3×3 convolutional layers and one fully connected layer, which fuses scene features with image features in the form of global features. The image features are initially extracted by the convolutional layers and then fused with the scene features processed by the fully connected layers through channel addition. The fused features are then passed through the convolutional layers again to obtain the output features of the residual part. The output is the sum of the original features and the residual part features.

[0014] The scene fusion attention block contains two multi-head attention layers: self-attention and cross-attention. Through self-attention and cross-attention, scene features and image features are fused in a way that combines local and global features. The image features first pass through the first self-attention layer to obtain feature output, and then cross-attention is calculated with the input scene features to achieve feature fusion. The fused features pass through a feedforward layer to obtain the final output.

[0015] The output image features are upsampled to map the image information to the corresponding image domain, generating a more realistic image. An adversarial loss function is used to train a GAN network, which is then applied to image mapping. For the mapping function G from visible light to SAR images... o2s :OPT→SAR and its discriminator D SAR The objective is represented as:

[0016]

[0017] Where OPT represents visible light image, SAR represents SAR image, and G... o2s D represents a generator that produces SAR images from visible light. SAR The discriminator representing the SAR image. This represents the operation of extracting a SAR vector sar from the prior distribution pdata(sar) in the latent space. This represents the operation of drawing an OPT vector opt from the prior distribution pdata(opt) in the latent space; G o2s Translate visible light images into SAR images, D SAR Used to distinguish between translation results and real images; for the SAR-to-visible light mapping function G s2o :SAR→OPT introduces a similar adversarial loss, namely

[0018] Step 3: Construct the combined discriminator;

[0019] The combined discriminator consists of two discriminator modules;

[0020] The first discriminator module is Patch-GAN, which transforms the original image into a 30×30 discriminator matrix through convolution. Each pixel in the matrix has a receptive field of 70×70 pixels, which is used to jointly determine the authenticity of the original image. The second discriminator module is a scene discriminator, which consists of seven 7×7 convolutional layers and one fully connected layer. The discriminator extracts features through convolution, and finally passes through a softmax layer to obtain the final scene probability discrimination result.

[0021] Preferably, in step 2, two loss functions are introduced: cycle consistency loss and style loss;

[0022] The cycle consistency loss function minimizes the loss of an image through a transformation cycle (sar→G). s2o (sar)→G o2s (G s2o The L1 loss of the images before and after (sar)≈sar) is used to guide network training, and is expressed as:

[0023]

[0024] Where, λ s2o and λ o2s The weights representing the losses in the two loop processes;

[0025] The style loss function is expressed as:

[0026]

[0027] The final loss is a combined loss function, expressed as:

[0028]

[0029] Where, λ cyc , λ id These represent the weights of the cycle consistency loss and style loss, respectively. The network is trained by optimizing this combined loss function.

[0030] Preferably, in step 3, scene classification loss is used to discriminate scene information in the image, and the cross-entropy loss function is calculated to guide network training, making the network translation result more consistent with the original scene; the scene classification loss function is expressed as:

[0031]

[0032] Where D SAR_cls D OPT_clsThis represents a scene discriminator for SAR and visible light images, where sce represents the real scene information of the image, and D... SAR_cls (sce|sar) and D OPT_cls (sce|opt) represent the discrimination results of the visible light and SAR scene discriminators on the input image, respectively;

[0033] The final combined loss function is expressed as:

[0034]

[0035] Where, λ cls The weights represent the scene classification loss.

[0036] Preferably, the λ cls Set to 0.01.

[0037] The beneficial effects of this invention are as follows:

[0038] This invention addresses the limitations of paired data by employing a weakly supervised approach for SAR-to-visible light image translation. By training the network with unpaired image information and other simple auxiliary information, translation quality is improved with minimal increase in manual labor costs, effectively enhancing model performance. Furthermore, a denoising module is introduced to address the characteristics of SAR images, effectively reducing the impact of speckle noise. A redesigned network structure with scene information fusion capabilities allows the network to focus more on scene information, further improving the translation performance of unpaired SAR-to-visible light images. Translation experiments on the public datasets SEN1-2 and QXS-SAROPT demonstrate improvements of 6.88, 14.58, and 8.37 percentage points respectively compared to existing methods in image quality assessment metrics PSNR, SSIM, and LPIPS. Attached Figure Description

[0039] Figure 1 This is a structural diagram of the scene filtering residual block and the scene fusion attention block of the present invention. Detailed Implementation

[0040] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0041] Existing SAR-to-visible light image translation methods suffer from limitations in obtaining paired datasets and issues such as blurred details and inconsistent colors after translation. These problems reduce the accuracy and realism of the translated SAR-to-visible light image results, impacting their practical value. To address these issues, this invention discloses a SAR-to-visible light image translation method based on generative adversarial networks.

[0042] The technical solution adopted includes the following steps: noise suppression module construction, scene information fusion generator construction, and combined discriminator construction.

[0043] (a) A noise suppression module is constructed. The SAR image is decomposed into four sub-bands using wavelet transform, thereby enabling the extraction and processing of different frequency components. A deep learning prediction network is used to predict the threshold, effectively reducing speckle noise in the SAR image. This invention generates a denoised image through threshold filtering and updates the parameters of the denoising module together with the entire network, achieving effective suppression and denoising of SAR image noise.

[0044] (b) A scene information fusion generator based on a Cycle-GAN network structure is constructed, which can effectively process complex SAR images. A scene-guided feature extraction module is used to achieve more complex and powerful image feature extraction, while effectively reducing scene translation errors when using coarse scene labels. The input image undergoes initial feature extraction via a downsampling module, followed by processing by the scene-guided feature extraction module, which includes a scene filtering residual block and a scene fusion attention block. The scene filtering residual block consists of multiple convolutional and fully connected layers to achieve global feature fusion; the scene fusion attention block includes self-attention and cross-attention layers, organically combining local and global features. Finally, an upsampling layer generates a more realistic image, making the network more focused on the relationship between the image and the scene, significantly improving the acquisition and expression of realistic information.

[0045] (c) A combined discriminator is constructed, consisting of two independent modules: Patch-GAN and a scene discriminator. By simultaneously judging the authenticity of the generated image and the scene, the generator can learn scene features more meticulously, thus generating more realistic images that conform to actual scenes. Patch-GAN transforms the original image into a 30×30 discrimination matrix, jointly judging the authenticity of the original image in a more detailed way. The scene discriminator consists of seven 7×7 convolutional layers and one fully connected layer, extracting features through convolution operations and finally outputting the scene probability discrimination result. The architecture of the combined discriminator enables the model to comprehensively consider the authenticity of the image and scene information, effectively improving the quality and realism of the generated image.

[0046] Example:

[0047] 1. A weakly supervised SAR-to-visible light image translation network that integrates scene information;

[0048] This invention constructs a weakly supervised SAR-to-visible light image translation network that fuses scene information. The network comprises three parts: a noise suppression module, a scene information fusion generator, and a combined discriminator. The input source SAR image and coarse scene information are preprocessed and randomly augmented before being fed into the network. The preprocessing includes bicubic interpolation, random left-right flipping, and random cropping, adjusting the image size from 256×256 to 286×286, and then cropping it back to 256×256 to reduce network overfitting. The processed image is then noise-suppressed by the noise suppression module before being input into the SAR-to-visible light translation network. The scene information fusion generator combines the scene information to convert the SAR image into a pseudo-visible light image. This pseudo-visible light image is simultaneously input into both the visible light image combined discriminator and the visible light-to-SAR translation network. The visible light combined discriminator determines the authenticity of the image and the scene it belongs to, making the generated pseudo-visible light image closer to the real visible light image. The visible light-to-SAR image translation network then combines the generated pseudo-visible light image with the input scene information and converts it back into a pseudo-SAR image. Cyclic consistency loss improves the stability of the entire network by minimizing the L1 loss between the input and output images in each cycle.

[0049] 2. Construction of the noise suppression module;

[0050] The noise suppression module combines wavelet thresholding and deep learning methods to optimize thresholds based on data, effectively reducing the interference of speckle noise in SAR images on subsequent networks. This module performs image transformation and wavelet decomposition on the input SAR image to obtain four sub-band images, each containing information from different frequencies of the original image. These sub-band images are then input into a prediction network to predict and obtain filtering thresholds for each sub-band, followed by threshold filtering. Finally, the sub-band images are inversely transformed to obtain the denoised SAR image. The parameters of the denoising module are updated based on the results of the entire network. The denoised image allows subsequent networks to focus more on the effective information within the image, improving network performance and stability.

[0051] 3. Construction of a scene information fusion generator;

[0052] This invention constructs a scene information fusion generator using a Cycle-GAN-based network architecture. This generator includes a scene-guided feature extraction module, featuring a more complex structure and stronger image feature extraction capabilities, making it more suitable for processing complex SAR images. Furthermore, by utilizing coarse scene labels, it effectively reduces translation errors. Table 1 details the structure of the scene information fusion generator, which is more complex than traditional generators and possesses the ability to fuse image scene features.

[0053] Table 1 Network Structure of Scene Information Fusion Generator

[0054]

[0055] The input image undergoes initial feature extraction via a downsampling module, reducing the size of the feature map while increasing its dimensionality. The extracted features are then used by an input scene-guided feature extraction module, which includes a scene filtering residual block and a scene fusion attention block. A detailed network structure diagram is shown below. Figure 1 As shown.

[0056] The scene filtering residual block consists of two 3×3 convolutional layers and one fully connected layer, fusing scene features with image features as global features. Image features undergo initial feature extraction via convolutional layers and are then fused with the scene features processed by the fully connected layer through channel addition. The fused features are then passed through another convolutional layer to obtain the output features of the residual part. The output is the sum of the original features and the residual features.

[0057] The scene fusion attention block comprises two multi-head attention layers: self-attention and cross-attention. Through self-attention and cross-attention, scene features are fused with image features by combining local and global features. Image features first pass through the first self-attention layer to obtain feature output, and then cross-attention is calculated with the input scene features to achieve feature fusion. The fused features then pass through a feedforward layer to obtain the final output.

[0058] The output image features are upsampled to map the image information to the corresponding image domain, generating a more realistic image. An adversarial loss function is used to train a GAN network, which is then applied to image mapping. For the mapping function G from visible light to SAR images... o2s :OPT→SAR and its discriminator D SAR The objective is represented as:

[0059]

[0060] Where OPT represents visible light image, SAR represents SAR image, and G... o2s D represents a generator that produces SAR images from visible light. SAR G represents the discriminator of the SAR image. o2s An attempt was made to translate a visible light image into a SAR image, D SAR This is used to distinguish between the translation result and the real image. For the SAR-to-visible light mapping function G... s2o :SAR→OPT introduces a similar adversarial loss, namely

[0061] The above design allows the network to focus on the relationship between the image and the scene, obtaining more realistic information from the image compared to traditional methods. Furthermore, this invention introduces two key loss functions: cycle consistency loss and style loss. Cycle consistency loss aims to ensure consistency between the generated visible light image and the original SAR image, thereby reducing information loss and semantic distortion; while style loss helps capture the stylistic features of the image, improving the visual quality and realism of the generated image.

[0062] Cyclic consistency loss function is often used in unpaired tasks, by minimizing the loss of an image through a transformation cycle (sar→G). s2o (sar)→G o2s (G s2o The L1 loss of the images before and after (sar)≈sar) is used to guide network training, and is expressed as:

[0063]

[0064] Where, λ s2o and λ o2s This represents the weights lost in the two cyclic processes.

[0065] The style loss function is typically expressed as:

[0066]

[0067] The final loss is a combined loss function, expressed as:

[0068]

[0069] Where, λ cyc , λ id These represent the weights of the cycle consistency loss and style loss, respectively. This invention trains the network by optimizing this combined loss function. By using the combined loss function, the network can achieve more accurate and higher-quality image translation in unsupervised environments.

[0070] 4. Construction of the combined discriminator;

[0071] The combined discriminator contains two discriminator modules. By simultaneously judging the authenticity of the generated image and the scene, it guides the generator to learn the scene features of the image in a more detailed way, and generates more realistic images that match the scene.

[0072] The first discriminator constructed is Patch-GAN. The original image is transformed into a 30×30 discriminant matrix through convolution operations. Each pixel in the matrix has a receptive field of 70×70 pixels, similar to the original image, and the original image is jointly judged for authenticity by dividing it into blocks. The second discriminator is a scene discriminator, consisting of seven 7×7 convolutional layers and one fully connected layer. Its specific structure is shown in Table 2, where scene_num represents the number of scene categories. The discriminator extracts features through convolution operations and finally passes them through a softmax layer to obtain the final scene probability discrimination result. Under this architecture, the model can judge the quality of the image from two aspects: image authenticity and image scene.

[0073] Table 2. Scene Discriminator Network Structure

[0074]

[0075]

[0076] This invention employs scene classification loss to discriminate scene information in images, and uses the cross-entropy loss function to guide network training, making the network's translation results more consistent with the original scene. The scene classification loss function is expressed as:

[0077]

[0078] Where D SAR_cls D OPT_cls This represents a scene discriminator for SAR and visible light images, where sce represents the real scene information of the image, and D... SAR_cls (sce|sar) and D OPT_cls (sce|opt) represent the discrimination results of the visible light and SAR scene discriminators on the input image, respectively.

[0079] The final combined loss function is expressed as:

[0080]

[0081] Where, λ cyc , λ id , λ cls λ represents the weights of the cycle consistency loss, style loss, and scene classification loss, respectively. In the experiments described in the paper, λ... cls Set to 0.01 to train the network by optimizing the combined loss function.

Claims

1. A weakly supervised SAR to visible light image translation method that integrates scene information, characterized in that, Includes the following steps: Step 1: Building the noise suppression module; The input SAR image is transformed and decomposed into four sub-band images. Each sub-band contains information of different frequencies in the original image. The sub-band images are then input into the prediction network to predict the filtering threshold of each sub-band and perform threshold filtering. Finally, the sub-band images are inversely transformed to obtain the denoised SAR image. Step 2: Constructing the scene information fusion generator; A scene information fusion generator was constructed using a Cycle-GAN-based network architecture; Table 1 shows the structure of the scene information fusion generator. Table 1 Network Structure of Scene Information Fusion Generator The input image undergoes initial feature extraction via a downsampling module, reducing the size of the feature map while increasing its dimensionality. The extracted features are then used by the input scene to guide the feature extraction module, which includes a scene filtering residual block and a scene fusion attention block. The scene filtering residual block consists of two 3×3 convolutional layers and one fully connected layer, which fuses scene features with image features in the form of global features; Image features are initially extracted through convolutional layers and then fused with scene features processed by fully connected layers through channel addition. The fused features are then passed through convolutional layers again to obtain the output features of the residual part. The output is the sum of the original features and the residual part features. The scene fusion attention block contains two multi-head attention layers: self-attention and cross-attention. Through self-attention and cross-attention, scene features and image features are fused in a way that combines local and global features. The image features first pass through the first self-attention layer to obtain feature output, and then cross-attention is calculated with the input scene features to achieve feature fusion. The fused features pass through a feedforward layer to obtain the final output. The output image features are upsampled to map the image information to the corresponding image domain, generating a more realistic image. An adversarial loss function is used to train a GAN network, which is then applied to image mapping. For the mapping function G from visible light to SAR images... o2s :OPT→SAR and its discriminator D SAR The objective is represented as: Where OPT represents visible light image, SAR represents SAR image, and G... o2s D represents a generator that produces SAR images from visible light. SAR The discriminator representing the SAR image. This represents the operation of extracting a SAR vector sar from the prior distribution pdata(sar) in the latent space. This represents the operation of drawing an OPT vector opt from the prior distribution pdata(opt) in the latent space; G o2s Translate visible light images into SAR images, D SAR Used to distinguish between translation results and real images; for the SAR-to-visible light mapping function G s2o :SAR→OPT introduces a similar adversarial loss, i.e. Step 3: Construct the combined discriminator; The combined discriminator consists of two discriminator modules; The first discriminator module is Patch-GAN, which transforms the original image into a 30×30 discriminator matrix through convolution. Each pixel in the matrix has a receptive field of 70×70 pixels, which is used to jointly determine the authenticity of the original image. The second discriminator module is a scene discriminator, which consists of seven 7×7 convolutional layers and one fully connected layer. The discriminator extracts features through convolution, and finally passes through a softmax layer to obtain the final scene probability discrimination result.

2. The weakly supervised SAR to visible light image translation method according to claim 1, characterized in that, In step 2, two loss functions are introduced: cycle consistency loss and style loss; The cycle consistency loss function minimizes the loss of an image through a transformation cycle (sar→G). s2o (sar)→G o2s (G s2o The L1 loss of the images before and after (sar)≈sar) is used to guide network training, and is expressed as: Where, λ s2o and λ o2s The weights representing the losses in the two cyclic processes; The style loss function is expressed as: The final loss is a combined loss function, expressed as: Where, λ cyc , λ id These represent the weights of the cycle consistency loss and style loss, respectively. The network is trained by optimizing this combined loss function.

3. The weakly supervised SAR to visible light image translation method according to claim 2, characterized in that, In step 3, scene classification loss is used to discriminate scene information in the image, and cross-entropy loss function is calculated to guide network training, making the network translation result more consistent with the original scene; the scene classification loss function is expressed as: Where D SAR_cls D OPT_cls This represents a scene discriminator for SAR and visible light images, where sce represents the real scene information of the image, and D... SAR_cls (sce|sar) and D OPT_cls (sce|opt) represent the discrimination results of the visible light and SAR scene discriminators on the input image, respectively; The final combined loss function is expressed as: Where, λ cls The weights represent the scene classification loss.

4. The weakly supervised SAR to visible light image translation method according to claim 3, characterized in that, The λ cls Set to 0.01.

Images