Weak supervision SAR to visible light image translation method fusing scene information
By designing a noise suppression module and a scene information fusion generator in SAR to visible image translation, combining wavelet decomposition and deep learning, the problem of insufficient learning ability and detail loss of unsupervised SAR image translation is solved, and high-quality image translation effect is achieved , improve PSNR, SSIM and LPIPS indicators.
Patent Information
- Application Number
- CN202410771516.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-14
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2044-06-14
Smart Images

Figure CN118918200A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of image processing, and in particular relates to a weakly supervised SAR to visible light image translation method that fuses scene information. Background Art
[0002] At present, image translation methods based on generative adversarial networks usually require paired data sets. However, in practical applications, paired data of SAR images and visible light images are difficult to obtain at the same time. Even if paired data is available, a lot of labor costs are required. Compared with visible light images, SAR images have a wide field of view, high viewing angle, rich content, and obvious scene distinction. Directly applying the generator suitable for natural image translation to SAR to visible light image translation will result in a simple network structure and insufficient learning ability. In unsupervised tasks, differences in imaging mechanisms may cause the network to incorrectly translate SAR images, affecting subsequent interpretation tasks.
[0003] The paper "Guo J, Li J, Fu H, et al. Alleviating semantics distortion in unsupervised low-level image-to-image translation via structure consistency constraint [C]. Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2022: 18249-18259." proposes a method to mitigate semantic distortion in unsupervised low-level image-to-image translation via structure consistency constraint. This method introduces a structure consistency constraint to ensure that the structure between the input image and the generated image remains consistent, thereby reducing semantic distortion. At the same time, a new metric is proposed to evaluate the quality of image translation to better capture semantic information. However, this method has the following problems: First, when dealing with certain complex image scenes or diverse data sets, the structure consistency constraint may cause the loss of details in image translation, local and global color errors, and affect the translation quality. Second, the introduction of the structure consistency constraint will increase the computational complexity of the model, thereby prolonging the training and inference time. This situation limits the applicability of this method on large-scale data sets. Summary of the invention
[0004] In order to overcome the shortcomings of the prior art, the present invention provides a weakly supervised SAR to visible light image translation method that integrates scene information, including the following steps: designing a noise suppression module, combining wavelet decomposition, threshold denoising and deep learning methods, effectively reducing the coherent speckle noise in the SAR image; the parameters of the denoising module are updated following the results of the entire network, so that the network pays more attention to the effective information in the image, and improves the performance and stability of the network; constructing a scene information fusion generator, reconstructing the scene guided feature extraction module on the network structure of Cycle-GAN, having a more powerful image feature extraction capability, reducing translation scene errors, and improving the processing capability of complex scene information; using scene filtering residual blocks and scene fusion attention blocks, using scene filtering residual blocks to achieve the fusion of image features and global scene features, in the scene fusion attention block, the combination of local features and global features is achieved through self-attention and cross-attention, and the relationship between the image and the scene is better integrated. The combined discriminator realizes the detailed discrimination of the authenticity and scene of the generated image through the two modules of Patch-GAN and scene discriminator, thereby guiding the generator to learn the scene features of the image and generate a real image that is more in line with the scene. Since the present invention integrates scene auxiliary information and combines it with the backbone network of the scene attention mechanism, it can effectively extract the effective features of the local and global scenes of the image, and at the same time construct a noise suppression module for removing coherent speckle noise. Combined with the prediction network update parameters, it accelerates the network convergence and realizes the generation of SAR to visible light images of complex scenes.
[0005] The technical solution adopted by the present invention to solve the technical problem is as follows:
[0006] Step 1: Noise suppression module construction;
[0007] The input SAR image is transformed and decomposed by wavelet to obtain four sub-band images. Each sub-band contains information of different frequencies of the original image. The sub-band images are then input into the prediction network to predict the filtering threshold of each sub-band and perform threshold filtering. Finally, the sub-band images are inversely transformed to obtain the denoised SAR image.
[0008] Step 2: Construction of scene information fusion generator;
[0009] The scene information fusion generator is constructed using the network structure based on Cycle-GAN; Table 1 shows the structure of the scene information fusion generator;
[0010] Table 1. Scene information fusion generator network structure
[0011]
[0012] The input image is initially extracted through the downsampling module to reduce the size of the feature map while increasing the dimension; the extracted features are input into the scene guided feature extraction module, which contains the scene filtering residual block and the scene fusion attention block;
[0013] The scene filtering residual block consists of two 3×3 convolutional layers and one fully connected layer, which fuses the scene features with the image features in the form of global features. The image features are extracted through the convolutional layer and then fused with the scene features processed by the fully connected layer through channel addition. The fused features are then passed through the convolutional layer to obtain the output features of the residual part, which is the sum of the original features and the residual features.
[0014] The scene fusion attention block contains two multi-head attention layers: self-attention and cross-attention. Through self-attention and cross-attention, scene features are fused with image features in a way that combines local features and global features. Image features first pass through the first self-attention layer to obtain feature output, and then calculate cross-attention with the input scene features to achieve feature fusion. The fused features pass through the feedforward layer to obtain the final output.
[0015] The output image features are mapped to the corresponding image domain through the upsampling layer to generate a more realistic image; the GAN network is trained using the adversarial loss function and applied to image mapping; the mapping function G from visible light to SAR image o2s :OPT→SAR and its discriminator D SAR , the target is expressed as:
[0016]
[0017] Where OPT represents visible light image, SAR represents SAR image, G o2s represents the generator for generating SAR images from visible light, D SAR represents the discriminator of SAR image, represents the operation of extracting a SAR vector sar from the prior distribution pdata(sar) in the latent space, represents the operation of extracting an OPT vector opt from the prior distribution pdata(opt) in the latent space; G o2s Translate visible light images into SAR images, D SAR Used to distinguish the translation result from the real image; for the mapping function G from SAR to visible light s2o :SAR→OPT introduces a similar adversarial loss, namely
[0018] Step 3: Combined discriminator construction;
[0019] The combined discriminator contains two discriminator modules;
[0020] The first discriminator module is Patch-GAN. The original image is converted into a discriminant matrix of size 30×30 through convolution operation. Each pixel in the matrix has a receptive field of size 70×70 of the original image, which is used to jointly discriminate the authenticity of the original image in blocks. The second discriminator module is the scene discriminator, which consists of seven 7×7 convolutional layers and one fully connected layer. The discriminator extracts features through convolution operation and finally obtains the final scene probability discrimination result through a layer of softmax.
[0021] Preferably, in step 2, two loss functions are introduced: cycle consistency loss and style loss;
[0022] The cycle consistency loss function is calculated by minimizing the s2o (sar) → G o2s (G s2o The L1 loss of the images before and after (sar))≈sar) is used to guide network training, expressed as:
[0023]
[0024] Among them, λ s2o and λ o2s Represents the weight of the loss of the two cyclic processes;
[0025] The style loss function is expressed as:
[0026]
[0027] The final loss is a combined loss function expressed as:
[0028]
[0029] Among them, λ cyc ,λ id Represent the weights of cycle consistency loss and style loss respectively, and the network is trained by optimizing this combined loss function.
[0030] Preferably, in step 3, scene classification loss is used to discriminate the scene information of the image, and the network training is guided by calculating the cross entropy loss function so that the result of network translation is more consistent with the original scene; the scene classification loss function is expressed as:
[0031]
[0032] Where D SAR_cls , D OPT_clsrepresents the scene discriminator of SAR image and visible light image, sce represents the real scene information of the image, and D SAR_cls (sce|sar) and D OPT_cls (sce|opt) represents the discrimination results of the visible light and SAR scene discriminators on the input image respectively;
[0033] The final combined loss function is expressed as:
[0034]
[0035] Among them, λ cls Represents the weight of the scene classification loss.
[0036] Preferably, the λ cls Set to 0.01.
[0037] The beneficial effects of the present invention are as follows:
[0038] In view of the limitation of paired data, the method of the present invention adopts a weakly supervised method to study the SAR to visible light image translation task. By obtaining unpaired image information and other simple auxiliary information to train the network, the translation quality is improved with almost no increase in labor costs, and the model performance is effectively improved. In addition, the introduction of a denoising module based on the characteristics of SAR images can effectively reduce the impact of coherent speckle noise. By redesigning the network structure with scene information fusion capabilities, the network is more focused on scene information, improving the effect of unpaired SAR to visible light image translation. The translation experiments of the present invention on the public data sets SEN1-2 and QXS-SAROPT show that the three indicators of image quality evaluation, PSNR, SSIM and LPIPS, are improved by 6.88, 14.58 and 8.37 percentage points respectively compared with the literature methods. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] Figure 1 It is a structural diagram of the scene filtering residual block and the scene fusion attention block of the present invention. DETAILED DESCRIPTION
[0040] The present invention is further described below in conjunction with the accompanying drawings and embodiments.
[0041] The existing SAR to visible light image translation methods have the problems of limited paired data set acquisition, blurred details and inconsistent colors after translation, which reduces the accuracy and realism of the SAR to visible light image translation results, affecting the use value in actual scenes. In view of the above problems, the present invention discloses a SAR to visible light image translation method based on a generative adversarial network.
[0042] The adopted technical solution includes the following steps: noise suppression module construction, scene information fusion generator construction, and combined discriminator construction.
[0043] (a) Construct a noise suppression module, decompose the SAR image into four sub-bands by performing wavelet transform decomposition, thereby realizing the extraction and processing of different frequency components. Use a deep learning prediction network to predict the threshold, and effectively perform noise reduction on the coherent speckle noise in the SAR image. The present invention generates a denoised image by threshold filtering, and updates the parameters of the denoising module together with the entire network, thereby realizing effective suppression and denoising of SAR image noise.
[0044] (b) A scene information fusion generator based on the Cycle-GAN network structure is constructed, which can effectively process complex SAR images. Through the scene-guided feature extraction module, more complex and powerful image feature extraction is achieved, and scene translation errors are effectively reduced when using rough scene labels. The input image is initially extracted by the downsampling module, and then processed by the scene-guided feature extraction module, including the scene filtering residual block and the scene fusion attention block. The scene filtering residual block is composed of multiple layers of convolution and fully connected layers to achieve global feature fusion; the scene fusion attention block contains self-attention and cross-attention layers, which organically combines local and global features. Finally, a more realistic image is generated through the upsampling layer, which makes the network pay more attention to the relationship between the image and the scene, significantly improving the acquisition and expression of real information.
[0045] (c) Construct a combined discriminator, which consists of two independent modules: Patch-GAN and scene discriminator. By simultaneously discriminating the authenticity and scene of the generated image, the generator can learn the scene features of the image in more detail, thereby generating a realistic image that is more in line with the actual scene. Patch-GAN converts the original image into a 30×30 discriminant matrix to jointly discriminate the authenticity of the original image in a more detailed manner. The scene discriminator consists of seven 7×7 convolutional layers and one fully connected layer. It extracts features through convolution operations and finally outputs the scene probability discrimination result. The architecture of the combined discriminator enables the model to comprehensively consider the authenticity and scene information of the image, effectively improving the quality and realism of image generation.
[0046] Example:
[0047] 1. Weakly supervised SAR to visible light image translation network integrating scene information;
[0048] The present invention constructs a weakly supervised SAR to visible light image translation network that integrates scene information. The network includes three parts, namely, a noise suppression module, a scene information fusion generator, and a combination discriminator. The source SAR image and the coarse scene information of the SAR image are input, and preprocessed and randomly enhanced before being input into the network. The preprocessing process includes bicubic interpolation, random left and right flipping, and random cropping, which adjusts the image size from 256×256 to 286×286, and then crops it to 256×256 to reduce network overfitting. The processed image is subjected to noise suppression by the noise suppression module and input into the SAR to visible light translation network. The scene information fusion generator converts the SAR image into a pseudo visible light image in combination with the scene information, and the pseudo visible light image is simultaneously input into the visible light image combination discriminator and the visible light to SAR translation network. The visible light combination discriminator makes the generated pseudo visible light image closer to the real visible light image by judging the authenticity of the image and the scene to which it belongs; and the visible light to SAR image translation network combines the generated pseudo visible light image with the input scene information and converts it back into a pseudo SAR image. The cycle consistency loss improves the stability of the entire network by minimizing the L1 loss between the input and output images during each cycle.
[0049] 2. Construction of noise suppression module;
[0050] The noise suppression module combines the wavelet threshold denoising method and the deep learning method. It can optimize the threshold according to the data and better reduce the interference of the coherent speckle noise in the SAR image on the subsequent network. The module transforms the input SAR image and performs wavelet decomposition to obtain four sub-band images. Each sub-band contains information of different frequencies of the original image. The sub-band images are then input into the prediction network to predict the filtering threshold of each sub-band and perform threshold filtering. Finally, the sub-band images are inversely transformed to obtain the denoised SAR image. The parameters of the denoising module are updated with the results of the entire network. The denoised image can make the subsequent network pay more attention to the effective information in the image, improving the performance and stability of the network.
[0051] 3. Construction of scene information fusion generator;
[0052] By using a network structure based on Cycle-GAN, the present invention constructs a scene information fusion generator. The generator includes a scene-guided feature extraction module, has a more complex structure and a more powerful image feature extraction capability, and is more suitable for processing SAR images with complex content. At the same time, when using coarse scene labels, the problem of translation scene errors can be effectively reduced. Table 1 specifically shows the structure of the scene information fusion generator, which is more complex than the traditional generator structure and has the ability to fuse image scene features.
[0053] Table 1. Scene information fusion generator network structure
[0054]
[0055] The input image is initially extracted through the downsampling module to reduce the size of the feature map while increasing the dimension; the extracted features are input into the scene guided feature extraction module, which contains the scene filtering residual block and the scene fusion attention block. The specific network structure diagram is shown in the figure Figure 1 shown.
[0056] The scene filter residual block consists of two 3×3 convolutional layers and a fully connected layer, which fuses the scene features with the image features in the form of global features. The image features are extracted by the convolutional layer and then fused with the scene features processed by the fully connected layer by channel addition. The fused features are then passed through the convolutional layer to obtain the output features of the residual part, which is the sum of the original features and the residual features.
[0057] The scene fusion attention block contains two multi-head attention layers: self-attention and cross-attention. Through self-attention and cross-attention, scene features are fused with image features in a way that combines local features and global features. The image features first pass through the first self-attention layer to obtain feature output, and then calculate cross-attention with the input scene features to achieve feature fusion. The fused features pass through the feedforward layer to obtain the final output.
[0058] The output image features are mapped to the corresponding image domain through the upsampling layer to generate a more realistic image. The GAN network is trained using the adversarial loss function and applied to image mapping. For the mapping function G from visible light to SAR images, o2s :OPT→SAR and its discriminator D SAR , the target is expressed as:
[0059]
[0060] Where OPT represents visible light image, SAR represents SAR image, G o2s represents the generator for generating SAR images from visible light, D SAR represents the discriminator of SAR image, G o2s Try to translate visible light images into SAR images, D SAR It is used to distinguish the translation result from the real image. For the SAR to visible light mapping function G s2o :SAR→OPT introduces a similar adversarial loss, namely
[0061] The above design can make the network pay attention to the relationship between the image and the scene, and can obtain more real information in the image compared with traditional methods. In addition, the present invention also introduces two key loss functions: cycle consistency loss and style loss. Cycle consistency loss aims to ensure the consistency between the generated visible light image and the original SAR image, thereby reducing information loss and semantic distortion; while style loss helps to capture the style characteristics of the image and improve the visual quality and realism of the generated image.
[0062] The cycle consistency loss function is often used in unpaired tasks, by minimizing the number of times an image passes through the conversion cycle (sar→G s2o (sar) → G o2s (G s2o The L1 loss of the images before and after (sar))≈sar) is used to guide network training, expressed as:
[0063]
[0064] Among them, λ s2o and λ o2s Represents the weight of the loss of the two cyclic processes.
[0065] The style loss function is usually expressed as:
[0066]
[0067] The final loss is a combined loss function expressed as:
[0068]
[0069] Among them, λ cyc ,λ id Represent the weights of cycle consistency loss and style loss respectively. The present invention trains the network by optimizing this combined loss function. By combining the loss function, the network can achieve more accurate and higher quality image translation in an unsupervised situation.
[0070] 4. Construction of combined discriminator;
[0071] The combined discriminator contains two discriminator modules. By simultaneously discriminating the authenticity and scene of the generated image, it guides the generator to learn the scene features of the image in a more detailed manner and generate real images that are more consistent with the scene.
[0072] The first discriminator constructed is Patch-GAN. The original image is converted into a discriminant matrix of size 30×30 through convolution operation. Each pixel in the matrix has a receptive field of size 70×70 of the original image, which is used to jointly discriminate the authenticity of the original image in blocks. The second discriminator is the scene discriminator, which consists of seven 7×7 convolutional layers and a fully connected layer. Its specific structure is shown in Table 2, and scene_num is the number of scene categories. The discriminator extracts features through convolution operation, and finally obtains the final scene probability discrimination result through a layer of softmax. Under this architecture, the model can judge the quality of the image from two directions: image authenticity and image scene.
[0073] Table 2 Scene discriminator network structure
[0074]
[0075]
[0076] The present invention uses scene classification loss to discriminate the scene information of the image, and guides network training by calculating the cross entropy loss function, so that the network translation result is more consistent with the original scene. The scene classification loss function is expressed as:
[0077]
[0078] Where D SAR_cls , D OPT_cls represents the scene discriminator of SAR image and visible light image, sce represents the real scene information of the image, and D SAR_cls (sce|sar) and D OPT_cls (sce|opt) represents the discrimination results of the visible light and SAR scene discriminators on the input image.
[0079] The final combined loss function is expressed as:
[0080]
[0081] Among them, λ cyc ,λ id ,λ cls Represent the weights of cycle consistency loss, style loss and scene classification loss respectively. In the experiments in this paper, λ cls It is set to 0.01 and the network is trained by optimizing the combined loss function.
Claims
1. A weakly supervised SAR to visible light image translation method integrating scene information, characterized in that: The steps include: Step 1: Noise suppression module construction; The input SAR image is transformed and decomposed by wavelet to obtain four sub-band images. Each sub-band contains information of different frequencies of the original image. The sub-band images are then input into the prediction network to predict the filtering threshold of each sub-band and perform threshold filtering. Finally, the sub-band images are inversely transformed to obtain the denoised SAR image. Step 2: Construction of scene information fusion generator; The scene information fusion generator is constructed using the network structure based on Cycle-GAN; Table 1 shows the structure of the scene information fusion generator; Table 1. Scene information fusion generator network structure The input image is initially extracted through the downsampling module to reduce the size of the feature map while increasing the dimension; the extracted features are input into the scene guided feature extraction module, which contains the scene filtering residual block and the scene fusion attention block; The scene filter residual block consists of two 3×3 convolutional layers and a fully connected layer, which fuses the scene features with the image features in the form of global features; The image features are extracted through the convolution layer and then fused with the scene features processed by the fully connected layer through channel addition. The fused features are then passed through the convolution layer to obtain the output features of the residual part, which is the sum of the original features and the residual features. The scene fusion attention block contains two multi-head attention layers: self-attention and cross-attention. Through self-attention and cross-attention, scene features are fused with image features in a way that combines local features and global features. Image features first pass through the first self-attention layer to obtain feature output, and then calculate cross-attention with the input scene features to achieve feature fusion. The fused features pass through the feedforward layer to obtain the final output. The output image features are mapped to the corresponding image domain through the upsampling layer to generate a more realistic image; the GAN network is trained using the adversarial loss function and applied to image mapping; the mapping function G from visible light to SAR image o2s :OPT→SAR and its discriminator D SAR , the target is expressed as: Where OPT represents visible light image, SAR represents SAR image, G o2s represents the generator for generating SAR images from visible light, D SAR represents the discriminator of SAR image, represents the operation of extracting a SAR vector sar from the prior distribution pdata(sar) in the latent space, represents the operation of extracting an OPT vector opt from the prior distribution pdata(opt) in the latent space; G o2s Translate visible light images into SAR images, D SAR Used to distinguish the translation result from the real image; for the mapping function G from SAR to visible light s2o :SAR→OPT introduces a similar adversarial loss, namely Step 3: Combined discriminator construction; The combined discriminator contains two discriminator modules; The first discriminator module is Patch-GAN. The original image is converted into a discriminant matrix of size 30×30 through convolution operation. Each pixel in the matrix has a receptive field of size 70×70 of the original image, which is used to jointly discriminate the authenticity of the original image in blocks. The second discriminator module is the scene discriminator, which consists of seven 7×7 convolutional layers and one fully connected layer. The discriminator extracts features through convolution operation and finally obtains the final scene probability discrimination result through a layer of softmax.
2. According to the weakly supervised SAR to visible light image translation method integrating scene information according to claim 1, it is characterized in that: In step 2, two loss functions are introduced: cycle consistency loss and style loss; The cycle consistency loss function is calculated by minimizing the s2o (sar) → G o2s (G s2o The L1 loss of the images before and after (sar))≈sar) is used to guide network training, expressed as: Among them, λ s2o and λ o2s Represents the weight of the loss of the two cyclic processes; The style loss function is expressed as: The final loss is a combined loss function expressed as: Among them, λ cyc ,λ id Represent the weights of cycle consistency loss and style loss respectively, and the network is trained by optimizing this combined loss function.
3. The weakly supervised SAR to visible light image translation method integrating scene information according to claim 2, characterized in that: In step 3, scene classification loss is used to discriminate the scene information of the image, and the network training is guided by calculating the cross entropy loss function so that the network translation result is more consistent with the original scene; the scene classification loss function is expressed as: Where D SAR_cls , D OPT_cls represents the scene discriminator of SAR image and visible light image, sce represents the real scene information of the image, and D SAR_cls (sce|sar) and D OPT_cls (sce|opt) represents the discrimination results of the visible light and SAR scene discriminators on the input image respectively; The final combined loss function is expressed as: Among them, λ cls Represents the weight of the scene classification loss.
4. The weakly supervised SAR to visible light image translation method integrating scene information according to claim 3, characterized in that: The lambda cls Set to 0.01.
Citation Information
Patent Citations
SAR-optical image translation method and system based on image evaluation and feature selection
CN115859606A
Method and device for generating optical image
CN116342902A
Systems and methods for enhancing synthetic aperture radar imagery
US20180172824A1
Cited By
Magnetic composite material, preparation method and magnetic device prepared from magnetic composite material
CN121191872A
A magnetic composite material, its preparation method, and magnetic devices prepared therefrom.
CN121191872B
