Method for converting SAR image into optical image based on diffusion model
Through a diffusion model-based method, combined with a denoising diffusion model and a noise prediction network, the problem of information loss and mode collapse in the conversion of SAR images to optical images is solved, high-quality, detailed optical image generation is achieved, and the model's accuracy is improved. Stability and effectiveness.
Patent Information
- Application Number
- CN202411206102.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-30
- Publication Date
- 2025-01-21
AI Technical Summary
The prior art has problems such as information loss, difficulty in obtaining data sets, and pattern collapse in the conversion of SAR images to optical images, resulting in unstable and poor effectiveness of the generative model in practical applications.
Using a diffusion model-based method, a stable SAR image to optical image conversion model is constructed through the denoising diffusion model and noise prediction network, combined with adversarial loss and perceptual loss.
It achieves the generation of high-quality, detailed optical images, avoids mode collapse, improves the stability and adversarial nature of the model, and enhances the effectiveness of image conversion.
Smart Images
Figure CN119338709A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the technical field of conversion from synthetic aperture radar image to optical image, and in particular to a method for converting SAR image to optical image based on a diffusion model. Background Art
[0002] Synthetic Aperture Radar (SAR) is a high-resolution imaging radar. Due to the penetrating nature of electromagnetic waves, SAR is not affected by clouds or light. Therefore, compared with optical remote sensing, SAR has the advantage of working around the clock and in all weather conditions, and is widely used in military and civilian fields.
[0003] However, interpreting SAR images requires specific expertise, professional training, and is highly dependent on the experience of the interpreter. This is because SAR images are very different from optical images and are usually inconsistent with the visual perception of the human eye. SAR images are complex. Due to the principles and mechanisms of SAR imaging, SAR images usually have unique geometric features and speckle noise, which greatly increases the difficulty of interpretation. Therefore, as one of the feasible methods for interpreting SAR images, converting SAR into optical images to assist in interpreting SAR images is a research hotspot.
[0004] In order to make SAR images consistent with human visual perception habits and help inexperienced people interpret SAR images, a generative model is generally required to achieve translation from SAR images to optical images. Generative adversarial networks (GANs) have become the main model for converting SAR images to optical images. GAN draws on the theory of two-person zero-sum games in game theory to generate high-quality images by adversarially training the generator and the discriminator. In the application of SAR to optical image conversion, its most notable feature is the use of SAR-optical image paired datasets to train the model. However, GAN has many problems in practical applications. First, the feature extraction module in the generator often encounters information loss, and high-quality feature maps are difficult to generate. Second, it is extremely difficult to obtain datasets of SAR-optical paired images, which greatly hinders the effectiveness of model training. In addition, the GAN network architecture will have the problem of mode collapse, especially for high-resolution images, which is more likely to cause model training failure. Until the proposal of the diffusion model, it was established as an alternative to GANs for generating and translating images, providing more stable training.
[0005] However, with the increasing demand and quality requirements for SAR image interpretation, how to use generative models to convert SAR images to optical images more effectively and stably while ensuring the texture details and edge clarity of the generated images is a problem that needs to be faced currently. Summary of the invention
[0006] In view of the problems existing in the prior art, the present invention provides a SAR image to optical image conversion method based on a diffusion model to solve the technical problem in the prior art that a generative model can be used more effectively and stably to convert SAR images to optical images.
[0007] The present invention provides a method for converting a SAR image to an optical image based on a diffusion model, comprising:
[0008] S1, a denoising diffusion model is used as a generator, and a discriminator is constructed. The original optical image is input into the denoising diffusion model, and Gaussian noise is gradually added to it to obtain a noisy optical image;
[0009] S2, connecting the noisy optical image and the real SAR image in series along the channel dimension to form a combined image;
[0010] S3. Input the combined graph into the noise prediction network, introduce adversarial loss and perceptual loss based on the noise loss term of the noise prediction network to calculate the total loss function, and output the conversion result.
[0011] Optionally, the step of inputting the original optical image into a denoising diffusion model and gradually adding Gaussian noise to the original optical image to obtain a noisy optical image includes:
[0012] The original optical image x0~q(x) is input into the denoising diffusion model, and the process of gradually adding Gaussian noise is expressed as:
[0013]
[0014] Among them, β t represents the hyperparameter, I represents the identity matrix, N represents Gaussian noise, and α t =1-β t , q(x t |x t-1 ), the above formula can be derived as:
[0015]
[0016] Optionally, inputting the combined graph into a noise prediction network comprises:
[0017] The combined image is input into the noise prediction network, and the generated optical image is output It is expressed as:
[0018]
[0019] Among them, x t represents a noisy optical image, y represents a reference SAR image, ∈ θ (xt ,y,t) represents the noise prediction network.
[0020] Optionally, compute the noisy optical image x t The distance between the noise added in the t-step forward and the noise predicted by the noise prediction network in the t-step is expressed as:
[0021]
[0022] Among them, ∈ θ represents the noise prediction network, ∈ represents the predicted random noise, L simple represents the noise loss term.
[0023] Optionally, the countermeasure loss includes:
[0024] The generated optical image and the original optical image are input into the discriminator together to calculate the generated adversarial loss L adv , and expressed as:
[0025]
[0026] Where D represents the discriminator.
[0027] Optionally, the perceptual loss includes:
[0028] The generated optical image and the original optical image are input into the VGG network together to calculate the perceptual loss L percep , and expressed as:
[0029]
[0030] Among them, ψ m Represents the VGG network.
[0031] Optionally, the calculating the total loss function includes:
[0032] The total loss function is expressed as:
[0033] L=L simple +λ1L percep +λ2L adv
[0034] Among them, λ1 and λ2 are hyperparameters, which represent adjustable parameters for balancing different loss weights.
[0035] Compared with the existing detection technology, the present invention has the following beneficial effects:
[0036] 1. Using a diffusion model, the underlying complexity of data distribution is captured through inverse diffusion to generate high-quality images with fine details and real textures. The inverse diffusion process maps complex data distribution back to a simple distribution, allowing the latent space to represent meaningful features, patterns, and latent variables in the data, with a more interpretable latent space.
[0037] 2. At the same time, by using likelihood-based training, the training diffusion model is usually more stable and avoids mode collapse; at the same time, due to the use of likelihood-based training and the inherent properties of inverse diffusion, the diffusion model is more robust to overfitting, which encourages more coherent and diverse sample generation. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.
[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0040] Figure 1 It is a schematic diagram of the method flow of the present invention;
[0041] Figure 2 It is a schematic diagram of the structural conversion of the present invention;
[0042] Figure 3 A schematic diagram of the present invention's countermeasure loss;
[0043] Figure 4 Schematic diagram of the perception loss of the present invention. DETAILED DESCRIPTION
[0044] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other implementation cases obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application. The functional units with the same labels in the examples of the present invention have the same and similar structures and functions.
[0045] See also Figure 1 The present invention provides a method for converting a SAR image to an optical image based on a diffusion model, comprising:
[0046] S1, a denoising diffusion model is used as a generator, and a discriminator is constructed. The original optical image is input into the denoising diffusion model, and Gaussian noise is gradually added to it to obtain a noisy optical image;
[0047] S2, connecting the noisy optical image and the real SAR image in series along the channel dimension to form a combined image;
[0048] S3. Input the combined graph into the noise prediction network, introduce adversarial loss and perceptual loss based on the noise loss term of the noise prediction network to calculate the total loss function, and output the conversion result.
[0049] In this embodiment, S1, a denoising diffusion model is used as a generator, and a discriminator is constructed, the original optical image is input into the denoising diffusion model, and Gaussian noise is gradually added to it to obtain a noisy optical image.
[0050] See also Figure 2 A denoising diffusion model (DDPM) is used as a generator to generate an optical image corresponding to the SAR image. The denoising diffusion model includes a forward denoising process and a reverse denoising process. A discriminator is constructed, which uses the patchGAN architecture to judge the similarity between the generated optical image and the original optical image.
[0051] The original optical image is input into the denoising diffusion model, and Gaussian noise is gradually added to it to obtain a noisy optical image, which is expressed as:
[0052] Gaussian noise is gradually added to the original optical image x0, where x0~q(x) represents the observed data x0 sampled according to a certain distribution q.
[0053] The Gaussian noise standard deviation is given by the hyperparameter β t The mean is determined by β t Together with the noisy optical image at the current time t, β t Take the variable table Then we can get the complete noisy optical image noise samples x1, x2, …, x T The mathematical expression of the forward diffusion process is as follows:
[0054]
[0055] Where N represents Gaussian noise and I is the identity matrix. According to the reparameterization technique, q(x t ) can be based on x0 and β t Calculate without iteration. Let α t =1-β t , q(x t |xt-1 ) can be deduced as follows:
[0056]
[0057] S2. Connecting the noisy optical image and the real SAR image in series along the channel dimension to form a combined image.
[0058] The obtained noisy optical image is connected with the reference SAR image in the dataset of the noise prediction network along the channel dimension. The combined image is passed through the noise prediction network of the reverse diffusion process. The dataset adopts the SAR2OPT public dataset, which contains 2077 images paired with SAR images and optical images.
[0059] S3. Input the combined graph into the noise prediction network, introduce adversarial loss and perceptual loss based on the noise loss term of the noise prediction network to calculate the total loss function, and output the conversion result.
[0060] The noise prediction network adopts the U-Net structure. U-Net is a fully convolutional neural network consisting of an Encoder-Decoder module and a Time Embedding module. The Encoder module is responsible for feature extraction and learning. The Decoder module is responsible for restoring the feature map to its original resolution. Unet uses skip-layer connections to fuse shallow position information with deep semantic information by splicing the number of channels.
[0061] The Time Embedding module encodes the t of the tth step and encodes it into the network to help U-Net predict noise in each iteration.
[0062] In the denoising diffusion process, the noise prediction network ∈ θ (x t ,y,t), the input is a noisy optical image x t And the given reference SAR image y. In the case of the original optical image x0 unknown, given x t , model∈ θ (x t ,y,t) input x t With the reference data y, we can predict ∈ θ , thus obtaining the denoised observation value, that is, the generated optical image The formula is as follows:
[0063]
[0064] The present invention not only adopts the general noise loss term of the diffusion model, but also adopts a constraint method that is more in line with the visual habits of the human eye. By introducing perceptual loss and adversarial loss, the perceptual difference between the output and the standard image can be captured, which is more in line with the visual habits of the human eye and restores a better optical image.
[0065] First, considering the fitted mean μ θ (x t ,y,t) can be obtained by fitting the noise part. Noise prediction network ∈ θ Output a random variable ∈, representing the noise of the prediction.
[0066] Since new samples with added noise are generated at step t So using ∈ θ (x t ,t,y) to estimate the noise sample x t The noise component of , the loss function is as follows:
[0067]
[0068] This equation is equivalent to a noise prediction task. The noise prediction network is trained to minimize the noise prediction
[0069] See also Figure 3 Secondly, the adversarial loss is introduced into this model. First, the observation value of the generated optical image is obtained by the prediction network. Then the generated optical image and the original optical image x0 are respectively input into the discriminator D for evaluation.
[0070] The parameters of the model and the discriminator are optimized through the idea of zero-sum game, so that the two networks can achieve joint optimization in the confrontation, thus obtaining better prediction results. adv The formula is as follows:
[0071]
[0072] See also Figure 4 Then, for the SAR to optical image conversion problem, we can compare By defining and optimizing ψ based on the pre-trained VGG network m The perceptual loss function of the high-level features extracted from the VGG16 model can generate high-quality images. The weights of the VGG16 model are trained by ImageNet.
[0073] First, the generated optical image and the original optical image are input into the pre-trained VGG network to extract x0 and The feature representation of the intermediate layer captures its semantic information and style information; the difference between the generated optical image and the original optical image in the selected intermediate layer feature representation is calculated using the Euclidean distance measurement method; finally, the feature differences of multiple selected intermediate layers are weighted and summarized to obtain the final perceptual loss L percep The perceptual loss part formula is as follows:
[0074]
[0075] Finally, the overall loss function formula is as follows:
[0076] L=L simple +λ1L percep +λ2L adv
[0077] Among them, λ1 and λ2 are hyperparameters, which represent adjustable parameters for balancing different loss weights.
[0078] The SAR to optical image conversion method based on the diffusion model provided by the present invention gradually adds Gaussian noise to the original optical image to obtain a complete noisy optical image noise sample; the noisy optical image and the SAR image are connected in series along the channel dimension and input into a noise prediction network to obtain a predicted value of the added noise; the network is optimized through iterative noise prediction; and the final conversion result is gradually sampled according to the output of the noise prediction network.
[0079] Compared with the traditional generative model-based SAR to optical image conversion method, this scheme adopts a diffusion model to capture the underlying complexity of the data distribution through inverse diffusion, generating high-quality images with fine details and realistic textures. The inverse diffusion process maps the complex data distribution back to a simple distribution, allowing the latent space to represent the meaningful features, patterns and latent variables present in the data, with a more interpretable latent space. At the same time, by using likelihood-based training, training diffusion models is usually more stable and avoids mode collapse. At the same time, due to the use of likelihood-based training and the inherent properties of inverse diffusion, the diffusion model is more robust to overfitting, which encourages more coherent and diverse sample generation.
[0080] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0081] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.
Claims
1. A method for converting SAR images to optical images based on a diffusion model, characterized in that: include: S1, a denoising diffusion model is used as a generator, and a discriminator is constructed. The original optical image is input into the denoising diffusion model, and Gaussian noise is gradually added to it to obtain a noisy optical image; S2, connecting the noisy optical image and the real SAR image in series along the channel dimension to form a combined image; S3. Input the combined graph into the noise prediction network, introduce adversarial loss and perceptual loss based on the noise loss term of the noise prediction network to calculate the total loss function, and output the conversion result.
2. The method for converting SAR images to optical images based on a diffusion model as claimed in claim 1, characterized in that: The original optical image is input into the denoising diffusion model, and Gaussian noise is gradually added to it to obtain a noisy optical image, including: The original optical image x0~q(x) is input into the denoising diffusion model, and the process of gradually adding Gaussian noise is expressed as: Among them, β t represents the hyperparameter, I represents the identity matrix, N represents Gaussian noise, and α t =1-β t , q(x t |x t-1 ), the above formula can be expressed as:
3. The method for converting SAR images to optical images based on a diffusion model as claimed in claim 2, characterized in that: The step of inputting the combined graph into the noise prediction network comprises: The combined image is fed into a noise prediction network, which outputs the generated optical image. It is expressed as: Among them, x t represents a noisy optical image, y represents a reference SAR image, ∈ θ (x t ,y,t) represents the noise prediction network.
4. The method for converting SAR images to optical images based on a diffusion model as claimed in claim 3, characterized in that: The noise loss term of the noise prediction network also includes: Calculate the noisy optical image x t The distance between the noise added in the t-step forward and the noise predicted by the noise prediction network in the t-step is expressed as: Among them, ∈ θ represents the noise prediction network, ∈ represents the predicted random noise, L simple represents the noise loss term.
5. The method for converting SAR images to optical images based on a diffusion model as claimed in claim 4, characterized in that: The adversarial losses include: The generated optical image and the original optical image are input into the discriminator together to calculate the generated adversarial loss L adv , and expressed as: Where D represents the discriminator.
6. The method for converting SAR images to optical images based on a diffusion model as claimed in claim 5, characterized in that: The perceptual losses include: The generated optical image and the original optical image are input into the pre-trained VGG network together to calculate the perceptual loss L percep , and expressed as: Among them, ψ m Represents the VGG network.
7. The method for converting SAR images to optical images based on a diffusion model as claimed in claim 6, characterized in that: The calculation of the total loss function includes: The total loss function is expressed as: L=L simple +λ1L percep +λ2L adv Among them, λ1 and λ2 are hyperparameters, which represent adjustable parameters for balancing different loss weights.