Method for translating SAR (Synthetic Aperture Radar) to optics based on time sequence transformer and GAN (Generalized Area Network)
Through the method based on timing transformer and GAN, the problem of insufficient utilization of time series information from SAR to optical image translation is solved, and the time series construction of high-precision optical remote sensing image is realized, which improves the optical image acquisition efficiency.
Patent Information
- Application Number
- CN202510576671.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-06
- Publication Date
- 2025-09-05
AI Technical Summary
The existing SAR to optical image translation methods have shortcomings in accuracy and time series information utilization, making it difficult to construct continuous, dense and cloudless optical remote sensing image time series.
Using a time-series transformer and GAN method, the transformer submodule and the image translation module are introduced through the feature extraction module, and the Unet submodule is introduced. Combined with the least squares generation adversarial network LSGAN loss function and reconstruction loss, a generator and discriminator are built, and the image translation module and discriminator are used to improve the image translation accuracy.
The translation accuracy of SAR images to optical images is improved, and a continuous, dense and cloudless optical remote sensing image time series is constructed, which improves the fusion effect of optical image acquisition efficiency and time information.
Smart Images
Figure CN120599460A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of remote sensing image data processing, and specifically relates to a SAR to optical translation method based on a temporal transformer and a GAN. Background Art
[0002] Continuous, dense, and cloud-free time series of remote sensing images have important application value in fields such as forest resource management, ecological and environmental monitoring, land use surveys, and agricultural monitoring. However, continuous optical remote sensing images are not easy to obtain due to various reasons. SAR images have strong penetration capabilities and high spatial resolution, and can penetrate cloud cover to achieve ground observation. Therefore, SAR images can assist in complementing optical images and constructing continuous, dense, and cloud-free time series of remote sensing images. Existing SAR to optical image translation methods are mostly based on generative adversarial networks (GANs), but there are still problems in image accuracy and other aspects, and they also fail to make good use of the information of image time series.
[0003] Transformer-based temporal information processing has garnered widespread attention. Transformers can better leverage time series features to reveal relationships between images over time. They can be used to extract time series features from images. This allows for better integration of temporal information into image translation, resulting in more accurate translations. Summary of the Invention
[0004] The technical problem of the present invention is: the present invention proposes a SAR to optical translation method based on a time series transformer and GAN, extracts effective time variation features from the SAR image time series, and constructs a model to convert the SAR time series features into fine optical time series features, and then uses the converted optical time series features to synthesize a continuous, dense and cloud-free optical remote sensing image time series.
[0005] The technical solution of the present invention is a SAR to optical translation method based on a temporal transformer and a GAN, comprising the following steps: The transformer submodule is introduced into the feature extraction module, and the Unet submodule is introduced into the image translation module; S1, collect SAR and optical remote sensing images and construct a remote sensing image time series set; S2, preprocess the time series set to form a data set; S3. Use the GAN network as the basic framework of the model to build the generator and discriminator respectively; S4. Select the loss function of the least squares generative adversarial network LSGAN plus the reconstruction loss as the loss function of the GAN network; S5. Input the training set of the data set into the GAN network for training, and input the test set into the GAN network, and then construct a new time series image from the obtained optical images.
[0006] Furthermore, in step S2, the time series set is preprocessed, including normalizing the input image and cropping it to an input size suitable for the model.
[0007] Preferably, in step S3, the generator includes a channel splicing module, a feature extraction module and an image translation module; the discriminator includes an MLP module and a ResBD module; the feature extraction module includes a Patch Embedding convolution layer, a position encoding, and multiple transformer basic modules.
[0008] Furthermore, the transformer basic module includes multi-head self-attention layer, MLP layer, residual connection, layer normalization, and fully connected layer.
[0009] Preferably, the image translation module comprises a Unet network, including a ResNetBlock module, multiple ResNetBlock modules, a skip connection module, a ResNetBlock module, a single convolutional layer and an activation function Tanh.
[0010] Furthermore, step S3 includes the following sub-steps: 1) Input the image and stitch it together through the channel stitching module; 2) The spliced image is divided into blocks through Patch Embed and then position encoded to obtain a set of feature blocks with spatial position information; 3) The feature block set is input into the multi-layer Transformer module to obtain the globally modeled features. After residual connection and layer normalization, a global feature vector is obtained, and then the corresponding shape features are obtained through the fully connected layer. 4) The image with the corresponding shape features is passed through the ResNetBlock module of the discriminator to enhance important features, suppress irrelevant features, reduce spatial dimensions and reduce computational complexity; 5) Through the central ResNetBlock module, features are extracted and spatial dimensions are fixed; 6) The features are input into the ResNetBlock module for deconvolution, and the feature space and details are restored through upsampling. All features are concatenated through a single convolutional layer, and the result is restored to an image using the Tanh activation function.
[0011] Preferably, in step S4, the loss function of the least squares generative adversarial network LSGAN plus the reconstruction loss is used as the loss function of the GAN network, and a loss function is constructed based on the least squares generative adversarial network to generate high-quality images and show greater stability during training, including discriminator loss and generator loss.
[0012] Furthermore, the discriminator loss and generator loss are calculated as: ; ; Where x represents the input image, y represents the target image, D() represents the discriminator, and G() represents the generator.
[0013] Loss function of GAN network Also includes VGG loss , spectral angle loss and visual loss , the loss function of the GAN network The calculation formula is: ; Where, represents the VGG loss, represents the spectral angle loss, Indicates visual loss, Represent the weights of the three loss functions respectively.
[0014] 1) VGG loss function is based on the feature matching loss function of the pre-trained VGG19 network, which is used to capture the details and semantic differences of the image and make the content of the generated image close to the original image. The calculation formula is: ; Where, Represent the generated image and target image respectively, i is the order of the 2nd, 7th, 12th, 21st and 30th layers in the VGG19 network, represents the feature map of the i-th layer in VGG19, C i represents the number of channels in the i-th layer, (H i ,W i ) represents the feature map size of each channel in the i-th layer, H and W represent the height and width of the image respectively; 2) Spectral angle loss, used to reduce spectral distortion in the image, is calculated as: ; Where, Represents a tensor with a value of 1; 3) Visual loss, using multi-scale structural similarity to evaluate the texture information of the generated image, is calculated as: ; ; ; ; Represents the comparison result of the image contrast calculated on the j-th scale, Indicates the result of comparing the image results calculated at the j-th scale; Indicates the brightness comparison result of two images calculated at scale M, where M represents the last scale; 、 represents the corresponding weighting coefficient; Respectively represent the corresponding images ; Represents images respectively variance; Representing an image The covariance of C 1、 C2 and C3 represent constants.
[0015] Preferably, the calculation formula of the final generator's overall loss factor is: ; Where, represents the input image, represents the weighting coefficient, Indicates the minimum, represents the reconstruction loss.
[0016] Preferably, the SAR to optical translation system based on temporal transformer and GAN includes: The preprocessing module batch processes the input image group to make it meet the input conditions; GAN network, the overall model structure consists of a GAN network, and the model is trained in the form of a GAN network; The generator, consisting of a feature extraction module and an image translation module, is used to generate optical images at corresponding time points; The feature extraction module uses transformer to extract the features of image time series, improving the utilization of time series information; The image translation module uses the extracted features through the UNet network and the CBAM attention mechanism to enhance the utilization of the spatial and detail information of the features, thereby improving the accuracy of SAR to optical image translation; The discriminator is used to judge the images generated by the generator. It is mainly composed of two sub-networks, which judge the relationship between the real data and the generated data from the one-dimensional single point time judgment and the two-dimensional image respectively. Finally, the weighted combination is used to judge the difference between the generated image and the target image.
[0017] Compared with the prior art, the present invention has the following beneficial effects: 1) This paper provides a SAR to optical translation method based on a temporal transformer and GAN. By constructing a continuous, dense, and cloud-free time series of remote sensing images, it can more efficiently acquire optical images and integrate temporal information into image translation, thereby improving the accuracy of image translation.
[0018] 2) This paper provides a SAR to optical translation method based on a temporal transformer and GAN, which incorporates the easy acquisition characteristics of SAR images into the construction of optical remote sensing image time series, thereby improving the efficiency of constructing sufficiently long optical remote sensing image time series. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] The present invention will be further described below with reference to the accompanying drawings and examples.
[0020] Figure 1 This is a flow chart of the SAR to optical translation method based on temporal transformer and GAN of the present invention; Figure 2 This is a schematic diagram of the model structure of the SAR to optical translation method based on temporal transformer and GAN in the present invention; Figure 3 Schematic diagram of the feature extraction module structure of the generator in the GAN network of the present invention; Figure 4 This is a schematic diagram of the image translation module structure of the generator in the GAN network of the present invention; Figure 5 Schematic diagram of the ResNetBlock module of the generator in the GAN network of the present invention; Figure 6 Schematic diagram comparing the image translation of the present invention and different methods. DETAILED DESCRIPTION
[0021] like Figure 1 As shown in the figure, the SAR to optical translation method based on temporal transformer and GAN includes the following steps: The transformer submodule is introduced into the feature extraction module, and the Unet submodule is introduced into the image translation module.
[0022] S1. Collect SAR and optical remote sensing images and construct a remote sensing image time series set.
[0023] Each image in the two SAR and optical image time series groups was preprocessed and cut into several square blocks with a side length of 256 pixels.
[0024] S2. Preprocess the time series set to form a data set.
[0025] In step S2, the time series set is preprocessed, including normalizing the input image and cropping it to an input size suitable for the model.
[0026] S3. Use the GAN network as the basic framework of the model to build the generator and discriminator respectively.
[0027] like Figure 2 As shown in step S3, the generator includes a channel splicing module, a feature extraction module and an image translation module; the discriminator includes a target image module, an extracted pixel timing module, an MLP module, a ResBD module and a Real / Fake module; the feature extraction module includes a Patch Embedding convolution layer, a position encoding, three transformer basic modules, and two 3×3 convolution layers and a pooling layer.
[0028] like Figure 3 As shown in the figure, the transformer basic module includes multi-head self-attention layer, MLP layer, residual connection, layer normalization and fully connected layer.
[0029] like Figure 4 As shown in Figure 1, the image translation module contains a Unet network, including a ResNetBlock module, 3 ResNetBlock modules, a skip connection module, a ResNetBlock module, a single convolutional layer, and an activation function Tanh.
[0030] like Figure 5 As shown, step S3 includes the following sub-steps: 1) Input the image and stitch it together through the channel stitching module; 2) The spliced image is divided into blocks through Patch Embed and then position encoded to obtain a set of feature blocks with spatial position information; 3) The feature block set is input into the multi-layer Transformer module to obtain the globally modeled features. After residual connection and layer normalization, a global feature vector is obtained, and then the corresponding shape features are obtained through the fully connected layer. 4) The image with the corresponding shape features is passed through the three ResNetBlock modules of the discriminator to enhance important features, suppress irrelevant features, reduce spatial dimensions and reduce computational complexity; 5) Through the central ResNetBlock module, features are extracted and spatial dimensions are fixed; 6) The features are input into three ResNetBlock modules for deconvolution. The feature space and details are restored through upsampling. All features are concatenated through a single convolutional layer, and the result is restored to an image using the Tanh activation function.
[0031] S4. Select the loss function of the least squares generative adversarial network LSGAN plus the reconstruction loss as the loss function of the GAN network.
[0032] In step S4, the loss function of the least squares generative adversarial network LSGAN plus the reconstruction loss is used as the loss function of the GAN network. A loss function is constructed based on the least squares generative adversarial network to produce high-quality images and show greater stability during training, including discriminator loss and generator loss.
[0033] The calculation formula of discriminator loss and generator loss is: ; ; Where x represents the input image, y represents the target image, D() represents the discriminator, and G() represents the generator.
[0034] The loss function of the GAN network also includes VGG loss, spectral angle loss and visual loss: 1) VGG loss function is based on the feature matching loss function of the pre-trained VGG19 network, which is used to capture the details and semantic differences of the image and make the content of the generated image close to the original image. The calculation formula is: ; Where, Represent the generated image and target image respectively, i is the order of the 2nd, 7th, 12th, 21st and 30th layers in the VGG19 network, represents the feature map of the i-th layer in VGG19, C i represents the number of channels in the i-th layer, (H i ,W i ) represents the feature map size of each channel in the i-th layer, and H and W represent the height and width of the image respectively.
[0035] 2) Spectral angle loss, used to reduce spectral distortion in the image, is calculated as: ; Where, Represents a tensor with a value of 1.
[0036] 3) Visual loss, using multi-scale structural similarity to evaluate the texture information of the generated image, is calculated as: ; ; ; ; Represents the comparison result of the image contrast calculated on the j-th scale, Indicates the result of comparing the image results calculated at the j-th scale; Indicates the brightness comparison result of two images calculated at scale M, where M represents the last scale; 、 represents the corresponding weighting coefficient; Respectively represent the corresponding images ; Represents images respectively variance; Representing an image covariance of ; C1, C2, and C3 represent constants.
[0037] S5. Input the training set of the data set into the GAN network for training, and input the test set into the GAN network, and then construct a new time series image from the obtained optical images.
[0038] SAR to optical translation system based on temporal transformer and GAN, including: The preprocessing module batch processes the input image group to make it meet the input conditions; GAN network, the overall model structure consists of a GAN network, and the model is trained in the form of a GAN network; The generator, consisting of a feature extraction module and an image translation module, is used to generate optical images at corresponding time points; The feature extraction module uses transformer to extract the features of image time series, improving the utilization of time series information; The image translation module uses the extracted features through the UNet network and the CBAM attention mechanism to enhance the utilization of the spatial and detail information of the features, thereby improving the accuracy of SAR to optical image translation; The discriminator is used to judge the images generated by the generator. It is mainly composed of two sub-networks, which judge the relationship between the real data and the generated data from the one-dimensional single point time judgment and the two-dimensional image respectively. Finally, the weighted combination is used to judge the difference between the generated image and the target image.
[0039] The feature extraction module includes: Block embedding: The input image or sequence is cut into blocks according to the specified patch submodule size and projected onto a fixed feature dimension for processing by the Transformer submodule.
[0040] Position encoding: The features of each patch are added to the embedding of the corresponding position to provide spatial position information for the Transformer submodule.
[0041] Transformer module stack: It is composed of several stacked Transformer sub-modules. Each Transformer sub-module implements the following steps: normalize the input, extract features, and capture the contextual information of the time series.
[0042] Dimensionality reduction projection: The obtained global features are mapped again to reduce the feature dimensions to a smaller size for subsequent fusion or decoding.
[0043] The image translation module includes: Encoding stage: Through multi-level downsampling, the resolution is gradually reduced and the number of channels is increased to extract multi-scale features. Before and after each level of downsampling, ResNetBlock is used for deeper local feature learning. ResNetBlock includes two convolutions and normalization, activation, and a residual connection branch to improve training stability.
[0044] Center layer: A feature transformation with a higher number of channels is performed at the bottom to capture deeper semantic information.
[0045] Decoding stage: Through step-by-step upsampling, the spatial resolution of the input image is restored. After each upsampling, the encoder output features of the same resolution are jump-connected and spliced, and then fused through ResNetBlock to make the output image more refined.
[0046] Output layer: Finally, the number of channels is compressed to the number of channels of the target image, and an activation function is used to ensure that the output value is in the appropriate range to obtain the final generated image result.
[0047] As shown in Table 1, in terms of five commonly used image evaluation indices, the results of comparing the images generated by the present invention and different methods on the same data set with the original optical images show that the images generated by the present invention are more similar to the original images than the other images.
[0048] As shown in Table 1, the results of the comparison between the images generated by the present invention and different methods on the same data set and the original optical image are obtained in terms of four commonly used image evaluation indices. The four evaluation indices represent: Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), Correlation Coefficient (CC), and Root Mean Square Error (RMSE). PSNR is mainly used to measure the pixel difference between images. The larger the PSNR, the more consistent the pixel values of the two images are. SSIM measures the similarity between two images by comparing the brightness, contrast, and structural information of the images. The output of SSIM is generally between 0 and 1. As the value increases, the similarity between the images is higher. CC mainly compares the linear correlation between each pixel in the two images. Its theoretical value range is -1 to 1, that is, from negative correlation to positive correlation. RMSE is an evaluation index obtained by taking the square root of MSE. Therefore, it directly reflects the difference between the pixels of the two images. The smaller the PSNR, the more consistent the pixel values of the two images are.
[0049] As can be seen from the table, the images generated by the present invention are more similar to the original images than those generated by other models. Not only the PSNR of 27.6321, RMSE of 0.0416, and CC of 0.9368, which reflect pixel consistency, but also the SSIM of 0.8339, which reflects the similarity of the overall structure of the image, have been improved by a greater margin. Compared with the latest model MTS2ONet, the present invention improves the PSNR, SSIM, CC, and RMSE by 0.0576, 0.006, 0.0111, and 0.0004, respectively, in terms of evaluation indexes. Compared with other previous methods, the improvement is more obvious.
[0050] like Figure 6 As shown, the present invention and the prior art generate image comparison graphs in scenes that are mostly mountainous and green forests. For ease of observation, the figure provides partial enlarged images of the first and second rows in the second and fourth rows. CycleGAN and ParallelGAN are affected by other factors in the mountainous and green forest environment, which causes the colors of the generated images to be different from the original images in most areas. ParallelGAN even has noise problems. Except for MTS2ONet, other comparison methods have problems such as unclear regional edge contours and loss of geographical details. MTS2ONet and the present invention show a high degree of consistency in geographical structure. In particular, the present invention has clearer regional edges, more complete internal color restoration, and is more consistent with the original image.
[0051] Table 1
[0052] Finally, it should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or system that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or system. In the absence of further limitations, an element defined by the phrase "comprises a..." does not preclude the presence of other identical elements in the process, method, article, or system that includes the element.
[0053] The serial numbers of the embodiments of the present invention are for descriptive purposes only and do not represent superiority or inferiority of the embodiments. In a unit claim that lists several means, several of these means may be embodied by the same item of hardware. The use of the terms first, second, and third, etc., does not denote any order and should be construed as identifiers.
[0054] The above are only preferred embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.
Claims
1. SAR to optical translation method based on temporal transformer and GAN, characterized by: The following steps are involved: The transformer submodule is introduced into the feature extraction module, and the Unet submodule is introduced into the image translation module; S1, collect SAR and optical remote sensing images and construct a remote sensing image time series set; S2, preprocess the time series set to form a data set; S3. Use the GAN network as the basic framework of the model to build the generator and discriminator respectively; S4. Select the loss function of the least squares generative adversarial network LSGAN plus the reconstruction loss as the loss function of the GAN network; S5. Input the training set of the data set into the GAN network for training, and input the test set into the GAN network, and then construct a new time series image from the obtained optical images.
2. The SAR to optical translation method based on temporal transformer and GAN according to claim 1 is characterized in that In step S2, the time series set is preprocessed, including normalizing the input image and cropping it to an input size suitable for the model.
3. The SAR to optical translation method based on temporal transformer and GAN according to claim 2 is characterized in that In step S3, the generator includes a channel splicing module, a feature extraction module, and an image translation module; the discriminator includes an MLP module and a ResBD module; and the feature extraction module includes a Patch Embedding convolutional layer, a position encoder, and multiple transformer basic modules.
4. The SAR to optical translation method based on temporal transformer and GAN according to claim 3, characterized in that The transformer basic module includes multi-head self-attention layer, MLP layer, residual connection, layer normalization and fully connected layer.
5. The SAR to optical translation method based on temporal transformer and GAN according to claim 4 is characterized in that The image translation module includes a Unet network, including a ResNetBlock module, multiple ResNetBlock modules, a skip connection module, a ResNetBlock module, a single convolutional layer and an activation function Tanh.
6. The SAR to optical translation method based on temporal transformer and GAN according to claim 5, characterized in that The step S3 includes the following sub-steps: 1) Input the image and stitch it together through the channel stitching module; 2) The spliced image is divided into blocks through Patch Embed and then position encoded to obtain a set of feature blocks with spatial position information; 3) The feature block set is input into the multi-layer Transformer module to obtain the globally modeled features. After residual connection and layer normalization, a global feature vector is obtained, and then the corresponding shape features are obtained through the fully connected layer. 4) The image with the corresponding shape features is passed through the ResNetBlock module of the discriminator to enhance important features, suppress irrelevant features, reduce spatial dimensions and reduce computational complexity; 5) Through the central ResNetBlock module, features are extracted and spatial dimensions are fixed; 6) The features are input into the ResNetBlock module for deconvolution, and the feature space and details are restored through upsampling. All features are concatenated through a single convolutional layer, and the result is restored to an image using the Tanh activation function.
7. The SAR to optical translation method based on temporal transformer and GAN according to claim 6, characterized in that In step S4, the loss function of the least squares generative adversarial network LSGAN plus the reconstruction loss is used as the loss function of the GAN network. A loss function is constructed based on the least squares generative adversarial network to produce high-quality images and show greater stability during training, including discriminator loss and generator loss.
8. The SAR to optical translation method based on temporal transformer and GAN according to claim 7, characterized in that The calculation formula of the discriminator loss and generator loss is: ； ； Where x represents the input image, y represents the target image, D( ) represents the discriminator, and G( ) represents the generator.
9. The SAR to optical translation method based on temporal transformer and GAN according to claim 8, characterized in that The loss function of the GAN network also includes VGG loss, spectral angle loss and visual loss; 1) VGG loss function is based on the feature matching loss function of the pre-trained VGG19 network, which is used to capture the details and semantic differences of the image and make the content of the generated image close to the original image. The calculation formula is: ； Where, Represent the generated image and target image respectively, i represents the order of the 2nd, 7th, 12th, 21st and 30th layers in the VGG19 network, represents the feature map of the i-th layer in VGG19, C i represents the number of channels in the i-th layer, H i ,W i Represents the feature map size of each channel in the i-th layer, H and W represent the height and width of the image respectively; 2) Spectral angle loss, used to reduce spectral distortion in the image, is calculated as: ； Where, Represents a tensor with a value of 1; 3) Visual loss: Multi-scale structural similarity is used to divide the loss into M scales. At each scale, the image contrast and brightness are calculated to evaluate the texture information of the generated image. The calculation formula is: ； ； ； ； Where, Represents the comparison result of the image contrast calculated on the j-th scale, Indicates the result of comparing the image results calculated at the j-th scale; Indicates the brightness comparison result of two images calculated at scale M, where M represents the last scale; 、 represents the corresponding weighting coefficient; Respectively represent the corresponding images ; Represents images respectively variance; Representing an image covariance of ; C1, C2, and C3 represent constants.
10. The SAR to optical translation system based on temporal transformer and GAN according to claims 1-9, characterized in that: include: The preprocessing module batch processes the input image group to make it meet the input conditions; GAN network, the overall model structure consists of a GAN network, and the model is trained in the form of a GAN network; The generator, consisting of a feature extraction module and an image translation module, is used to generate optical images at corresponding time points; The feature extraction module uses transformer to extract the features of image time series, improving the utilization of time series information; The image translation module uses the extracted features through the UNet network and the CBAM attention mechanism to enhance the utilization of the spatial and detail information of the features, thereby improving the accuracy of SAR to optical image translation; The discriminator is used to judge the images generated by the generator. It is mainly composed of two sub-networks, which judge the relationship between the real data and the generated data from the one-dimensional single point time judgment and the two-dimensional image respectively. Finally, the weighted combination is used to judge the difference between the generated image and the target image.