A method and device for converting SAR images into optical images

Through the cross-fusion inference structure and loss function optimization of the CFR-GAN model, the problems of blurred outlines and unclear details in SAR image to optical images are solved, and high-quality image conversion is achieved.

CN115100091BActive Publication Date: 2024-11-26NAT UNIV OF DEFENSE TECH
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202210719467.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-06-23
Publication Date
2024-11-26
Estimated Expiration
2042-06-23

AI Technical Summary

Technical Problem

In the existing SAR image to optical image method, structural information is easily lost during downsampling and upsampling, resulting in blurred outlines of the generated pseudo-optical image, unclear details, and the 9-layer residual block structure cannot complete features on different scales. Convert, resulting in low image quality.

Method used

Using the CFR-GAN model, it includes the generation network and the identification network. Through the cross-fusion inference structure, SAR image features are processed at different resolutions, combined with the least squares generation adversarial loss, perceptual loss and high-dimensional matching loss, optimize the generation network to improve pseudo-reducing Sharpness of optical images.

Benefits of technology

In the CFR-GAN optimization model, the feature information of different scales is retained through the cross-fusion inference structure, which improves the conversion performance of SAR image to optical image, and the clarity of the generated pseudo-optical image contour is improved and the image quality is improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115100091B_ABST
    Figure CN115100091B_ABST
Patent Text Reader

Abstract

The present application relates to a method and device for converting SAR images into optical images, the method comprising: obtaining a SAR image and a corresponding optical image to form a training set; inputting the SAR image and the optical image in the training set into a CFR-GAN model, and outputting a discrimination result, wherein the CFR-GAN model comprises a generation network and a discrimination network, and the generation network comprises an encoder, a cross-fusion inference structure, and a decoder; inputting the optical image and the discrimination result into a pre-constructed total loss function, training the model, and obtaining a CFR-GAN optimization model; inputting the SAR image to be processed into the CFR-GAN optimization model for conversion, and obtaining an optical conversion image. The present method can utilize the cross-fusion inference structure in the CFR-GAN optimization model to realize the conversion of SAR images of different scales to optical images at different resolutions, retain the feature information of each scale of the SAR image, and thus improve the conversion performance of the SAR image to the optical image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of remote sensing image processing, and in particular to a method and device for converting a SAR image into an optical image. Background Art

[0002] Synthetic Aperture Radar (SAR) images can be collected in all-weather conditions and have been widely used in disaster monitoring, resource exploration, agricultural planning, military strikes and other fields. However, due to the characteristics of radar imaging, the interpretation of SAR images is difficult. Therefore, the method of converting SAR images into optical images and then interpreting them has gradually become the mainstream. Realizing the conversion of SAR images to optical images is a branch of the image conversion task.

[0003] Generative adversarial networks (GANs) have made great progress in processing image conversion tasks. At present, the pix2pix model and cycleGAN model based on generative adversarial networks have become the focus of image-to-image conversion. The generative network in the Pix2pix model is a U-Net network structure. The U-Net architecture is a u-shaped structure that encodes the image first and then decodes it, keeping the input and output sizes unchanged. This u-shaped structure was first used for image segmentation. The generative network in the cycleGAN model is a cascade structure of 9 residual blocks. The main component of the cascade structure of 9 residual blocks is Resblock, which improves the performance of the algorithm by continuously deepening the network and solves the optimization problem of the model to a certain extent.

[0004] However, when using the U-Net structure, the structural information in the SAR image is mostly lost during the downsampling and upsampling process, resulting in blurred details of the converted pseudo-optical features. When using a 9-layer residual block structure cascade, since feature inference is only performed at the same resolution, it is impossible to fully complete the conversion from SAR features to optical features at different scales, resulting in low quality of the generated image. Therefore, it is necessary to design a new feature inference structure to solve the problem that the current conversion method converts SAR image features into optical image features, resulting in blurred image contours and unclear details. Summary of the invention

[0005] Based on this, it is necessary to provide a SAR image to optical image conversion method and device that can improve the clarity of the generated pseudo optical image contour in order to solve the above technical problems.

[0006] A method for converting a SAR image into an optical image, the method comprising:

[0007] Acquire SAR images and corresponding optical images to form a training set;

[0008] The SAR images and optical images in the training set are input into the CFR-GAN model, and the discrimination results are output, wherein the CFR-GAN model includes a generation network and a discrimination network, and the generation network includes an encoder, a cross-fusion inference structure, and a decoder; the encoder encodes the SAR image to obtain SAR image features, the cross-fusion inference structure infers the SAR image features to obtain pseudo-optical image features, and the decoder decodes the pseudo-optical image features to obtain a pseudo-optical image; the discrimination network discriminates between the optical image and the pseudo-optical image, and outputs the discrimination results;

[0009] The optical image and the discrimination result are input into the pre-built total loss function to train the CFR-GAN model and obtain the CFR-GAN optimization model;

[0010] The SAR image to be processed is input into the CFR-GAN optimization model for conversion to obtain an optically converted image.

[0011] In one embodiment, the cross-fusion reasoning structure infers the SAR image features to obtain pseudo-optical image features, including:

[0012] The SAR image features are divided into two image features x1 and image features x2 of different scales. The image features x1 and x2 are respectively input into the three-layer residual block for convolution to obtain the first-stage output feature maps p1 and p2;

[0013] The first-stage output feature maps p1 and p2 are cross-fused to obtain the second-stage input feature maps y1, y2, and y3. The second-stage input feature maps y1, y2, and y3 are respectively input into the three-layer residual block for convolution to obtain the second-stage output feature maps q1, q2, and q3;

[0014] The second stage output feature maps q1, q2 and q3 are cross-fused to obtain the third stage input feature maps z1, z2, z3 and z4. The third stage input feature maps z1, z2, z3 and z4 are respectively input into the three-layer residual block for convolution to obtain the third stage output feature maps k1, k2, k3 and k4;

[0015] The third stage output feature maps k1, k2, k3 and k4 are fused to obtain pseudo optical image features.

[0016] In one embodiment, the first stage output feature maps p1 and p2 are cross-fused to obtain the second stage input feature maps y1, y2 and y3, including:

[0017] Fuse p1 and p2 after one upsampling to obtain the second stage input feature map y1;

[0018] Fuse p2 with the downsampled p1 to obtain the second-stage input feature map y2;

[0019] The twice downsampled p1 and the successively downsampled p2 are fused to obtain the second stage input feature map y3.

[0020] In one embodiment, the second stage output feature maps q1, q2 and q3 are cross-fused to obtain the third stage input feature maps z1, z2, z3 and z4, including:

[0021] Fuse q1, q2 upsampled once, and q3 upsampled twice to obtain the third-stage input feature map z1;

[0022] Fusion of q2, q1 after one downsampling, and q3 after two upsampling, to obtain the third stage input feature map z2;

[0023] Fusion of q3, q1 after two downsamplings, and q2 after one downsampling, to obtain the third stage input feature map z3;

[0024] The thrice downsampled q1, the twice downsampled q2, and the once downsampled q3 are fused to obtain the third stage input feature map z4.

[0025] In one of the embodiments, upsampling is performed according to a nearest neighbor interpolation algorithm, and downsampling is performed according to a convolution operation with a kernel size of 3×3 and a stride of 2.

[0026] In one embodiment, the identification network distinguishes the optical image and the pseudo-optical image and outputs the distinguishing result, including:

[0027] The identification network constructs a two-layer image pyramid structure to distinguish pseudo-optical images from optical images and obtains the discrimination results.

[0028] In one embodiment, the optical image and the discrimination result are input into a pre-constructed loss function, and the CFR-GAN model is trained to obtain a CFR-GAN optimization model, including:

[0029] Pre-built loss functions include the least squares generative adversarial loss L LSGAN , the perceptual loss L VGG and high-dimensional matching loss L m , respectively expressed as

[0030]

[0031]

[0032]

[0033] Among them, x represents the SAR image, y represents the optical image, G is the generation network, D is the identification network, and p data(x) represents the probability distribution of the SAR image x, p data(y) represents the probability distribution that the optical image y obeys, and represents the expected value, V i (·) represents the i-th feature extraction layer in the VGG19 network, C i , W i , H i They represent the number of channels, width, and height of the feature map respectively, and N represents the number of feature extraction layers.

[0034] D i (·) represents the i-th feature extraction layer in the identification network, M is the number of feature extraction layers in the identification network, D(y) represents the output result of the optical image after identification by the identification network; G(x) represents the pseudo optical image generated by the SAR image through the generation network;

[0035] The optical image and the discrimination result are input into the pre-constructed loss function, the loss functions are weighted and combined into a total loss function, and the CFR-GAN model is trained according to the total loss function to obtain the CFR-GAN optimization model.

[0036] In one embodiment, the loss functions are weighted and combined into a total loss function, and the CFR-GAN model is trained according to the total loss function to obtain a CFR-GAN optimization model, including:

[0037] The loss functions are weighted and combined into a total loss function L tatol , expressed as

[0038] L tatol =L LSGAN +μ1L VGG +μ2L m

[0039] Among them, μ1 and μ2 represent the weights for adjusting the balance between various loss functions;

[0040] According to the total loss function L tatol The CFR-GAN model is trained to obtain the CFR-GAN optimized model.

[0041] In one embodiment, obtaining a SAR image and a corresponding optical image to form a training set includes:

[0042] Acquire SAR images and corresponding optical images in the QXS-SAROPT data set to construct a first data set;

[0043] Acquire the SAR images and corresponding optical images in the SEN1-2 dataset to construct a second dataset;

[0044] A training set is formed according to the first data set and the second data set.

[0045] A SAR image to optical image conversion device, the device comprising:

[0046] Construct a data set module to obtain SAR images and corresponding optical images to form a training set;

[0047] The model training module is used to input the SAR image and the optical image in the training set into the CFR-GAN model and output the discrimination result, wherein the CFR-GAN model includes a generation network and a discrimination network, and the generation network includes an encoder, a cross-fusion inference structure, and a decoder; the encoder encodes the SAR image to obtain SAR image features, the cross-fusion inference structure infers the SAR image features to obtain pseudo-optical image features, and the decoder decodes the pseudo-optical image features to obtain a pseudo-optical image; the discrimination network discriminates between the optical image and the pseudo-optical image and outputs the discrimination result;

[0048] A model optimization module is used to input the optical image and the discrimination result into a pre-built total loss function to train the CFR-GAN model and obtain a CFR-GAN optimization model;

[0049] The test module is used to input the SAR image to be processed into the CFR-GAN optimization model for conversion to obtain an optically converted image.

[0050] The above-mentioned SAR image to optical image conversion method and device realize the conversion of SAR image to optical image through the CFR-GAN optimization model obtained by training. In the generation network of the CFR-GAN optimization model, a cross-fusion inference structure is used to simultaneously consider the high-resolution detail features and low-resolution semantic features. By realizing the conversion of SAR images of different scales to pseudo-optical images at different resolutions, the feature information of each scale of the SAR image is retained, thereby improving the conversion performance of the SAR image to the optical image. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 A schematic diagram of a flow chart of a method for converting a SAR image into an optical image in one embodiment;

[0052] Figure 2 A schematic diagram of the structure of a CFR-GAN model in one embodiment;

[0053] Figure 3 A schematic diagram of a cross-fusion reasoning structure in an embodiment;

[0054] Figure 4 A schematic diagram of pseudo optical image results generated by three feature inference structures in one embodiment;

[0055] Figure 5 It is a schematic diagram of pseudo optical image results generated by four network models on the dataset Dataset2 in one embodiment;

[0056] Figure 6 Schematic diagram for comparing the relationship between the parameter sizes of four network models and the model performance in one embodiment: (a) is a schematic diagram for comparing the relationship between the parameter sizes of four network models and the SSIM index value, (b) is a schematic diagram for comparing the relationship between the parameter sizes of four network models and the FSIM index value DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0058] In one embodiment, Figure 1 As shown, a method for converting a SAR image into an optical image is provided, comprising the following steps:

[0059] Step 102, the step is to obtain SAR images and corresponding optical images to form a training set.

[0060] It can be understood that the dataset contains various types of land features, including residential areas, roads, ports, farmlands, etc.

[0061] Step 104, input the SAR image and the optical image in the training set into the CFR-GAN model, and output the discrimination result, wherein the CFR-GAN model includes a generation network and a discrimination network, and the generation network includes an encoder, a cross-fusion inference structure, and a decoder; the encoder encodes the SAR image to obtain SAR image features, the cross-fusion inference structure infers the SAR image features to obtain pseudo-optical image features, and the decoder decodes the pseudo-optical image features to obtain a pseudo-optical image; the discrimination network discriminates between the optical image and the pseudo-optical image, and outputs the discrimination result.

[0062] It is understandable that Figure 2As shown in the figure, the CFR-GAN (Cross Fusion Reasoning-Generative Adversarial Network) model includes a generative network and a discriminative network. The network balance is achieved through the confrontation between the two networks, the conversion of SAR images to optical images is completed, and the large-scale and small-scale features are processed simultaneously through the cross-fusion reasoning structure in the generative network, thereby improving the clarity of the generated pseudo-optical image contour.

[0063] It can be understood that the encoder converts the input SAR image into high-dimensional SAR image features to achieve high-dimensional expression of SAR image features. The cross-fusion inference structure converts the SAR image features into pseudo-optical image features to complete the conversion of image features at high latitudes. The converted pseudo-optical image features are input into the subsequent decoding part to complete the mapping of high-dimensional features to the image to obtain the converted pseudo-optical image. The pseudo-optical image and the real optical image are input into the identification network for discrimination to discriminate the authenticity of the pseudo-optical image and obtain the discrimination result. The discrimination result indicates the degree of closeness between the pseudo-optical image and the optical image, and the value is 0-1. The closer the discrimination result is to 1, the higher the quality of the image conversion is, and the closer the pseudo-optical image is to the optical image.

[0064] Step 106, input the optical image and the discrimination result into a pre-constructed total loss function, train the CFR-GAN model, and obtain a CFR-GAN optimization model.

[0065] It can be understood that the optical image and the discrimination result are input into the pre-constructed total loss function, and the model is trained according to the total loss function. When the total loss function is optimal, the closer the pseudo-optical image is to the optical image, the closer the discrimination result is to 1. The parameters of the model at this time are used as the parameters of the optimization model to obtain the optimization model.

[0066] Step 108: Input the SAR image to be processed into the CFR-GAN optimization model for conversion to obtain an optically converted image.

[0067] It can be understood that by inputting the SAR image to be processed into the CFR-GAN optimization model for conversion, an optical conversion image is obtained, and the conversion performance of the CFR-GAN optimization model designed in the present invention can be verified.

[0068] The above-mentioned SAR image to optical image conversion method and device realize the conversion of SAR image to optical image through the CFR-GAN optimization model obtained by training. In the generation network of the CFR-GAN optimization model, a cross-fusion inference structure is used to simultaneously consider the high-resolution detail features and low-resolution semantic features. By realizing the conversion of SAR images of different scales to pseudo-optical images at different resolutions, the feature information of each scale of the SAR image is retained, thereby improving the conversion performance of the SAR image to the optical image.

[0069] In one embodiment, the cross-fusion reasoning structure is as follows Figure 3 As shown in Figure 1, the reasoning process of the cross-fusion reasoning structure includes three stages. Stage 1, stage 2, and stage 3 contain two, three, and four scale branches respectively. A single scale branch consists of three residual blocks. The specific steps include:

[0070] The SAR image features are divided into two image features x1 and image features x2 of different scales. The image features x1 and x2 are respectively input into the three-layer residual block for convolution to obtain the first-stage output feature maps p1 and p2, where the size of the feature map p1 is twice that of the feature map p2;

[0071] The first-stage output feature maps p1 and p2 are cross-fused to obtain the second-stage input feature maps y1, y2, and y3. The second-stage input feature maps y1, y2, and y3 are respectively input into the three-layer residual block for convolution to obtain the second-stage output feature maps q1, q2, and q3, where the size of feature map y1 is twice that of y2, and the size of feature map y2 is twice that of y3;

[0072] The second stage output feature maps q1, q2 and q3 are cross-fused to obtain the third stage input feature maps z1, z2, z3 and z4. The third stage input feature maps z1, z2, z3 and z4 are respectively input into the three-layer residual block for convolution to obtain the third stage output feature maps k1, k2, k3 and k4;

[0073] Finally, the output feature maps k1, k2, k3 and k4 of the third stage are fused to obtain the pseudo-optical image features.

[0074] In one embodiment, the first stage output feature maps p1 and p2 are cross-fused to obtain the second stage input feature maps y1, y2 and y3, including:

[0075] Fuse p1 and p2 after one upsampling to obtain the second stage input feature map y1;

[0076] Fuse p2 with the downsampled p1 to obtain the second-stage input feature map y2;

[0077] The twice downsampled p1 and the successively downsampled p2 are fused to obtain the second stage input feature map y3.

[0078] In one embodiment, the second stage output feature maps q1, q2 and q3 are cross-fused to obtain the third stage input feature maps z1, z2, z3 and z4, including:

[0079] Fuse q1, q2 upsampled once, and q3 upsampled twice to obtain the third-stage input feature map z1;

[0080] Fusion of q2, q1 after one downsampling, and q3 after two upsampling, to obtain the third stage input feature map z2;

[0081] Fusion of q3, q1 after two downsamplings, and q2 after one downsampling, to obtain the third stage input feature map z3;

[0082] The thrice downsampled q1, the twice downsampled q2, and the once downsampled q3 are fused to obtain the third stage input feature map z4.

[0083] In one of the embodiments, the cross-fusion inference structure performs upsampling according to a nearest neighbor interpolation algorithm and performs downsampling according to a convolution operation with a kernel size of 3×3 and a stride of 2.

[0084] It can be understood that the cross-fusion reasoning structure combines the horizontal deepening network and the vertical widening network, makes full use of features of different sizes, and processes high-resolution and low-resolution features simultaneously in the entire feature reasoning process, retaining the feature information of each scale, and there are high-resolution detail features and low-resolution semantic features in the entire feature reasoning process.

[0085] In one embodiment, the identification network distinguishes the optical image and the pseudo-optical image and outputs the distinguishing result, including:

[0086] The identification network constructs a two-layer image pyramid structure to distinguish pseudo-optical images from optical images and obtains the discrimination results.

[0087] It can be understood that the identification network is used to determine whether the generated image is an optical image, and a two-layer image pyramid structure is constructed to distinguish the pseudo-optical image from the optical image, and the discrimination result is output to determine the similarity between the two images.

[0088] In one embodiment, the optical image and the discrimination result are input into a pre-constructed loss function, and the CFR-GAN model is trained to obtain a CFR-GAN optimization model, including:

[0089] Pre-built loss functions include the least squares generative adversarial loss L LSGAN , the perceptual loss L VGG and high-dimensional matching loss L m , respectively expressed as

[0090]

[0091]

[0092]

[0093] Among them, x represents the SAR image, y represents the optical image, G is the generation network, D is the identification network, and p data(x) represents the probability distribution of the SAR image x, p data(y) represents the probability distribution that the optical image y obeys, and represents the expected value, V i (·) represents the i-th feature extraction layer in the VGG19 network, C i , W i , H i They represent the number of channels, width, and height of the feature map, respectively. N represents the number of feature extraction layers. D i (·) represents the i-th feature extraction layer in the identification network, M is the number of feature extraction layers in the identification network, D(y) represents the output result of the optical image after identification by the identification network; G(x) represents the pseudo optical image generated by the SAR image through the generation network;

[0094] The optical image and the discrimination result are input into the pre-constructed loss function, the loss functions are weighted and combined into a total loss function, and the CFR-GAN model is trained according to the total loss function to obtain the CFR-GAN optimization model.

[0095] Specifically, the VGG19 network exists in the CFR-GAN model as a perceptual loss function that calculates the network behavior constraints. The VGG19 network extracts high-dimensional features of the true values ​​of the pseudo-optical image and the real optical image, and calculates the difference between the two. Network training optimizes the CFR-GAN model parameters by continuously reducing the difference between the two.

[0096] In one embodiment, the loss functions are weighted and combined into a total loss function, and the CFR-GAN model is trained according to the total loss function to obtain a CFR-GAN optimization model, including:

[0097] The loss functions are weighted and combined into a total loss function L tatol , expressed as

[0098] L tatol=L LSGAN +μ1L VGG +μ2L m

[0099] Among them, μ1 and μ2 represent the weights for adjusting the balance between various loss functions;

[0100] According to the total loss function L tatol The CFR-GAN model is trained to obtain the CFR-GAN optimized model.

[0101] Specifically, the model was trained on PyTorch and RTX2080Ti single GPU, with the number of training rounds and batch size set to 200 and 10 respectively. The generative network and the discriminative network were trained using the Adam optimizer on the pre-built total loss function, with the training parameters set to β1=0.5 and β2=0.999. The model was trained according to the total loss function, with the learning rate of the first 100 rounds of training remaining unchanged at 0.0002 and the learning rate of the next 100 rounds of training decreasing linearly to 0. In addition, the generative network in the model used dropout operation during training to prevent the loss of the discriminative network from dropping rapidly to 0.

[0102] It can be understood that the model is trained according to the total loss function. When the total loss function is optimal, the closer the pseudo-optical image is to the optical image, the closer the discrimination result is to 1, and the higher the quality of the model image conversion. The parameters of the model at this time are used as the parameters of the optimization model to obtain the optimization model.

[0103] In one embodiment, obtaining a SAR image and a corresponding optical image to form a training set includes:

[0104] 13333 pairs of SAR images and optical images were obtained from the QXS-SAROPT (Qian Xuesen Laboratory of Space Technology-Synthetic aperture radar Optical) dataset to construct the first dataset Dataset1. QXS-SAROPT includes Gaofen-3 images and multispectral images from Google Earth. The size of each image is 256×256, and its spatial resolution is 1 meter.

[0105] 3123 pairs of SAR images and optical images in the SEN1-2 (Sentinel 1-2) dataset are obtained to construct the second dataset Dataset2, where SEN1-2 contains sentinel-1 SAR images and sentinel-2 optical images. The size of each image is 256×256 and the spatial resolution is 10 meters. The first dataset Dataset1 and the second dataset Dataset2 are divided into a training set and a data set in a ratio of 3:1.

[0106] It can be understood that the constructed CFR-GAN model is optimized and trained according to the training set, and the SAR image to be processed in the test set is input into the CFR-GAN optimization model for conversion to obtain the optical conversion image, and the CFR-GAN optimization model is tested and verified.

[0107] Specifically, in order to further verify the quality of images generated by the CFR-GAN optimization model constructed by the present invention, image quality assessment (IQA) - root mean square error (RMSE), peak signal-to-noise ratio (PSNR), structural similarity index metric (SSIM) and feature similarity index (FSIM) are used to evaluate the quality of images generated by different models. Among them, the lower the RMSE value, the higher the image quality; the higher the PSNR, SSIM and FSIM values, the higher the quality of the generated image.

[0108] In a specific embodiment, in order to evaluate the effectiveness of the cross-fusion reasoning structure, the loss function is kept unchanged, and the pseudo optical images generated by the cross-fusion reasoning structure proposed in the present invention and the 9 residual block cascade structure and the U-Net reasoning structure are compared with quantitative indicators on the data set Dataset1. The specific quantitative indicators are shown in Table 1. As can be seen from Table 1, compared with the 9 residual block cascade structure and the U-Net reasoning structure, the pseudo optical image generated by the cross-fusion reasoning structure has the lowest RMSE and the lowest PSNR value, and in terms of the two indicators SSIM and FSIM that match the human visual system, the SSIM value and FSIM value of the pseudo optical image generated by the cross-fusion reasoning structure are the highest, that is, the pseudo optical image generated by the cross-fusion reasoning structure has the highest quality.

[0109] Table 1 Quality evaluation results of pseudo optical images generated by different feature inference structures

[0110] Reasoning structure RMSE PSNR SSIM FSIM 9-residual blocks 49.8475 15.0062 0.3161 0.3191 U-net 50.0839 14.9479 0.3414 0.3301 Cross-fusion 48.9523 15.1370 0.3566 0.3495

[0111] In addition, pseudo-optical images can be generated to intuitively compare the performance of these three inference structures, e.g. Figure 4 As shown in the figure, column (a) is the input SAR image, columns (b) to (d) are pseudo optical images generated by networks with U-Net inference structure, 9 residual block inference structure and cross-fusion inference structure, respectively, and column (e) is the true value of the optical image. Figure 4 It can be seen that the pseudo optical images generated by the cross-fusion inference structure are closer to the basic facts. Compared with the images generated by other methods, they have clearer contour edges and more precise details (the specific details are marked in Figure 4 in the gray box in the figure), for example, Figure 4In the first row, the image generated by the cross-fusion reasoning structure has a clearer road in the middle of the residential building than other methods, in the second row, the cross-fusion reasoning structure successfully restores the texture in the middle right of the image, in the third row, the road in the green vegetation is more complete in the image generated by the cross-fusion reasoning structure, and in the last row, the white intersection in the image generated by the cross-fusion reasoning structure is more detailed, which shows that the cross-fusion reasoning structure of the present invention can capture more detailed structural information than other reasoning structures.

[0112] It can be understood that both quantitative evaluation indicators and intuitive visual inspection show the superiority of the cross-fusion reasoning structure in the conversion process between SAR images and optical images, which shows that the cross-fusion reasoning structure of the present invention has a higher ability to retain details in the conversion between SAR and optical images.

[0113] In a specific embodiment, three SAR and optical conversion models, pix2pix, cycleGAN and NICE-GAN, are selected and compared with the CFR-GAN model. Keeping other experimental conditions unchanged, the four models are trained and tested on Dataset1 and Dataset2, respectively. The quantitative measurement results are shown in Tables 2 and 3, respectively.

[0114] According to Table 2 and Table 3, the pseudo optical image generated by the CFR-GAN model proposed in the present invention has the lowest RMSE, the highest PSNR, the highest SSIM and the highest FSIM among the four methods, that is, the image generated by the CFR-GAN model is the best among the four models. In addition, the pseudo optical image results generated by different models on the dataset Dataset2 are qualitatively analyzed, as shown in Figure 2. Figure 5 As shown in Figure 2, different images generated by different models are shown above. Figure 5 Column (a) is the input SAR image, columns (b) to (e) are pseudo optical images generated by the pix2pix model, cycleGAN model, NICE-GAN model, and CFR-GAN model, respectively, and column (f) is the true value of the optical image. It can be found that the image quality gradually improves from column (b) to column (e). Figure 5 The details of the generated image are marked with gray boxes. Figure 5 In the first and second rows, the images generated by the CFR-GAN model have more details in the residential areas than other methods, while in the last row, the images generated by the CFR-GAN model are almost as clear as the ground truth. Figure 5 In the third row of images, the CFR-GAN model proposed in this invention is better and consistent than other models.

[0115] Table 2 Quality evaluation results of pseudo optical images generated by different models on the dataset Datasat1

[0116] Methods RMSE PSNR SSIM FSIM pix2pix 58.4413 13.1464 0.2319 0.3278 cycleGAN 58.7937 13.4339 0.2871 0.3305 NICE-GAN 64.3323 12.9229 0.2776 0.3352 CFR-GAN(ours) 48.9523 15.1370 0.3566 0.3495

[0117] Table 3 Quality evaluation results of pseudo optical images generated by different models on the Datasat2 dataset

[0118] Methods RMSE PSNR SSIM FSIM pix2pix 62.7168 12.4586 0.1492 0.3235 cycleGAN 62.8372 12.7533 0.1718 0.3448 NICE-GAN 63.9169 12.5358 0.1650 0.3312 CFR-GAN(ours) 52.2850 14.7328 0.2798 0.4002

[0119] In a specific embodiment, in order to further verify the robustness of different models, specifically, the image quality generated by input images of different ground object types is evaluated, the dataset Dataset2 is divided into five categories, and the quality of images of different ground object categories generated by different models is calculated, where the results of RMSE, PSNR, SSIM, and FSIM are shown in Table 4. It can be seen from Table 4 that, in general, the quality of images generated by the CFR-GAN model is better than other methods in terms of the four indicators, and in different categories of images, the CRF-GAN model is generally better than the pix2pix, cycleGAN, and NICE-GAN models.

[0120] In a specific embodiment, the performance of four models and the relationship between their parameter sizes are compared, such as Figure 6 As shown, Figure 6 (a) shows a schematic diagram of the relationship between the parameter size of the four network models and the SSIM index value, and (b) shows a schematic diagram of the relationship between the parameter size of the four network models and the FSIM index value. The horizontal coordinates of (a) and (b) are both model parameter sizes, the vertical coordinate of (a) is the SSIM index value, and the vertical coordinate of (b) is the FSIM index value. Compared with the pix2pix model and the NICEGAN model, the CRF-GAN model proposed in the present invention has a smaller model and higher performance. Although the CFR-GAN model is slightly larger than the cycleGAN model, the performance of the CFR-GAN model is much higher than that of the cycleGAN model. Therefore, through qualitative and quantitative comparisons, the superiority of the CFR-GAN model proposed in the present invention is verified.

[0121] Table 4 Quality evaluation results of pseudo optical images generated by different models from input SAR images of different ground object types

[0122]

[0123]

[0124] It should be understood that although Figure 1The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0125] In one embodiment, a SAR image to optical image conversion device is provided, comprising: a training set construction module, a model training module, a model optimization module and a testing module, wherein:

[0126] The training set building module is used to obtain SAR images and corresponding optical images to form a training set.

[0127] It can be understood that the training set contains various types of land features, including residential areas, roads, ports, farmlands, etc.

[0128] The model training module is used to input the SAR images and optical images in the training set into the CFR-GAN model and output the discrimination results, wherein the CFR-GAN model includes a generation network and a discrimination network, and the generation network includes an encoder, a cross-fusion inference structure, and a decoder; the encoder encodes the SAR image to obtain SAR image features, the cross-fusion inference structure infers the SAR image features to obtain pseudo-optical image features, and the decoder decodes the pseudo-optical image features to obtain a pseudo-optical image; the discrimination network discriminates between the optical image and the pseudo-optical image and outputs the discrimination results.

[0129] It can be understood that the CFR-GAN model includes a generation network and a discrimination network. The network balance is achieved through the confrontation between the two networks, the conversion of SAR images to optical images is completed, and the large-scale and small-scale features are processed simultaneously through the cross-fusion reasoning structure in the generation network, thereby improving the clarity of the generated pseudo-optical image contour.

[0130] It can be understood that the encoder converts the input SAR image into high-dimensional SAR image features to achieve high-dimensional expression of SAR image features. The cross-fusion inference structure converts the SAR image features into pseudo-optical image features to complete the conversion of image features at high latitudes. The converted pseudo-optical image features are input into the subsequent decoding part to complete the mapping of high-dimensional features to the image to obtain the converted pseudo-optical image. The pseudo-optical image and the real optical image are input into the identification network for discrimination to discriminate the authenticity of the pseudo-optical image and obtain the discrimination result. The discrimination result indicates the degree of closeness between the pseudo-optical image and the optical image, and the value is 0-1. The closer the discrimination result is to 1, the higher the quality of the image conversion is, and the closer the pseudo-optical image is to the optical image.

[0131] The model optimization module is used to input the optical image and the discrimination result into the pre-built total loss function, train the CFR-GAN model, and obtain the CFR-GAN optimization model.

[0132] It can be understood that the optical image and the discrimination result are input into the pre-constructed total loss function, and the model is trained according to the total loss function. When the total loss function is optimal, the closer the pseudo-optical image is to the optical image, the closer the discrimination result is to 1. The parameters of the model at this time are used as the parameters of the optimization model to obtain the optimization model.

[0133] The test module is used to input the SAR image to be processed into the CFR-GAN optimization model for conversion to obtain an optically converted image.

[0134] It can be understood that by inputting the SAR image to be processed into the CFR-GAN optimization model for conversion to obtain an optically converted image, the conversion performance of the CFR-GAN optimization model designed in the present invention can be verified.

[0135] The specific definition of a SAR image to optical image conversion device can refer to the definition of a SAR image to optical image conversion method mentioned above, which will not be repeated here. Each module in the above-mentioned SAR image to optical image conversion device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0136] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0137] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for converting a SAR image into an optical image, characterized in that: The method comprises: Acquire SAR images and corresponding optical images to form a training set; The SAR image and the optical image in the training set are input into the CFR-GAN model, and a discrimination result is output, wherein the CFR-GAN model includes a generation network and a discrimination network, and the generation network includes an encoder, a cross-fusion inference structure, and a decoder; the encoder encodes the SAR image to obtain SAR image features, the cross-fusion inference structure infers the SAR image features to obtain pseudo-optical image features, and the decoder decodes the pseudo-optical image features to obtain a pseudo-optical image; the discrimination network discriminates the optical image and the pseudo-optical image, and outputs a discrimination result; Inputting the optical image and the discrimination result into a pre-constructed loss function, training the CFR-GAN model, and obtaining a CFR-GAN optimization model; Inputting the SAR image to be processed into the CFR-GAN optimization model for conversion to obtain an optically converted image; The optical image and the discrimination result are input into a pre-constructed loss function, the CFR-GAN model is trained, and a CFR-GAN optimization model is obtained, including: Pre-built loss functions include least squares generative adversarial loss , perceptual loss and high-dimensional matching loss , respectively expressed as ; ; ; Wherein, x represents the SAR image, y represents the optical image, G is the generation network, and D is the identification network. represents the probability distribution of the SAR image x, represents the probability distribution that the optical image y obeys, and represents the expected value, represents the i-th feature extraction layer in the VGG19 network, They represent the number of channels, width, and height of the feature map respectively, and N represents the number of feature extraction layers. represents the i-th feature extraction layer in the identification network, M is the number of feature extraction layers in the identification network, D(y) represents the output result of the optical image after identification by the identification network; G(x) represents the pseudo optical image generated by the SAR image through the generation network; Inputting the optical image and the discrimination result into a pre-constructed loss function, weighting and combining the loss functions into a total loss function, and training the CFR-GAN model according to the total loss function to obtain a CFR-GAN optimization model; The loss functions are weighted and combined into a total loss function, and the CFR-GAN model is trained according to the total loss function to obtain a CFR-GAN optimization model, including: The loss functions are weighted and combined into a total loss function , expressed as in, and Represents the weight for adjusting the balance between various loss functions; According to the total loss function The CFR-GAN model is trained to obtain a CFR-GAN optimization model.

2. The method according to claim 1, characterized in that The cross-fusion reasoning structure infers the SAR image features to obtain pseudo-optical image features, including: The SAR image features are divided into two image features of different scales. and image features , the image features and image features Input three layers of residual blocks for convolution respectively to obtain the output feature map of the first stage and ; The first stage output feature map and Perform cross fusion to obtain the second stage input feature map , and , the second stage input feature map , and Input three layers of residual blocks for convolution respectively to obtain the second stage output feature map , and ; The second stage output feature map , and Perform cross fusion to obtain the third stage input feature map , , and , the third stage input feature map , , and Input three layers of residual blocks for convolution respectively to obtain the output feature map of the third stage , , and ; The third stage output feature map , , and The pseudo optical image features are obtained by fusion.

3. The method according to claim 2, characterized in that The first stage output feature map and Perform cross fusion to obtain the second stage input feature map , and ,include: The and the one-time upsampling Fusion is performed to obtain the second stage input feature map ; The and the one-time downsampling Fusion is performed to obtain the second stage input feature map ; The two-downsampled and the successively downsampled Fusion is performed to obtain the second stage input feature map .

4. The method according to claim 2, characterized in that: The second stage output feature map , and Perform cross fusion to obtain the third stage input feature map , , and ,include: The , after one upsampling and the two-upsampled Fusion is performed to obtain the third stage input feature map ; The , after one downsampling and the two-upsampled Fusion is performed to obtain the third stage input feature map ; The , after two downsampling and the one-time downsampling Fusion is performed to obtain the third stage input feature map ; The three-times downsampled , after two downsampling and the one-time downsampling Fusion is performed to obtain the third stage input feature map .

5. The method according to any one of claims 3 to 4, characterized in that: Upsampling is performed according to the nearest neighbor interpolation algorithm, and downsampling is performed according to the convolution operation with a kernel size of 3×3 and a stride of 2.

6. The method according to claim 1, characterized in that The identification network discriminates the optical image and the pseudo-optical image and outputs a discrimination result, including: The identification network discriminates the pseudo optical image and the optical image by constructing a two-layer image pyramid structure to obtain a discrimination result.

7. The method according to claim 1, characterized in that Acquire SAR images and corresponding optical images to form a training set, including: Acquire paired SAR images and optical images in the QXS-SAROPT dataset to construct the first dataset; Acquire the SAR image and the optical image constituting the pair of SEN1-2 data sets to construct the second data set; A training set is formed according to the first data set and the second data set.

8. A SAR image to optical image conversion device, characterized in that: The device comprises: Construct a data set module to obtain SAR images and corresponding optical images to form a training set; A model training module is used to input the SAR image and the optical image in the training set into the CFR-GAN model and output a discrimination result, wherein the CFR-GAN model includes a generation network and a discrimination network, and the generation network includes an encoder, a cross-fusion inference structure, and a decoder; the encoder encodes the SAR image to obtain SAR image features, the cross-fusion inference structure infers the SAR image features to obtain pseudo-optical image features, and the decoder decodes the pseudo-optical image features to obtain a pseudo-optical image; the discrimination network discriminates the optical image and the pseudo-optical image and outputs a discrimination result; A model optimization module, used for inputting the optical image and the discrimination result into a pre-constructed total loss function, training the CFR-GAN model, and obtaining a CFR-GAN optimization model; A testing module, used for inputting the SAR image to be processed into the CFR-GAN optimization model for conversion to obtain an optically converted image; The optical image and the discrimination result are input into a pre-constructed loss function, the CFR-GAN model is trained, and a CFR-GAN optimization model is obtained, including: Pre-built loss functions include least squares generative adversarial loss , perceptual loss and high-dimensional matching loss , respectively expressed as ; ; ; Wherein, x represents the SAR image, y represents the optical image, G is the generation network, and D is the identification network. represents the probability distribution of the SAR image x, represents the probability distribution that the optical image y obeys, and represents the expected value, represents the i-th feature extraction layer in the VGG19 network, They represent the number of channels, width, and height of the feature map respectively, and N represents the number of feature extraction layers. represents the i-th feature extraction layer in the identification network, M is the number of feature extraction layers in the identification network, D(y) represents the output result of the optical image after identification by the identification network; G(x) represents the pseudo optical image generated by the SAR image through the generation network; Inputting the optical image and the discrimination result into a pre-constructed loss function, weighting and combining the loss functions into a total loss function, and training the CFR-GAN model according to the total loss function to obtain a CFR-GAN optimization model; The loss functions are weighted and combined into a total loss function, and the CFR-GAN model is trained according to the total loss function to obtain a CFR-GAN optimization model, including: The loss functions are weighted and combined into a total loss function , expressed as in, and Represents the weight for adjusting the balance between various loss functions; According to the total loss function The CFR-GAN model is trained to obtain a CFR-GAN optimization model.

Citation Information

Patent Citations

  • Method for converting SAR image into optical image based on generative adversarial network

    CN110363215A

  • SAR and optical image bidirectional translation method based on cascade residual generative adversarial network

    CN111784560A

  • SAR optical image mapping model lightweight method based on conditional generative adversarial network

    CN114202017A

  • Recovering occluded image data using machine learning

    US20210133936A1