SAR-optical image conversion method based on semantic segmentation driving

Abstract

The invention provides an SAR-optical image conversion method based on semantic segmentation driving. Firstly, SAR and optical image data are collected, and semantic segmentation labels of the SAR and optical image data are made; then, respectively training the SAR and the optical image to obtain corresponding semantic segmentation models; secondly, two downstream task branches are added under an existing conversion model basic framework, a semantic segmentation task driven conversion model is constructed, a loss function of the model is designed, and the model is trained through collected data; and finally, converting an input SAR image into an optical image by using the trained model. Since the semantic segmentation task is introduced into the conversion model, guidance information can be continuously provided and fed back during model training, so that the integrity, definition and the like of the target in the optical image obtained by conversion are improved, and a better image conversion effect is achieved.

Description

Technical Field

[0001] The invention belongs to the technical fields of computer vision, image conversion and SAR image interpretation, and in particular relates to a SAR-optical image conversion method based on semantic segmentation drive. Background Art

[0002] In the evolution of remote sensing imaging technology, optical remote sensing imaging dominated by visible light and synthetic aperture radar (SAR) imaging based on electromagnetic waves have become two indispensable technologies for earth observation. SAR imaging, with its active detection and imaging characteristics that are not restricted by weather, enables it to perform imaging at any time and weather conditions, which undoubtedly brings important value to many application fields. However, its complex imaging mechanism makes the interpretation of SAR images quite difficult, which also restricts its widespread use in practical applications.

[0003] At the same time, optical remote sensing images are favored for their high spatial resolution and rich color information, but unfortunately, they are easily disturbed by bad weather, which limits their application scope to a certain extent. Nevertheless, SAR images and optical images are complementary in many application scenarios. However, due to the significant differences in their physical properties and imaging principles, there are obvious differences between them.

[0004] In recent years, with the rapid development of image conversion technology, especially the SAR to optical image conversion (S2OIT) technology based on generative adversarial networks, a new perspective has been provided for the interpretation of SAR images. In order to combine the advantages of SAR and optical images, researchers have proposed a series of image conversion methods based on machine learning and deep learning technology. These methods realize the conversion from SAR images to images with optical features by learning and simulating the complex mapping relationship between SAR and optical images.

[0005] Existing image conversion methods focus more on improving image pixel indicators such as SSIM and PSNR, which only evaluate the difference between the real optical image and the generated image in isolation, and ignore the fact that pixels must be connected to each other to form a specific identifiable target. As a result, although the generated optical image has high indicators such as SSIM, some targets are lost, incomplete, and blurred. However, the significance of SAR-optical image conversion is more to facilitate the interpretation of personnel or programs. The loss of key targets in the image after conversion will make the image conversion meaningless. Summary of the invention

[0006] In order to overcome the shortcomings of the prior art, the present invention provides a SAR-optical image conversion method driven by semantic segmentation. First, SAR image and optical image data are collected, and their binary semantic segmentation labels are made respectively; then, the corresponding semantic segmentation models are obtained by training the SAR image and the optical image respectively; then, two downstream task branches are added to the basic framework of the existing conversion model to construct a conversion model driven by the semantic segmentation task, design the corresponding model loss function, and train the model with the collected data to obtain the optimal conversion model; finally, the input SAR image is converted into an optical image using the trained model. Since the semantic segmentation task is introduced into the conversion model, it can continuously provide guidance information and feedback to the model during training, so that the integrity and clarity of the target in the converted optical image are improved, with better image conversion effect, and more accurate data support is provided for subsequent image interpretation and analysis tasks.

[0007] A SAR-optical image conversion method based on semantic segmentation drive is characterized by the following steps:

[0008] Step 1, data collection: collect SAR images and optical images containing the same scene targets and the same resolution as the training data set, classify the registered SAR images and optical images into the paired data set, and classify the unregistered SAR images and optical images into the unpaired data set. Then, create pixel-level binary classification semantic segmentation labels for the images in the data set, where the SAR images and optical images in the paired data set share the same semantic segmentation labels;

[0009] Step 2, data preprocessing: preprocess the SAR images and optical images in the data set;

[0010] Step 3, semantic segmentation model pre-training: using the pre-processed SAR image, optical image and its semantic segmentation label as input, the semantic segmentation model is trained, wherein the SAR image and the semantic segmentation label are used to train the SAR-semantic segmentation model, and the optical image and the semantic segmentation label are used to train the optical-semantic segmentation model;

[0011] Step 4, build a conversion model driven by the semantic segmentation task: use a GAN network or a diffusion model as the basic framework of the conversion model, wherein for paired data sets, a supervised conversion model is used, and for unpaired data sets, an unsupervised conversion model is used. Two downstream task branches are added to the basic framework of the conversion model to obtain a conversion model driven by the semantic segmentation task, wherein one downstream task branch uses the SAR-semantic segmentation model to segment the SAR image input into the conversion model, and the segmentation result and the SAR image are input into the conversion model together; the other downstream task branch uses the optical-semantic segmentation model to segment the pseudo-optical image generated by the conversion and the real optical image respectively, and compares and evaluates the segmentation results, and feeds the evaluation results back to the conversion model;

[0012] Step 5: Design the model loss function: The total model loss is obtained by adding the adversarial loss, supervision loss, and semantic segmentation loss. For the unsupervised conversion model, the cross entropy loss is used as the semantic segmentation loss. For the supervised conversion model, the cross entropy loss, feature matching loss, and semantic segmentation-guided L1 loss are added as the final semantic segmentation loss.

[0013] Step 6, iterative training: using the image preprocessed in step 2 and its semantic segmentation label as input, iteratively train the network model constructed in steps 4-5. During each training, adjust the parameters of the conversion model generator according to the feedback signal of the conversion model discriminator until a conversion model with the best output result is obtained;

[0014] Step 7, model application: the SAR image to be converted is input into the optimal conversion model obtained after training, and the generator of the conversion model outputs the corresponding converted optical image.

[0015] Specifically, SAR images and optical images are obtained from public datasets, satellite image databases or self-collection.

[0016] Specifically, the pixel-level binary classification semantic segmentation labels for the images in the data set are produced manually or by machine, wherein machine production refers to production using relevant software tools including EISeg.

[0017] Specifically, the data preprocessing includes image scaling, cropping, and image enhancement operations.

[0018] Specifically, the semantic segmentation model adopts the FCN model.

[0019] Specifically, the adversarial loss adopts the least square loss, and the supervised loss adopts the L1 loss.

[0020] Specifically, the specific calculation method of the cross entropy loss is: using the optical-semantic segmentation model to segment the optical image generated by the conversion model, and calculating the cross entropy loss of the segmentation result and the semantic segmentation label of the SAR image as the final cross entropy loss;

[0021] The specific calculation method of the feature matching loss is: using the downsampling network of the optical-semantic segmentation model, downsampling the input optical image and the pseudo optical image generated by the conversion model respectively, calculating an L1 loss for the features obtained by downsampling at each level, and calculating the average of the L1 losses of all levels as the final feature matching loss;

[0022] The specific calculation method of the semantic segmentation-guided enhanced L1 loss is: use the optical-semantic segmentation model to perform binary segmentation on the input optical image, multiply the segmentation result element by element with the input optical image and the pseudo optical image generated by the conversion model, calculate the L1 loss of the two images obtained after the multiplication, and multiply it by a weight as the final semantic segmentation-guided enhanced L1 loss, wherein the weight value is 10 to 20.

[0023] The beneficial effects of the present invention are as follows: by introducing SAR image semantic segmentation as information supplement and optical image semantic segmentation feedback information as constraint, the converted image is more in line with the requirements of specific tasks, the generation of the target of interest is more complete and clear, and closer to the real optical image. The present invention has a wide range of application prospects and is of great significance in the fields of disaster relief, military reconnaissance, urban planning, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is an example diagram of a data set collected by an embodiment of the present invention;

[0025] Figure 2 It is a schematic diagram of the conversion model driven by the semantic segmentation task of the present invention. DETAILED DESCRIPTION

[0026] The present invention is further described below in conjunction with the accompanying drawings and embodiments. The present invention includes but is not limited to the following embodiments.

[0027] The present invention provides a SAR-optical image conversion method based on semantic segmentation drive, and its specific implementation process is as follows:

[0028] 1. Data collection. In the technical solution of the present invention, SAR images and corresponding optical images must first be collected as training data sets. The data can be obtained from public data sets, satellite image databases or self-collection. The data set should contain the same scene targets and have consistent resolution. According to the actual application conditions, a paired data set or an unpaired data set is constructed, wherein the paired data set refers to a SAR image and an optical image in which the SAR image and the optical image have been registered. Then, pixel-level binary classification semantic segmentation labels should be made for each SAR image and optical image. For a paired data set, the SAR image and the optical image can share a semantic segmentation label. Semantic segmentation labels can be made manually or by machine, such as using EISeg as a semantic segmentation annotation tool to perform semantic segmentation on the collected images. This step is intended to provide training data for subsequent model training. In order to obtain a better conversion model, data with diversity can be collected as much as possible during the data collection process, that is, SAR images and corresponding optical images of different scenes and different targets. Figure 1 An example of the SEN1-2 road dataset constructed by the embodiment of the present invention is given, where Figure (a) is the paired dataset of SEN1-2 in spring, and Figure (b) is the paired dataset of SEN1-2 in summer. The first column in Figure (a) and Figure (b) is the optical image, the second column is the SAR image paired with the optical image, the third column is the binary road segmentation label made with the SAR image and the optical image, and the fourth column is the visualized image of the label.

[0029] 2. Data preprocessing. Use common preprocessing methods to preprocess the collected SAR images and optical images to improve the robustness of the training model. Common preprocessing methods include image scaling, cropping, image enhancement and other operations.

[0030] 3. Pre-training of semantic segmentation model. According to the specific target segmentation task, select an existing segmentation model with high accuracy (such as FCN), use the previously collected SAR images, optical images and their semantic segmentation labels, and train the semantic segmentation model separately to prepare for the subsequent construction of the downstream task-driven framework. Among them, the SAR-semantic segmentation model is trained using SAR images and semantic segmentation labels, and the optical-semantic segmentation model is trained using optical images and semantic segmentation labels.

[0031] 4. Construct a conversion model driven by the semantic segmentation task. According to the task requirements, you can select the GAN network framework or the diffusion model framework as the basic framework of the conversion model, or you can build a specific model framework yourself. For paired data sets, a supervised conversion model is adopted, and for unpaired data sets, an unsupervised conversion model is adopted. The present invention adds two downstream task branches to the original framework: the SAR image input to the conversion model is segmented by the SAR-semantic segmentation model, and the segmentation result is input into the conversion model together with the SAR image; the optical-semantic segmentation model pre-trained on the real optical image is used to segment the pseudo-optical image generated by the conversion and the real optical image, and the segmentation results are compared and evaluated, and the evaluation results are fed back to the conversion model.

[0032] In the design of the loss function of the model, in addition to using the adversarial loss and supervised loss of the basic model, the present invention also introduces semantic segmentation loss to promote the conversion of SAR images to optical images. The total loss of the model is obtained by adding the adversarial loss, supervised loss and semantic segmentation loss. Different loss combinations are used for different conversion models: for the unsupervised conversion model, the cross entropy loss is used as the semantic segmentation loss. For the supervised conversion model, the cross entropy loss, feature matching loss and semantic segmentation-guided L1 loss are added as the final semantic segmentation loss. In this way, the generated optical image can achieve better performance on specific tasks.

[0033] The least squares loss can be used for adversarial loss, and the L1 loss can be used for supervised loss. The specific design of semantic segmentation loss is as follows:

[0034] (1) Cross-entropy loss: The optical image generated by the conversion model is segmented using the optical-semantic segmentation model pre-trained on the real optical image, and the cross-entropy loss between the segmentation result and the SAR image semantic segmentation label of the image is calculated as the final cross-entropy loss;

[0035] (2) Feature matching loss: The downsampling network of the optical-semantic segmentation model pre-trained on real optical images is used to downsample the input real optical image and the pseudo optical image generated by the conversion model. The L1 loss is calculated for the features obtained by downsampling at each level, and the average of the L1 losses at all levels is calculated as the final feature matching loss.

[0036] (3) Semantic segmentation-guided enhanced L1 loss: The optical-semantic segmentation model pre-trained on the real optical image is used to perform binary segmentation on the input optical image. The 01 matrix of the segmentation result is element-wise multiplied with the input real optical image and the pseudo optical image generated by the conversion model. The L1 loss of the two images obtained after the multiplication is calculated and multiplied by a weight as the final semantic segmentation-guided enhanced L1 loss, where the weight is 10 to 20.

[0037] Figure 2 A schematic block diagram of the conversion model driven by semantic segmentation task of the present invention is given.

[0038] 5. Iterative training. The network model constructed in step 4 is iteratively trained using the image preprocessed in step 2 and its semantic segmentation label as input. During each training, the parameters of the conversion model generator are adjusted according to the feedback signal of the conversion model discriminator until the conversion model with the best output result is obtained.

[0039] The training process is as follows: first, the SAR-semantic segmentation model trained on SAR images is used to segment the SAR images to obtain the probability matrix of the segmentation results, and the matrix is ​​concatenated with the SAR images by channel as the actual input of the conversion model; in the process of repeated iterative training, the parameters of the generator and the discriminator are continuously optimized to improve the quality and segmentation performance of the generated optical images. In each iteration, the parameters of the conversion model generator are adjusted according to the feedback signal of the conversion model discriminator to reduce the difference between the generated image and the real image.

[0040] 6. Model application. In practical applications, the trained network model is used to convert the input SAR image into an image with optical image features, that is, the generator model of the trained conversion model is used to convert the input SAR image into the corresponding optical image.

[0041] The converted image can be used for tasks related to optical images, such as semantic segmentation, target recognition, etc. In the process of model training, the present invention further makes full use of the semantic information of SAR images and optical images, improves the quality and fidelity of the synthesized image, and focuses on the generation integrity of the target, so that the generated image better meets the requirements of downstream tasks.

Claims

1. A SAR-optical image conversion method driven by semantic segmentation, characterized in that Here are the steps: Step 1, data collection: collect SAR images and optical images containing the same scene targets and the same resolution as the training data set, classify the registered SAR images and optical images into the paired data set, and classify the unregistered SAR images and optical images into the unpaired data set. Then, create pixel-level binary classification semantic segmentation labels for the images in the data set, where the SAR images and optical images in the paired data set share the same semantic segmentation labels; Step 2, data preprocessing: preprocess the SAR images and optical images in the data set; Step 3, semantic segmentation model pre-training: using the pre-processed SAR image, optical image and its semantic segmentation label as input, the semantic segmentation model is trained, wherein the SAR image and the semantic segmentation label are used to train the SAR-semantic segmentation model, and the optical image and the semantic segmentation label are used to train the optical-semantic segmentation model; Step 4, build a conversion model driven by the semantic segmentation task: use a GAN network or a diffusion model as the basic framework of the conversion model, wherein for paired data sets, a supervised conversion model is used, and for unpaired data sets, an unsupervised conversion model is used. Two downstream task branches are added to the basic framework of the conversion model to obtain a conversion model driven by the semantic segmentation task, wherein one downstream task branch uses the SAR-semantic segmentation model to segment the SAR image input into the conversion model, and the segmentation result and the SAR image are input into the conversion model together; the other downstream task branch uses the optical-semantic segmentation model to segment the pseudo-optical image generated by the conversion and the real optical image respectively, and compares and evaluates the segmentation results, and feeds the evaluation results back to the conversion model; Step 5: Design the model loss function: The total model loss is obtained by adding the adversarial loss, supervision loss, and semantic segmentation loss. For the unsupervised conversion model, the cross entropy loss is used as the semantic segmentation loss. For the supervised conversion model, the cross entropy loss, feature matching loss, and semantic segmentation-guided L1 loss are added as the final semantic segmentation loss. Step 6, iterative training: using the image preprocessed in step 2 and its semantic segmentation label as input, iteratively train the network model constructed in steps 4-5. During each training, adjust the parameters of the conversion model generator according to the feedback signal of the conversion model discriminator until a conversion model with the best output result is obtained; Step 7, model application: the SAR image to be converted is input into the optimal conversion model obtained after training, and the generator of the conversion model outputs the corresponding converted optical image.

2. The SAR-optical image conversion method based on semantic segmentation drive according to claim 1, characterized in that: Acquire SAR and optical images from public datasets, satellite imagery databases, or your own collection.

3. The SAR-optical image conversion method based on semantic segmentation drive according to claim 1, characterized in that: The pixel-level binary classification semantic segmentation labels for the images in the data set are produced manually or by machine, wherein machine production refers to production using relevant software tools including EISeg.

4. The SAR-optical image conversion method based on semantic segmentation drive according to claim 1, characterized in that: The data preprocessing includes image scaling, cropping, and image enhancement operations.

5. The SAR-optical image conversion method based on semantic segmentation drive according to claim 1, characterized in that: The semantic segmentation model adopts the FCN model.

6. The SAR-optical image conversion method based on semantic segmentation drive according to claim 1, characterized in that: The adversarial loss adopts the least square loss, and the supervised loss adopts the L1 loss.

7. The SAR-optical image conversion method based on semantic segmentation drive according to claim 1, characterized in that: The specific calculation method of the cross entropy loss is: using the optical-semantic segmentation model to segment the optical image generated by the conversion model, and calculating the cross entropy loss of the segmentation result and the semantic segmentation label of the SAR image as the final cross entropy loss; The specific calculation method of the feature matching loss is: using the downsampling network of the optical-semantic segmentation model, downsampling the input optical image and the pseudo optical image generated by the conversion model respectively, calculating an L1 loss for the features obtained by downsampling at each level, and calculating the average of the L1 losses of all levels as the final feature matching loss; The specific calculation method of the semantic segmentation-guided enhanced L1 loss is: use the optical-semantic segmentation model to perform binary segmentation on the input optical image, multiply the segmentation result element by element with the input optical image and the pseudo optical image generated by the conversion model, calculate the L1 loss of the two images obtained after the multiplication, and multiply it by a weight as the final semantic segmentation-guided enhanced L1 loss, wherein the weight value is 10 to 20.

Images