Multi-task SAR image processing method based on cooperation of image conversion and semantic segmentation

Abstract

The invention provides a multitask SAR image processing method based on cooperation of image conversion and semantic segmentation. A multi-task collaborative network model composed of an image conversion module and a semantic segmentation module is constructed by utilizing a framework in which downstream tasks drive image conversion, and an image conversion result is supervised in real time through the downstream task of semantic segmentation. A conversion generation result obtained by using the model can retain more structure and texture information, and more powerful data support is provided for a subsequent remote sensing task.

Description

Technical Field

[0001] The present invention belongs to the technical fields of computer vision, image conversion and SAR image interpretation, and in particular relates to a multi-task SAR image processing method based on the collaboration of image conversion and semantic segmentation. Background Art

[0002] With the rapid development of imaging technology, Synthetic Aperture Radar (SAR) imaging has gradually become an important means of earth observation in the field of remote sensing. Different from traditional optical imaging technology, SAR adopts an active imaging method. SAR sensors actively emit microwaves and form SAR images based on the strength of echo signals. Thanks to the strong penetration of microwaves, SAR imaging technology overcomes weather interference and achieves all-day and all-weather imaging. However, SAR's unique imaging method also brings many problems. Problems such as speckle noise and geometric distortion seriously affect the quality of SAR images and make the image visibility poor. In order to overcome the above problems and give full play to the imaging advantages of SAR images all day, SAR image interpretation has become a long-lasting research hotspot in the field of remote sensing.

[0003] In recent years, inspired by deep learning image conversion technology, more and more scholars have tried to transfer image conversion algorithms in the field of computer vision to the field of SAR image interpretation. By learning the characteristics of SAR and optical images, a mapping from SAR to optical images is established, so that SAR images retain the advantages of all-day and all-weather imaging while having good visibility. Most of the existing SAR-optical image conversion technologies are direct transfers of technologies in the field of image conversion, including Pix2pix, a pixel-level image conversion model based on Generative Adversarial Network (GAN), which uses conditional generative adversarial networks as the model architecture and U-net as the feature extraction module of the generator, supplemented by pixel-level image loss functions to constrain the generated results and achieve high-quality image conversion; CycleGAN, an unsupervised image conversion model based on GAN, proposes cycle consistency loss to achieve unsupervised image style constraints. Thanks to the robustness of algorithms in the field of image conversion, the transferred SAR-optical image conversion technology can basically generate high-quality optical images, especially in terms of human visual perception, and has achieved good results.

[0004] However, the existing SAR-optical image conversion technology still has problems that cannot be ignored. The remote sensing field usually has higher requirements for the structure and texture information of the image. Remote sensing target detection, change detection and other tasks are usually more sensitive to the target of interest in the image, which requires the conversion model to have sufficient fine-grained conversion capabilities to better preserve the texture information of the ground objects, especially the targets of interest such as houses, airplanes, and ships. However, the existing SAR-optical image conversion technology, the conversion results of the target level are mostly blurred, and sometimes even the target is lost after the conversion. Remote sensing ground object classification, resource exploration and other tasks have high requirements for the structural information of the image. Taking the ground object classification task as an example, to obtain a good ground object segmentation result, it is necessary to require the image to have sufficient structural information, so that the segmentation model can extract sufficient edge features and achieve high-precision ground object classification. However, the existing SAR-optical image conversion technology is difficult to retain complete structural information. The converted image usually only conforms to the style of the optical image, but the ground object edges are blurred, and there are multiple ground objects entangled. The problem of information loss in the existing technology is directly caused by algorithm migration. As mentioned above, most of the existing technologies directly migrated the classic algorithms in the field of image conversion, such as Pix2pix and CycleGAN. The algorithms in the field of image conversion are oriented to natural images, which contain far less semantic information than remote sensing images. Therefore, the algorithms in this field usually only need to use pixel-level loss functions (such as L1 distance, mean square error, etc.) to better constrain the generation network and obtain good results. Since the pixel-level loss function does not pay attention to the overall structural information and semantic information of the image, if these algorithms are directly applied to SAR-optical image conversion, it is difficult to fully retain the rich structural and texture information in the SAR image during the conversion by relying only on the pixel-level loss function. Therefore, the conversion results under the existing technology are difficult to further provide data support for downstream remote sensing tasks. In summary, although the existing technology can present optical images with good visual effects, it is difficult to play a positive role in downstream tasks. The SAR-optical image conversion technology only stays at the visual level. This is also contrary to the original intention of SAR-optical image conversion. If the conversion only pursues the visual effect of the human eye but loses the key information in the SAR image, then the conversion is not of practical significance. Summary of the invention

[0005] In order to overcome the shortcomings of the prior art, the present invention provides a multi-task SAR image processing method based on image conversion and semantic segmentation collaboration. Using the framework of image conversion driven by downstream tasks, a multi-task collaborative network model consisting of an image conversion module and a semantic segmentation module is constructed. The image conversion results are supervised in real time through the downstream task of semantic segmentation, so that the conversion generation results obtained by the model can retain more structural and texture information, providing more powerful data support for subsequent remote sensing tasks.

[0006] A multi-task SAR image processing method based on image conversion and semantic segmentation collaboration is characterized by the following steps:

[0007] Step 1, data collection: collect SAR images, optical images and corresponding semantic label data containing the same scene targets, where the SAR images and optical images have the same resolution and have been registered one by one, and the semantic labels match the real SAR and optical images;

[0008] Step 2, data preprocessing: preprocess the collected SAR images and optical images;

[0009] Step 3, constructing a SAR-optical image conversion module: using a generative adversarial network (GAN) as the SAR-optical image conversion module;

[0010] Step 4, constructing a semantic segmentation module: using a semantic segmentation network model as the semantic segmentation module;

[0011] Step 5, multi-task collaborative framework design: let the generator of the SAR-optical image conversion module and the semantic segmentation module share the same downsampling module as their shared encoder, and then connect their respective decoders to form a multi-task collaborative network framework with parameter sharing;

[0012] Step 6, loss function design: For the SAR-optical image conversion module, the conversion loss function composed of GAN adversarial loss and similarity loss is adopted; for the semantic segmentation module, the semantic segmentation loss function composed of cross entropy loss is adopted; the weighted sum of conversion loss and semantic segmentation loss is used as the final loss function;

[0013] Step 7, iterative training: using the image preprocessed in step 2 and its semantic label as input, iteratively train the network model constructed in steps 3-6 until a network model with the best output result is obtained;

[0014] Step 8, model application: input the SAR image to be interpreted into the best network model obtained after training, wherein the SAR-optical image conversion module outputs the converted optical image, and the semantic segmentation module outputs the semantic segmentation result.

[0015] Specifically, SAR images and optical images are obtained from public data sets, satellite image databases or self-collection, and the corresponding semantic label data are obtained by manual or machine annotation.

[0016] Specifically, SAR images, optical images and corresponding semantic segmentation labels in the public dataset WHU-OPT-SAR are used for data collection.

[0017] Specifically, the data preprocessing includes image scaling, cropping, and image enhancement operations.

[0018] Specifically, the SAR-optical image conversion module uses the network model Pix2pix or CycleGAN in the field of image conversion as a baseline model.

[0019] Specifically, the SAR-optical image conversion module uses ResNet or U-net based on convolutional neural network as the generator of the generative adversarial network GAN, and uses PatchGAN or multi-scale discriminator as the discriminator of the generative adversarial network GAN.

[0020] Specifically, the semantic segmentation module adopts U-net, SegNet or FCN based on convolutional neural network.

[0021] Specifically, the workflow of the multi-task collaborative network is as follows: the SAR image is used as the input of the shared encoder, and high-dimensional features are obtained through encoding, which are used as the input of the decoders of the image conversion module and the semantic segmentation module respectively. Then, the high-dimensional features pass through the decoder of the image conversion module to obtain the generated optical image, which is then input into the discriminator of the image conversion module to form an adversarial loss, and the distance between the high-dimensional features and the real optical image is calculated to obtain the similarity loss; at the same time, the high-dimensional features pass through the decoder of the semantic segmentation module to obtain the predicted semantic label, and the distance between the high-dimensional features and the real semantic label is calculated to obtain the segmentation loss.

[0022] Specifically, the shared encoder mainly includes three convolution operations. First, the input image is reflected and filled once, and then the output after reflection filling is subjected to the first convolution operation, including three operations: convolution, batch normalization, and activation function. Then, the output after the first convolution is connected to two modules with the same first convolution operation, and finally the high-dimensional features of the input image are obtained.

[0023] Specifically, the formula of the final loss function, which is the weighted sum of the conversion loss and the semantic segmentation loss, is expressed as follows:

[0024] L total =λ1L Tran +λ2L seg (1)

[0025] Among them, L total represents the total loss, L Tran represents the conversion loss, L seg represents the semantic segmentation loss, λ1 is the conversion loss coefficient, and λ2 is the semantic segmentation loss coefficient, which are set to λ1=10 and λ2=30 respectively.

[0026] The beneficial effects of the present invention are as follows: by adopting an image conversion framework driven by downstream tasks, a multi-task collaborative network model consisting of an image conversion module and a semantic segmentation module is constructed, which can improve the network's ability to retain image structure and texture information, while providing semantic segmentation results, and effectively solving the problems in the prior art of fuzzy and lost target conversion results, blurred edges between different objects, and weak structural information of conversion results; since the optical image obtained by the present invention retains sufficient structural information and texture information, a more accurate SAR image object classification result can be obtained, which better supports further remote sensing tasks and can be used as an effective end-to-end SAR image multi-task interpretation method. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a schematic diagram of the overall framework of the multi-task collaborative network of the present invention;

[0028] Figure 2 It is a schematic diagram of the network structure of the shared encoder;

[0029] Figure 3 It is a schematic diagram of the network structure of the decoder of the image conversion module;

[0030] Figure 4 It is a schematic diagram of the network structure of the semantic segmentation module decoder. DETAILED DESCRIPTION

[0031] The present invention is further described below in conjunction with the accompanying drawings and embodiments. The present invention includes but is not limited to the following embodiments.

[0032] The present invention provides a multi-task SAR image processing method based on image conversion and semantic segmentation collaboration, and its implementation process is as follows:

[0033] 1. Data collection. In the technical solution of the present invention, it is necessary to include SAR images, optical images and semantic labels of semantic segmentation tasks of the same scene targets, and the resolution should be consistent, paired and registered. SAR and optical images can be obtained from public data sets, satellite image databases or self-collection, and the corresponding semantic label data can be obtained by manual or machine annotation. For example, the public data set WHU-OPT-SAR contains 100 pairs of fully registered SAR and optical images, as well as corresponding semantic segmentation labels.

[0034] In addition, to better support model training, the larger the data scale, the better. To ensure data diversity, you can choose images containing different scenes and different targets.

[0035] 2. Data preprocessing. Use common preprocessing methods to preprocess the collected SAR images and optical images to improve the robustness of the training model. Common preprocessing methods include image scaling, cropping, image enhancement and other operations, which can improve image quality to a certain extent.

[0036] 3. Construct a multi-task collaborative network. In order to improve the network's ability to retain image structure and texture information while providing semantic segmentation results, the present invention constructs a multi-task collaborative network model consisting of an image conversion module and a semantic segmentation module, such as Figure 1-4 As shown in Figure 2. The basic idea is to implement an image translation framework driven by downstream tasks.

[0037] Among them, the SAR-optical image conversion module is realized with the help of the Generative adversarial Network (GAN) in the field of deep learning image conversion. According to the actual situation, the current deep learning-based baseline network model in the field of image conversion, such as Pix2pix and CycleGAN, can be directly adopted. It is also possible to build a conversion module based on the GAN architecture, such as using ResNet and U-net based on convolutional neural network (CNN) as the generator of GAN to realize SAR-optical image conversion, and the discriminator uses PatchGAN or multi-scale discriminator to urge the generator to improve the conversion quality.

[0038] The present invention selects semantic segmentation tasks as downstream tasks to achieve driving. The semantic segmentation module can directly select classic semantic segmentation network models, such as U-net, SegNet, etc.

[0039] The key to the present invention lies in the specific implementation of the downstream task-driven image conversion framework. On the basis of this framework, a network model is formed in which image conversion and semantic segmentation can be trained collaboratively, thus realizing end-to-end multi-task SAR image interpretation. Therefore, for the generator of the image conversion module and the semantic segmentation module, they are made to share the same downsampling module to become a shared encoder, and then the respective upsampling modules, that is, the decoders of the image conversion module and the semantic segmentation module are connected, thereby forming a parameter sharing, so that during network training, the conversion task and the semantic segmentation task jointly optimize the downsampling module parameters to realize multi-task collaborative training. The details of the multi-task collaborative network will be introduced in detail below in conjunction with the accompanying drawings.

[0040] The workflow of the multi-task collaborative network is as follows: Figure 1As shown in the figure, the SAR image is used as the input of the shared encoder, and then encoded to obtain high-dimensional features, which are used as the input of the image conversion module and the semantic segmentation module decoder. First, the high-dimensional features pass through the image conversion module decoder to obtain the generated optical image, which is input into the GAN discriminator to form an adversarial loss, and the distance between it and the real optical image is calculated to obtain the similarity loss. Secondly, the high-dimensional features also pass through the decoder of the semantic segmentation module to obtain the predicted semantic label, and the distance between it and the real semantic label is calculated to obtain the segmentation loss.

[0041] The internal network structure of the shared encoder is as follows Figure 2 As shown in the figure, it mainly includes three convolution operations. First, the SAR image is filled with reflection (Reflect layer) so that the convolution layer can more fully extract the image edge information. Secondly, the output is subjected to the first convolution operation (conv layer) for feature extraction. It should be noted that the three operations of convolution (conv), batch normalization (Norm), and activation function (ReLU) are integrated into a standardized convolution operation, which is conducive to stabilizing network training and alleviating the gradient vanishing problem. After the first convolution, the output is connected to two similar convolution modules for further feature extraction, and finally the high-dimensional features of the SAR image are obtained.

[0042] The internal network structure of the image conversion module decoder is as follows Figure 3 As shown in the figure. The high-dimensional features obtained by the shared encoder are used as input and first pass through a residual network, which includes a reflection padding layer, a convolution layer, and 9 groups of residual blocks (Resnet Block). The residual block can ensure the normal propagation of the gradient while increasing the network depth, thereby enhancing the network feature extraction capability. After the residual network, it is connected to an upsampling module, which includes two groups of deconvolution operations (ConvTrans) to restore the features to the original size of the image. Finally, after a reflection padding layer, a convolution layer, and a Tanh activation function layer, the generated optical image is output.

[0043] The network structure of the semantic segmentation module decoder is as follows Figure 4 As shown in the figure, it mainly includes three convolution operations and two deconvolution operations. First, the high-dimensional features obtained by the shared encoder are used as input, and then a convolution operation is performed, and the output is output through the ReLU activation function. Then two convolution operations are performed in the order of convolution, batch normalization, and ReLU activation function. Finally, two deconvolution operations are performed to output the predicted semantic label.

[0044] For the above-mentioned image conversion and semantic segmentation modules, they correspond to conversion loss and segmentation loss respectively. Among them, the conversion loss constrains the training of the SAR-optical image conversion module, which consists of the adversarial loss of GAN and the loss function that measures the image similarity (such as L1 loss, perceptual loss). The specific formula is as follows:

[0045]

[0046] L Tran =L GAN +L pix (4)

[0047] Among them, formula (2) and formula (3) are adversarial loss and similarity loss respectively. In formula (2), min and max indicate that the generator and discriminator of GAN jointly optimize the loss function. During the network training process, the discriminator needs to maximize the adversarial loss, while the generator needs to minimize the adversarial loss. SAR 、DC tran , D represent shared encoder, image conversion decoder, and discriminator processing, respectively, L GAN represents adversarial loss, x~X represents SAR image samples that obey the real SAR image distribution, y~Y represents optical image samples that obey the real optical image distribution, and E(·) represents the expected value. Formula (3) is the similarity loss function used. In particular, L1 loss is selected here. pix Represents the similarity loss. The conversion loss is the sum of the adversarial loss and the similarity loss, as shown in formula (5).

[0048] The semantic segmentation loss of the semantic segmentation module adopts cross entropy loss. First, the predicted label is processed into a one-hot vector, and then the predicted probability score is normalized by the softmax function, as shown in formula (5), where z represents the generated multi-channel probability map, z i represents the value corresponding to the i-th pixel, and n represents the number of categories. Next, the cross entropy corresponding to each pixel value is calculated, and finally the average is taken to obtain L CE , L CE Denote the final cross entropy loss L seg , as shown in formula (6), formula (7) and formula (8), where l ij 、z ij denotes the label and prediction score of the jth category corresponding to the ith pixel, H and W denote the width and height of the input image, EC denotes the shared encoder, and DC seg Represents the semantic segmentation decoder.

[0049]

[0050] Combining the conversion loss and semantic segmentation loss, the final total loss function is as follows:

[0051] L total =λ1L Tran +λ2L seg (9)

[0052] Among them, L total represents the total loss, λ1 is the conversion loss coefficient, and λ2 is the semantic segmentation loss coefficient, which are set to λ1=10 and λ2=30 respectively.

[0053] 4. Iterative training. The image preprocessed in step 2 and its semantic label are used as input to iteratively train the network model constructed above. During the repeated iterative training, the parameters of the generator, discriminator and semantic segmentation network of the generative adversarial network are continuously optimized, and the quality of the generated optical image and the accuracy of semantic segmentation are improved. Until a set of optimal network weight parameters is determined to balance various losses, so that the network output results are optimal and the optimal network model is obtained.

[0054] 5. Model application. In practical applications, the SAR image to be interpreted is input into the trained multi-task collaborative network model for conversion, which can simultaneously obtain high-quality optical images and more accurate semantic segmentation results, that is, the SAR image to be interpreted is input into the best network model obtained after training, wherein the SAR-optical image conversion module outputs the converted optical image, which has good texture details, clear target conversion, and clear contours between different objects while maintaining good visual effects. Since the converted optical image retains sufficient structural and texture information, it is possible to further perform remote sensing processing on it. At the same time, the multi-task collaborative network model also provides a semantic segmentation result output, that is, the segmentation module segments the optical image output by the conversion module to obtain a high-precision segmentation result, realizing a simple object classification task and verifying the effect of the conversion result in the remote sensing object classification task.

Claims

1. A multi-task SAR image processing method based on image conversion and semantic segmentation collaboration, characterized in that Here are the steps: Step 1, data collection: collect SAR images, optical images and corresponding semantic label data containing the same scene targets, where the SAR images and optical images have the same resolution and have been registered one by one, and the semantic labels match the real SAR and optical images; Step 2, data preprocessing: preprocess the collected SAR images and optical images; Step 3, constructing a SAR-optical image conversion module: using a generative adversarial network (GAN) as the SAR-optical image conversion module; Step 4, constructing a semantic segmentation module: using a semantic segmentation network model as the semantic segmentation module; Step 5, multi-task collaborative framework design: let the generator of the SAR-optical image conversion module and the semantic segmentation module share the same downsampling module as their shared encoder, and then connect their respective decoders to form a multi-task collaborative network framework with parameter sharing; Step 6, loss function design: For the SAR-optical image conversion module, the conversion loss function composed of GAN adversarial loss and similarity loss is adopted; for the semantic segmentation module, the semantic segmentation loss function composed of cross entropy loss is adopted; the weighted sum of conversion loss and semantic segmentation loss is used as the final loss function; Step 7, iterative training: using the image preprocessed in step 2 and its semantic label as input, iteratively train the network model constructed in steps 3-6 until a network model with the best output result is obtained; Step 8, model application: input the SAR image to be interpreted into the best network model obtained after training, wherein the SAR-optical image conversion module outputs the converted optical image, and the semantic segmentation module outputs the semantic segmentation result.

2. The multi-task SAR image processing method based on image conversion and semantic segmentation collaboration as claimed in claim 1, characterized in that: SAR images and optical images are obtained from public data sets, satellite image databases or self-collection, and the corresponding semantic label data are obtained by manual or machine annotation.

3. The multi-task SAR image processing method based on image conversion and semantic segmentation collaboration as claimed in claim 1, characterized in that: The data were collected using SAR images, optical images and corresponding semantic segmentation labels from the public dataset WHU-OPT-SAR.

4. The multi-task SAR image processing method based on image conversion and semantic segmentation collaboration as claimed in claim 1, characterized in that: The data preprocessing includes image scaling, cropping, and image enhancement operations.

5. The multi-task SAR image processing method based on image conversion and semantic segmentation collaboration as claimed in claim 1, characterized in that: The SAR-optical image conversion module uses the network model Pix2pix or CycleGAN in the field of image conversion as a baseline model.

6. The multi-task SAR image processing method based on image conversion and semantic segmentation collaboration as claimed in claim 1, characterized in that: The SAR-optical image conversion module adopts ResNet or U-net based on convolutional neural network as the generator of the generative adversarial network GAN, and adopts PatchGAN or multi-scale discriminator as the discriminator of the generative adversarial network GAN.

7. The multi-task SAR image processing method based on image conversion and semantic segmentation collaboration as claimed in claim 1, characterized in that: The semantic segmentation module adopts U-net, SegNet or FCN based on convolutional neural network.

8. The multi-task SAR image processing method based on image conversion and semantic segmentation collaboration as claimed in claim 1, characterized in that: The workflow of the multi-task collaborative network is as follows: the SAR image is used as the input of the shared encoder, and high-dimensional features are obtained through encoding, which are used as the input of the image conversion module and the semantic segmentation module decoder respectively. Then, the high-dimensional features pass through the decoder of the image conversion module to obtain the generated optical image, which is then input into the discriminator of the image conversion module to form an adversarial loss, and the distance between the high-dimensional features and the real optical image is calculated to obtain the similarity loss; at the same time, the high-dimensional features pass through the decoder of the semantic segmentation module to obtain the predicted semantic label, and the distance between the high-dimensional features and the real semantic label is calculated to obtain the segmentation loss.

9. The multi-task SAR image processing method based on image conversion and semantic segmentation collaboration as claimed in claim 1, characterized in that: The shared encoder mainly includes three convolution operations. First, the input image is reflected and filled once, and then the output after reflection filling is subjected to the first convolution operation, which includes three operations: convolution, batch normalization, and activation function. Then, the output after the first convolution is connected to two modules with the same first convolution operation, and finally the high-dimensional features of the input image are obtained.

10. The multi-task SAR image processing method based on image conversion and semantic segmentation collaboration as claimed in claim 1, characterized in that: The formula of the final loss function, which is the weighted sum of the conversion loss and the semantic segmentation loss, is expressed as follows: L total =λ1L Tran +λ2L seg (1) Among them, L total represents the total loss, L Tran represents the conversion loss, L seg represents the semantic segmentation loss, λ1 is the conversion loss coefficient, and λ2 is the semantic segmentation loss coefficient, which are set to λ1=10 and λ2=30 respectively.

Images