Multi-task SAR image processing method based on image conversion and semantic segmentation collaboration

Abstract

This paper provides a multi-task SAR image processing method based on the collaboration of image conversion and semantic segmentation. Using a framework where image conversion is driven by downstream tasks, a multi-task collaborative network model consisting of an image conversion module and a semantic segmentation module is constructed. The downstream task of semantic segmentation provides real-time supervision of the image conversion results. This ensures that the conversion results generated using this model retain more structural and texture information, providing stronger data support for subsequent remote sensing tasks.

Description

Technical Field

[0001] The present invention belongs to the technical fields of computer vision, image conversion and SAR image interpretation, and particularly relates to a multi-task SAR image processing method based on the collaboration of image conversion and semantic segmentation. Background Art

[0002] With the rapid development of imaging technology, Synthetic Aperture Radar (SAR) imaging has gradually become a key tool for Earth observation in the remote sensing field. Unlike traditional optical imaging techniques, SAR uses an active imaging method. SAR sensors actively emit microwaves and form SAR images based on the strength of the echo signals. Thanks to the strong penetrating properties of microwaves, SAR imaging technology overcomes weather interference and enables all-day, all-weather imaging. However, SAR's unique imaging method also brings many challenges. Issues such as speckle noise and geometric distortion seriously affect SAR image quality, resulting in poor image visibility. To overcome these challenges and fully utilize the advantages of SAR imagery for all-day imaging, SAR image interpretation has become a long-standing research hotspot in the remote sensing field.

[0003] In recent years, inspired by deep learning image conversion techniques, a growing number of researchers have attempted to transfer image conversion algorithms from computer vision to SAR image interpretation. By learning the characteristics of SAR and optical images, they establish a mapping from SAR to optical images. This allows SAR images to retain the advantages of all-day, all-weather imaging while also maintaining good visibility. Existing SAR-to-optical image conversion techniques are mostly direct transfers of techniques from this field. These include the Pix2pix pixel-level image conversion model based on a generative adversarial network (GAN). This employs a conditional GAN ​​as its model architecture, a U-net as the generator's feature extraction module, and a pixel-level image loss function to constrain the generated results, achieving high-quality image conversion. CycleGAN, an unsupervised image conversion model also based on a GAN, proposes a cycle consistency loss to achieve unsupervised image style constraints. Thanks to the robustness of algorithms from the image conversion field, these transferred SAR-to-optical image conversion techniques can generally generate high-quality optical images, particularly with impressive results in terms of human visual perception.

[0004] However, existing SAR-to-optical image conversion techniques still present significant challenges. The remote sensing field typically places higher demands on image structure and texture information. Remote sensing tasks such as target detection and change detection are often more sensitive to objects of interest within an image. This requires the conversion model to possess sufficiently fine-grained conversion capabilities to effectively preserve the texture information of objects, particularly those of interest such as houses, airplanes, and ships. However, existing SAR-to-optical image conversion techniques often produce blurry object-level conversion results, sometimes even resulting in target loss. Remote sensing tasks such as object classification and resource exploration place high demands on image structural information. For example, achieving good object segmentation results requires sufficient image structural information, enabling the segmentation model to extract sufficient edge features and achieve high-precision object classification. However, existing SAR-to-optical image conversion techniques struggle to preserve complete structural information. The converted images often only conform to the style of optical images, but with blurred object edges and the appearance of multiple objects becoming entangled. This information loss in existing conversion techniques is a direct result of algorithm migration. As mentioned above, most existing technologies directly transfer classic algorithms from the field of image conversion, such as Pix2pix and CycleGAN. Image conversion algorithms are designed for natural images, which contain far less semantic information than remote sensing images. Therefore, these algorithms typically only need to employ pixel-level loss functions (such as L1 distance and mean squared error) to effectively constrain the generative network and achieve good results. Because pixel-level loss functions do not consider the overall structural and semantic information of the image, directly applying these algorithms to SAR-to-optical image conversion, relying solely on pixel-level loss functions, makes it difficult to fully preserve the rich structural and texture information in SAR images. Therefore, the conversion results produced by existing technologies are unlikely to provide data support for downstream remote sensing tasks. In summary, while existing technologies can produce optical images with good visual effects, they are unlikely to contribute positively to downstream tasks, leaving SAR-to-optical image conversion technology limited to the visual level. This also contradicts the original intention of SAR-to-optical image conversion. If the conversion only pursues visual effects similar to those of the human eye while losing key information in the SAR image, then the conversion is meaningless. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies, this paper provides a multi-task SAR image processing method based on the collaboration of image conversion and semantic segmentation. Leveraging a framework where image conversion is driven by downstream tasks, a multi-task collaborative network model consisting of an image conversion module and a semantic segmentation module is constructed. The downstream task of semantic segmentation provides real-time supervision of the image conversion results, enabling the generated conversions using this model to retain more structural and texture information, providing stronger data support for subsequent remote sensing tasks.

[0006] A multi-task SAR image processing method based on the collaboration of image conversion and semantic segmentation is characterized by the following steps:

[0007] Step 1, data collection: Collect SAR images, optical images, and corresponding semantic label data containing the same scene targets. The SAR images and optical images have the same resolution and are pairwise registered. The semantic labels match the real SAR and optical images.

[0008] Step 2, data preprocessing: preprocess the collected SAR images and optical images;

[0009] Step 3: Constructing a SAR-optical image conversion module: Using a generative adversarial network (GAN) as the SAR-optical image conversion module;

[0010] Step 4: Build a semantic segmentation module: Use the semantic segmentation network model as the semantic segmentation module;

[0011] Step 5: Design a multi-task collaborative framework: The generator of the SAR-optical image conversion module and the semantic segmentation module share the same downsampling module as their shared encoder. Then, their respective decoders are connected to form a parameter-sharing multi-task collaborative network framework.

[0012] Step 6, loss function design: For the SAR-optical image conversion module, a conversion loss function consisting of GAN adversarial loss and similarity loss is used; for the semantic segmentation module, a semantic segmentation loss function consisting of cross entropy loss is used; the final loss function is the weighted sum of the conversion loss and semantic segmentation loss;

[0013] Step 7, iterative training: using the image preprocessed in step 2 and its semantic label as input, iteratively train the network model constructed in steps 3-6 until the network model with the best output result is obtained;

[0014] Step 8, model application: The SAR image to be interpreted is input into the optimal network model obtained after training. The SAR-optical image conversion module outputs the converted optical image, and the semantic segmentation module outputs the semantic segmentation result.

[0015] Specifically, SAR images and optical images are obtained from public datasets, satellite image databases or self-collection, and the corresponding semantic label data are obtained by manual annotation or machine annotation.

[0016] Specifically, the data were collected using SAR images, optical images and corresponding semantic segmentation labels from the public dataset WHU-OPT-SAR.

[0017] Specifically, the data preprocessing includes image scaling, cropping, and image enhancement operations.

[0018] Specifically, the SAR-optical image conversion module adopts the network model Pix2pix or CycleGAN in the field of image conversion as a baseline model.

[0019] Specifically, the SAR-optical image conversion module adopts ResNet or U-net based on convolutional neural network as the generator of the generative adversarial network GAN, and adopts PatchGAN or multi-scale discriminator as the discriminator of the generative adversarial network GAN.

[0020] Specifically, the semantic segmentation module adopts U-net, SegNet or FCN based on convolutional neural network.

[0021] Specifically, the workflow of the multi-task collaborative network is as follows: the SAR image is used as the input of the shared encoder, and high-dimensional features are obtained through encoding, which are used as the input of the image conversion module and the semantic segmentation module decoder respectively. Then, the high-dimensional features pass through the decoder of the image conversion module to obtain the generated optical image, which is then input into the discriminator of the image conversion module to form an adversarial loss, and the distance between it and the real optical image is calculated to obtain the similarity loss; at the same time, the high-dimensional features pass through the decoder of the semantic segmentation module to obtain the predicted semantic label, and the distance between it and the real semantic label is calculated to obtain the segmentation loss.

[0022] Specifically, the shared encoder mainly includes three convolution operations. First, the input image is reflection-filled once, and then the output after reflection filling is subjected to the first convolution operation, which includes three operations: convolution, batch normalization, and activation function. Then, the output after the first convolution is connected to two modules with the same first convolution operation, and finally the high-dimensional features of the input image are obtained.

[0023] Specifically, the formula for the final loss function, which is the weighted sum of the conversion loss and the semantic segmentation loss, is expressed as follows:

[0024] L total =λ1L Tran +λ2L seg (1)

[0025] Among them, L total Represents the total loss, L Tran Indicates the conversion loss, L seg represents the semantic segmentation loss, λ1 is the conversion loss coefficient, and λ2 is the semantic segmentation loss coefficient, which are set to λ1 = 10 and λ2 = 30 respectively.

[0026] The beneficial effects of the present invention are as follows: due to the adoption of an image conversion framework driven by downstream tasks, a multi-task collaborative network model consisting of an image conversion module and a semantic segmentation module is constructed, which can improve the network's ability to retain image structure and texture information, while providing semantic segmentation results, and effectively solve the problems in the prior art of fuzzy and lost target conversion results, blurred edges between different objects, and weak structural information of conversion results; since the optical image obtained by the present invention retains sufficient structural information and texture information, more accurate SAR image object classification results can be obtained, which better supports further remote sensing tasks and can be used as an effective end-to-end SAR image multi-task interpretation method. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a schematic diagram of the overall framework of the multi-task collaborative network of the present invention;

[0028] Figure 2 This is a schematic diagram of the network structure of the shared encoder;

[0029] Figure 3 This is a schematic diagram of the network structure of the image conversion module decoder;

[0030] Figure 4 It is a schematic diagram of the network structure of the semantic segmentation module decoder. DETAILED DESCRIPTION

[0031] The present invention is further described below with reference to the accompanying drawings and examples. The present invention includes but is not limited to the following examples.

[0032] The present invention provides a multi-task SAR image processing method based on the collaboration of image conversion and semantic segmentation, the implementation process of which is as follows:

[0033] 1. Data collection. The technical solution of the present invention requires SAR images, optical images, and semantic labels for semantic segmentation tasks that contain the same scene targets, and the resolution should be consistent, and they should be paired and registered. SAR and optical images can be obtained from public datasets, satellite image databases, or collected by themselves. The corresponding semantic label data can be obtained by manual or machine annotation. For example, the public dataset WHU-OPT-SAR contains 100 pairs of fully registered SAR and optical images, as well as corresponding semantic segmentation labels.

[0034] In addition, to better support model training, the larger the data scale, the better. To ensure data diversity, you can choose images containing different scenes and different targets.

[0035] 2. Data preprocessing: Use common preprocessing methods to preprocess the acquired SAR and optical images to improve the robustness of the training model. Common preprocessing methods include image scaling, cropping, and image enhancement, which can improve image quality to a certain extent.

[0036] 3. Construct a multi-task collaborative network. In order to improve the network's ability to retain image structure and texture information while providing semantic segmentation results, the present invention constructs a multi-task collaborative network model consisting of an image conversion module and a semantic segmentation module, such as Figure 1-4 The basic idea is to implement an image conversion framework driven by downstream tasks.

[0037] The SAR-to-optical image conversion module utilizes a generative adversarial network (GAN), a deep learning-based image conversion framework. Depending on the application, current deep learning-based baseline network models in the image conversion field, such as Pix2pix and CycleGAN, can be directly adopted. Alternatively, a custom conversion module can be constructed based on the GAN architecture. For example, a convolutional neural network (CNN)-based ResNet or U-net can be used as the GAN generator to achieve SAR-to-optical image conversion, while a discriminator, such as PatchGAN or a multi-scale discriminator, can be used to encourage the generator to improve conversion quality.

[0038] The present invention uses semantic segmentation as a downstream task to achieve driving. The semantic segmentation module can directly use classic semantic segmentation network models such as U-net, SegNet, etc.

[0039] The key to the present invention lies in the specific implementation of the downstream task-driven image conversion framework. Based on this framework, a network model is formed in which image conversion and semantic segmentation can be trained collaboratively, thus realizing end-to-end multi-task SAR image interpretation. Therefore, for the generator of the image conversion module and the semantic segmentation module, they are made to share the same downsampling module to become a shared encoder, and then the respective upsampling modules, that is, the decoders of the image conversion module and the semantic segmentation module, are connected to form a parameter sharing, so that during network training, the conversion task and the semantic segmentation task jointly optimize the downsampling module parameters, realizing multi-task collaborative training. The details of the multi-task collaborative network will be introduced in detail below with reference to the accompanying drawings.

[0040] The workflow of the multi-task collaborative network is as follows: Figure 1As shown in the figure, the SAR image serves as input to the shared encoder, which is then encoded to generate high-dimensional features, which serve as input to the decoders of the image conversion module and semantic segmentation module, respectively. First, the high-dimensional features pass through the decoder of the image conversion module to obtain the generated optical image, which is then input into the GAN discriminator to form an adversarial loss. The distance between the generated optical image and the true optical image is calculated to obtain the similarity loss. Second, the high-dimensional features also pass through the decoder of the semantic segmentation module to obtain the predicted semantic label. The distance between the predicted semantic label and the true semantic label is calculated to obtain the segmentation loss.

[0041] The internal network structure of the shared encoder is as follows Figure 2 As shown in the figure, it mainly includes three convolution operations. First, the SAR image is filled with reflections (Reflect layer) to enable the convolution layer to more fully extract the image edge information. Second, the output undergoes the first convolution operation (conv layer) for feature extraction. It is important to note that the three operations here, convolution (conv), batch normalization (Norm), and activation function (ReLU), are integrated into a standardized convolution operation, which helps stabilize network training and alleviate the gradient vanishing problem. After the first convolution, the output is connected to two similar convolution modules for further feature extraction, ultimately obtaining the high-dimensional features of the SAR image.

[0042] The internal network structure of the image conversion module decoder is as follows Figure 3 As shown in the figure. The high-dimensional features obtained by the shared encoder are used as input and first pass through a residual network, which includes a reflection padding layer, a convolutional layer, and 9 groups of residual blocks (Resnet Block). The residual block can ensure the normal propagation of gradients while increasing the network depth, thereby enhancing the network's feature extraction capabilities. After the residual network, it is connected to an upsampling module, which includes two groups of deconvolution operations (ConvTrans) to restore the features to the original size of the image. Finally, after a reflection padding layer, a convolutional layer, and a Tanh activation function layer, the generated optical image is output.

[0043] The network structure of the semantic segmentation module decoder is as follows Figure 4 As shown in the figure, the network mainly consists of three convolution operations and two deconvolution operations. First, the high-dimensional features obtained by the shared encoder are used as input, and then a convolution operation is performed. The output is activated by the ReLU function. Then, two convolution operations are performed in the order of convolution, batch normalization, and ReLU activation. Finally, two deconvolution operations are performed to output the predicted semantic labels.

[0044] For the above-mentioned image conversion and semantic segmentation modules, they correspond to conversion loss and segmentation loss respectively. Among them, the conversion loss constrains the training of the SAR-optical image conversion module and consists of the GAN adversarial loss and the loss function that measures image similarity (such as L1 loss, perceptual loss). The specific formula is as follows:

[0045]

[0046] L Tran =L GAN +L pix (4)

[0047] Among them, formula (2) and formula (3) are adversarial loss and similarity loss respectively. In formula (2), min and max indicate that the generator and discriminator of GAN jointly optimize the loss function. During the network training process, the discriminator needs to maximize the adversarial loss, while the generator needs to minimize the adversarial loss. SAR 、DC tran , D represent shared encoder, image conversion decoder, and discriminator processing respectively, L GAN represents the adversarial loss, x~X represents SAR image samples that obey the real SAR image distribution, y~Y represents optical image samples that obey the real optical image distribution, and E(·) represents the expected value. Formula (3) is the similarity loss function used. In particular, L1 loss is selected here. pix Represents the similarity loss. The conversion loss is the sum of the adversarial loss and the similarity loss, as shown in formula (5).

[0048] The semantic segmentation loss of the semantic segmentation module adopts cross entropy loss. First, the predicted label is processed into a one-hot vector, and then the predicted probability score is normalized by the softmax function, as shown in formula (5), where z represents the generated multi-channel probability map, z i represents the value corresponding to the i-th pixel, and n represents the number of categories. Next, calculate the cross entropy corresponding to each pixel value, and finally take the average to get L CE , L CE Denote the final cross entropy loss L seg , as shown in formula (6), formula (7), and formula (8), where l ij 、z ij Represents the label and prediction score of the jth category corresponding to the i-th pixel, H and W represent the width and height of the input image, EC represents the shared encoder, DC seg Represents the semantic segmentation decoder.

[0049]

[0050] Combining the conversion loss and semantic segmentation loss, the final total loss function is as follows:

[0051] L total =λ1L Tran +λ2L seg (9)

[0052] Among them, L total represents the total loss, λ1 is the conversion loss coefficient, and λ2 is the semantic segmentation loss coefficient, which are set to λ1 = 10 and λ2 = 30 respectively.

[0053] 4. Iterative training. Using the image and its semantic label preprocessed in step 2 as input, the network model constructed above is iteratively trained. During this iterative training process, the parameters of the generator, discriminator, and semantic segmentation network of the generative adversarial network are continuously optimized, improving the quality of the generated optical image and the accuracy of semantic segmentation. This is done until an optimal set of network weight parameters is determined to balance the various losses, optimize the network output, and ultimately achieve the optimal network model.

[0054] 5. Model application. In practical applications, the SAR image to be interpreted is input into the trained multi-task collaborative network model for conversion, which can simultaneously obtain high-quality optical images and relatively accurate semantic segmentation results. That is, the SAR image to be interpreted is input into the best network model obtained after training, wherein the SAR-optical image conversion module outputs the converted optical image. The obtained optical image maintains good visual effects while also having good texture details, clear focus target conversion, and clear contours between different ground objects. Since the converted optical image retains sufficient structural and texture information, it makes further remote sensing processing possible. At the same time, the multi-task collaborative network model also provides a semantic segmentation result output, that is, the segmentation module segments the optical image output by the conversion module to obtain a high-precision segmentation result, realizing a simple ground object classification task and verifying the effect of the conversion result in the remote sensing ground object classification task.

Claims

1. A multi-task SAR image processing method based on image conversion and semantic segmentation, characterized by Here are the steps: Step 1, data collection: Collect SAR images, optical images, and corresponding semantic label data containing the same scene targets. The SAR images and optical images have the same resolution and are pairwise registered. The semantic labels match the real SAR and optical images. Step 2, data preprocessing: preprocess the collected SAR images and optical images; Step 3: Constructing a SAR-optical image conversion module: Using a generative adversarial network (GAN) as the SAR-optical image conversion module; Step 4: Build a semantic segmentation module: Use the semantic segmentation network model as the semantic segmentation module; Step 5: Design a multi-task collaborative framework: The generator of the SAR-optical image conversion module and the semantic segmentation module share the same downsampling module as their shared encoder. Then, their respective decoders are connected to form a parameter-sharing multi-task collaborative network framework. Step 6, loss function design: For the SAR-optical image conversion module, a conversion loss function consisting of GAN adversarial loss and similarity loss is used; for the semantic segmentation module, a semantic segmentation loss function consisting of cross entropy loss is used; the final loss function is the weighted sum of the conversion loss and semantic segmentation loss; Step 7, iterative training: using the image preprocessed in step 2 and its semantic label as input, iteratively train the network model constructed in steps 3-6 until the network model with the best output result is obtained; Step 8, model application: The SAR image to be interpreted is input into the optimal network model obtained after training. The SAR-optical image conversion module outputs the converted optical image, and the semantic segmentation module outputs the semantic segmentation result.

2. The multi-task SAR image processing method based on image conversion and semantic segmentation collaboration according to claim 1, characterized in that: SAR images and optical images are obtained from public datasets, satellite image databases or self-collection, and the corresponding semantic label data are obtained by manual or machine annotation.

3. The multi-task SAR image processing method based on image conversion and semantic segmentation collaboration according to claim 1, characterized in that: The data were collected using SAR images, optical images and corresponding semantic segmentation labels from the public dataset WHU-OPT-SAR.

4. The multi-task SAR image processing method based on image conversion and semantic segmentation collaboration according to claim 1, characterized in that: The data preprocessing includes image scaling, cropping, and image enhancement operations.

5. The multi-task SAR image processing method based on image conversion and semantic segmentation collaboration according to claim 1, characterized in that: The SAR-optical image conversion module uses the network model Pix2pix or CycleGAN in the field of image conversion as a baseline model.

6. The multi-task SAR image processing method based on image conversion and semantic segmentation collaboration according to claim 1, characterized in that: The SAR-optical image conversion module adopts ResNet or U-net based on convolutional neural network as the generator of the generative adversarial network (GAN), and adopts PatchGAN or multi-scale discriminator as the discriminator of the generative adversarial network (GAN).

7. The multi-task SAR image processing method based on image conversion and semantic segmentation collaboration according to claim 1, characterized in that: The semantic segmentation module adopts U-net, SegNet or FCN based on convolutional neural network.

8. The multi-task SAR image processing method based on image conversion and semantic segmentation collaboration according to claim 1, characterized in that: The workflow of the multi-task collaborative network is as follows: the SAR image is used as the input of the shared encoder, and high-dimensional features are obtained through encoding, which are used as the input of the image conversion module and the semantic segmentation module decoder respectively. Then, the high-dimensional features pass through the decoder of the image conversion module to obtain the generated optical image, which is then input into the discriminator of the image conversion module to form an adversarial loss, and the distance between it and the real optical image is calculated to obtain the similarity loss; at the same time, the high-dimensional features pass through the decoder of the semantic segmentation module to obtain the predicted semantic label, and the distance between it and the real semantic label is calculated to obtain the segmentation loss.

9. The multi-task SAR image processing method based on image conversion and semantic segmentation collaboration according to claim 1, characterized in that: The shared encoder mainly includes three convolution operations. First, the input image is reflected and padded once, and then the output after reflection and padding is subjected to the first convolution operation, which includes three operations: convolution, batch normalization, and activation function. Then, the output after the first convolution is connected to two modules with the same first convolution operation, and finally the high-dimensional features of the input image are obtained.

10. The multi-task SAR image processing method based on image conversion and semantic segmentation collaboration according to claim 1, characterized in that: The formula for the final loss function, which is the weighted sum of the conversion loss and the semantic segmentation loss, is expressed as follows: L total =λ1L Tran +λ2L seg (1) Among them, L total Represents the total loss, L Tran Indicates the conversion loss, L seg represents the semantic segmentation loss, λ1 is the conversion loss coefficient, and λ2 is the semantic segmentation loss coefficient, which are set to λ1 = 10 and λ2 = 30 respectively.

Images