A segmentation-guided self-supervised SAR image denoising method

Abstract

The application discloses a kind of segmentation guide self-supervised SAR image denoising methods, obtain the SAR image to be denoised, input pre-trained self-supervised denoising network, wherein self-supervised denoising network is trained by training data generated with noisy SAR image, and self-supervised denoising network training method is: training data input self-supervised denoising network, output denoising SAR image and input SAR image semantic segmentation network by denoising-segmentation connection network, output semantic segmentation label;Joint training loss of self-supervised denoising loss and semantic segmentation loss is constructed, and self-supervised denoising network, denoising-segmentation connection network using convolutional wavelet transform and SAR image semantic segmentation network are jointly trained, to obtain the self-supervised denoising network after training;Output denoising SAR image.The application realizes self-supervised SAR image denoising and simultaneously through perception semantic information and fully retains the feature of SAR image ground object details.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of remote sensing image processing, and relates to a segmentation-guided self-supervised SAR image denoising method, in particular to a self-supervised SAR image denoising method based on segmentation-guided semantic perception. BACKGROUND

[0002] Synthetic Aperture Radar (SAR) has the characteristics of all-weather and all-day observation of images, and can also penetrate rain, sand, cloud and other areas. These advantages make SAR an irreplaceable observation tool in many fields such as forest monitoring, urban planning, disaster assessment, etc. However, due to the coherence of the SAR imaging system, a kind of noise called speckle is inevitably generated in its imaging process, which makes the visual interpretation of SAR images very difficult. Therefore, denoising (i.e. speckle noise suppression) is an important basic step in SAR image processing, and has important research significance.

[0003] The rapid development of deep learning technology has also inspired many SAR image denoising methods based on deep learning. For example, in the Chinese invention patent (202011046838.6) "SAR image denoising method based on multi-scale residual attention network", on the one hand, the multi-scale convolution enriches the richness of the extracted image features, and on the other hand, the mask branch generates attention weights to learn the long-distance dependency between image pixels, so as to distinguish the features and noise of SAR images. For example, Wenfu Wu, Xiao Huang, et al. published a paper in the journal "Neurocomputing" entitled "SAR-DRDNet: A SAR image despeckling network with detail recovery", which designed a non-local processing module and a detail recovery module. The former suppresses speckle through the global information of the SAR image, and the latter further recovers the image details by considering the pixel's scale context information, so as to achieve a good balance between speckle suppression and texture detail preservation. These methods usually require "noisy-clean" SAR image pairs. However, under real observation conditions, it is impossible to obtain absolutely clean SAR images. Therefore, these methods usually use simulated data as the training set, which limits the performance of denoising on real SAR images.

[0004] To this end, in the past two years, there have been self-supervised learning-based SAR image denoising methods. For example, Huangxing Lin, Yihong Zhuang, et al. published a paper entitled "Self-Supervised SAR Despeckling Powered by Implicit Deep Denoiser Prior" in the journal "IEEE Geoscience and Remote Sensing Letters", which proposed an implicit deep denoising prior method for SAR image denoising, which takes the output of the convolutional neural network as the prior, and maximizes the similarity between the final denoised image and its prior image. These methods get rid of the restriction of clean SAR images as training labels. However, these methods usually only rely on the neighborhood information of image pixels, that is, the similarity and relevance of surrounding pixels. This makes these methods limited in perceiving and preserving the semantic features of SAR image observations.

[0005] Through domestic and foreign patent and literature retrieval, there are more studies on SAR image despeckling at present, but the existing SAR image denoising methods based on deep learning mainly rely on supervised learning, and some self-supervised learning methods also mainly rely on the neighborhood information of pixels, lack of semantic information perception, and there is no self-supervised SAR image denoising method using segmentation-guided semantic perception. SUMMARY

[0006] In view of the above prior art, the technical problem to be solved by the present application is to provide a segmentation-guided self-supervised SAR image denoising method, which realizes self-supervised SAR image denoising while fully preserving the ground feature details of the SAR image by perceiving semantic information.

[0007] To solve the above technical problems, a segmentation-guided self-supervised SAR image denoising method according to the present application comprises:

[0008] obtaining a SAR image to be denoised;

[0009] input the to-be-de-noised SAR image into a pre-trained self-supervised de-noising network, wherein the self-supervised de-noising network is trained by training data generated by a SAR image with noise, and a training method of the self-supervised de-noising network comprises the following steps: connecting, by the self-supervised de-noising network, a SAR image semantic segmentation network through a de-noising-segmentation connection network adopting convolutional wavelet transform, inputting training data into the self-supervised de-noising network, outputting a de-noised SAR image, inputting the de-noised SAR image into the SAR image semantic segmentation network through the de-noising-segmentation connection network adopting convolutional wavelet transform, and outputting a semantic segmentation label corresponding to the input SAR image; constructing a joint training loss composed of a self-supervised de-noising loss corresponding to the self-supervised de-noising network and a semantic segmentation loss corresponding to the SAR image semantic segmentation network, and performing end-to-end joint training on the self-supervised de-noising network, the de-noising-segmentation connection network adopting convolutional wavelet transform, and the SAR image semantic segmentation network to obtain a trained self-supervised de-noising network;

[0010] output the de-noised SAR image.

[0011] Further, the training data are:

[0012]

[0013] wherein y is the SAR image with noise, b is a binary matrix generated by a Bernoulli random distribution, and represents multiplication of corresponding elements between matrices. is input of the self-supervised de-noising network, is a label of the self-supervised de-noising network.

[0014] Further, the self-supervised de-noising network adopts a de-noising network based on a convolutional network.

[0015] Further, the de-noising-segmentation connection network adopting convolutional wavelet transform comprises, in sequence, a discrete wavelet transform (DWT) module, a Conv module, a ReLU module, and an inverse discrete wavelet transform (IDWT) module.

[0016] Further, the SAR image semantic segmentation network outputs a result satisfying:

[0017]

[0018] wherein F seg represents the SAR image semantic segmentation network, F de-seg represents the de-noising-segmentation connection network adopting convolutional wavelet transform, and F de represents the self-supervised de-noising network, y is the SAR image with noise, represents a segmentation result corresponding to y.

[0019] Further, the joint training loss is:

[0020] L total =L SSD +λL seg

[0021] wherein, L SSD denotes a self-supervised denoising loss, L seg denotes a semantic segmentation loss, and λ denotes a weight coefficient for controlling the semantic segmentation loss, and λ<1.

[0022] Further, the self-supervised denoising loss is:

[0023]

[0024] wherein, F de denotes a self-supervised denoising network, denotes an input of the self-supervised denoising network, corresponding output, and ||·||1 denotes an L1 norm. de denotes a parameter learnable by the self-supervised denoising network F de (·), and L SSD denotes the designed self-supervised denoising loss.

[0025] Further, the semantic segmentation loss is:

[0026]

[0027] wherein, θ de-seg and θ seg respectively denote a parameter learnable by the denoising-segmentation connected network F de-seg and the SAR image semantic segmentation network F seg adopting a convolutional wavelet transform, denotes a segmentation result corresponding to the SAR image with noise y, s denotes a semantic segmentation label corresponding to the SAR image with noise y, CrossEntropy denotes cross-entropy calculation, and L seg denotes the adopted semantic segmentation loss.

[0028] Further, the denoised SAR image satisfies:

[0029]

[0030] wherein, denotes a pre-trained self-supervised denoising network, is the finally obtained denoised SAR image.

[0031] Compared with the prior art, the present application has the following beneficial effects:

[0032] 1. Effective SAR image denoising can be achieved by using a self-supervised deep learning training method without requiring clean SAR images as training labels.

[0033] 2. Semantic information perception capability can be introduced, and semantic segmentation network can be used as a guide to achieve self-supervised denoising of SAR images while fully preserving the detailed features of ground objects in SAR images by perceiving semantic information in SAR images. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of the process of the self-supervised SAR image denoising method based on semantic awareness under segmentation guidance in this invention. Detailed Implementation

[0035] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0036] The objective of this invention is achieved as follows:

[0037] A self-supervised SAR image denoising method based on semantic awareness under segmentation guidance, such as Figure 1 As shown, the overall structure consists of three sub-networks, namely the front-end SAR image self-supervised denoising network F de Denoising-segmentation connection network F based on convolutional wavelet transform de-seg And the subsequent SAR image semantic segmentation network F seg F de It is responsible for performing self-supervised denoising on SAR images, while F seg Then, as a mentor, F de Provides semantic awareness guidance. And F de-seg Located in F de and F seg In the middle, it is responsible for connecting the two together. During training, the noisy SAR image is used as the F... de The input, F de The output is the denoised SAR image. F de The output is also F de-seg The input, F de-seg The output is F. seg Input. F seg The output is the semantic segmentation label corresponding to the input SAR image.

[0038] A self-supervised SAR image denoising method based on semantic awareness under segmentation guidance, such as Figure 1 As shown, its training process requires two loss functions, one applied to F. deThe self-supervised denoising (SSD) loss on the output. SSD and its effect on F seg semantic segmentation loss L on the output seg .

[0039] This invention discloses a self-supervised SAR image denoising method based on semantic awareness guided by segmentation. The main implementation steps are as follows:

[0040] Step 1: Build a self-supervised denoising network for front-end SAR images;

[0041] Step 2: Construct a denoising-segmentation connection network based on convolutional wavelet transform;

[0042] Step 3: Construct a semantic segmentation network for post-SAR images;

[0043] Step 4: Construct a joint training loss and perform end-to-end overall training on the pre-network, connection network, and post-network;

[0044] Step 5: Use the trained denoising network to perform denoising tests on noisy SAR images.

[0045] Step 1: Construct a pre-existing SAR image self-supervised denoising network; denote the original noisy SAR image as y. Generate a binarized (0 or 1) matrix, denoted as b, using a Bernoulli random distribution, and then generate training data for self-supervised denoising, which can be formally represented as:

[0046]

[0047] Where ⊙ denotes the element-wise multiplication of matrices, the calculated result This is the input to the network. This refers to the network's labels. With such training data, a denoising convolutional neural network can be trained. Specifically, this is achieved using a self-supervised denoising loss, which can be formally expressed as:

[0048]

[0049] Among them, F de This represents the trained denoising convolutional neural network. Indicates will The input value is the output obtained by this network, ||·||1 represents the L1 norm, and θ de This represents the trained denoising convolutional neural network F. de (·) Learnable parameters, L SSD This represents the designed self-supervised denoising loss.

[0050] The step is for the denoising convolutional neural network F de There is no special requirement, the mainstream convolution-based denoising network in the field of image denoising can be used, and some more typical network can be selected, such as UNet, DnCNN and the like.

[0051] The step two is to construct a denoising-segmentation connection network F based on convolutional wavelet transform de-seg The connection network is composed of the following parts, which can be formally represented as:

[0052] F de-seg DWT→Conv→ReLU→IDWT (3)

[0053] Wherein, DWT and IDWT represent discrete wavelet transform (Discrete Wavelet Transform, DWT) and inverse discrete wavelet transform (Inverse Discrete Wavelet Transform, IDWT) respectively, Conv represents convolution (Convolution, Conv) operation, ReLU represents rectified linear unit (Rectified Linear Unit, ReLU), F de-seg represents the designed denoising-segmentation connection network based on convolutional wavelet transform.

[0054] The role of the connection network in this step is to guide the semantic information provided by the post SAR image semantic segmentation network to be effectively transmitted to the pre SAR image supervised denoising network. Through the discrete wavelet transform DWT, the image can be decomposed into different frequency bands. Through the inverse discrete wavelet transform IDWT, the different frequency bands can be converted back to the image. Based on this effect, the semantic information provided by the post network can guide in the frequency domain. In addition, we expect that the semantic information guidance can have different degrees of emphasis on different frequency bands decomposed by DWT. Further, we expect that the degree of emphasis can be automatically learned by network training. Therefore, we add a convolution operation with ReLU between DWT and IDWT. Thus, when the post network guides the pre network semantically, it can automatically transmit different degrees of semantic information to different frequency bands in the frequency domain.

[0055] The step three is to build a post SAR image semantic segmentation network; the original SAR image y with noise is input into the self-supervised denoising network F de , and the output result is processed by the connection network F de-seg described in step two, and then input into the semantic segmentation network F seg , to obtain the segmentation result corresponding to y, which can be formally represented as:

[0056]

[0057] Among them, F seg F represents a semantic segmentation network. seg (F de-seg (F de (y))) means that the original noisy SAR image y is processed sequentially through F de F de-seg and F seg The result obtained from the processing is denoted as A semantic segmentation network can be trained using cross-entropy calculation, and the loss used for training can be formally expressed as:

[0058]

[0059] Where, θ de-seg and θ seg They represent the connection networks F respectively. de-seg and semantic segmentation network F seg The learnable parameters are s, which represents the semantic segmentation label corresponding to the original noisy SAR image y, and CrossEntropy, which represents the cross-entropy calculation. seg This represents the semantic segmentation loss used.

[0060] This step is for semantic segmentation network F seg There are no special requirements; any mainstream segmentation network in the field of image semantic segmentation is acceptable. Some typical networks that can be used include PSPNet and DeepLabV3.

[0061] Step four: Construct a joint training loss to perform end-to-end training on the pre-network, connection network, and post-network; combine the self-supervised denoising loss and semantic segmentation loss mentioned above into a single overall loss, and perform end-to-end joint training on the three networks (pre-network, connection network, and post-network). The overall loss can be formally expressed as:

[0062]

[0063] Where λ represents the weighting coefficient used to control the segmentation loss, L total This is the designed overall loss. The network is trained using this overall loss, and the training process is complete when the loss stabilizes.

[0064] The overall loss described in this step includes two parts: self-supervised denoising loss L. SSD and semantic segmentation loss L seg L SSD This represents the network's fundamental task objective: self-supervised denoising. L segThis represents the network's additional task objective, namely, semantic awareness. Therefore, L seg The weighting coefficient should be less than L. SSD The weight coefficient λ. There are no specific requirements for setting the weight coefficient λ in this step; a value can be selected within the range of [0.01, 0.1], depending on the chosen self-supervised denoising network F. de and semantic segmentation network F seg Subjective adjustments are made based on the different network structures.

[0065] Step five: Use the trained denoising network to test the denoising of the noisy SAR image; the SAR image denoising process can be formally represented as:

[0066]

[0067] in, This represents the self-supervised denoising network obtained after training. This is the final denoised SAR image.

Claims

1. A segmentation-guided self-supervised SAR image denoising method, characterized in that, The method comprises: acquiring a to-be-de-noised SAR image; inputting the to-be-de-noised SAR image into a pre-trained self-supervised de-noising network, wherein the self-supervised de-noising network is trained by training data generated by a SAR image with noise, and a training method of the self-supervised de-noising network comprises: connecting, by the self-supervised de-noising network, a SAR image semantic segmentation network through a de-noising-segmentation connection network adopting a convolutional wavelet transform, inputting training data into the self-supervised de-noising network, outputting a de-noised SAR image, inputting the de-noised SAR image into the SAR image semantic segmentation network through the de-noising-segmentation connection network adopting the convolutional wavelet transform, and outputting a semantic segmentation label corresponding to the input SAR image; constructing a joint training loss composed of a self-supervised de-noising loss corresponding to the self-supervised de-noising network and a semantic segmentation loss corresponding to the SAR image semantic segmentation network, and performing end-to-end joint training on the self-supervised de-noising network, the de-noising-segmentation connection network adopting the convolutional wavelet transform, and the SAR image semantic segmentation network to obtain the trained self-supervised de-noising network; outputting the de-noised SAR image; The denoising-segmentation connected network using the convolutional wavelet transform comprises a discrete wavelet transform DWT module, a denoising module, a Conv module, a deconvolution module, a module, a Conv module, a module, a Conv module, a module, and an inverse discrete wavelet transform IDWT module.

2. The segmentation-guided self-supervised SAR image denoising method according to claim 1, characterized in that: the training data are: wherein, is the SAR image with noise, is a binarization matrix generated using a Bernoulli random distribution, denotes multiplication of corresponding elements between matrices, is the input to the self-supervised denoising network, is the label for the self-supervised denoising network.

3. The segmentation-guided self-supervised SAR image denoising method according to claim 1, characterized in that: the self-supervised de-noising network adopts a de-noising network based on a convolutional network.

4. The segmentation-guided self-supervised SAR image denoising method according to claim 1, characterized in that: the SAR image semantic segmentation network outputs a result satisfying: wherein, denotes a SAR image semantic segmentation network, denotes a denoising-segmentation connected network with convolutional wavelet transform, denotes a self-supervised denoising network, is a SAR image with noise, denotes a ground truth segmentation map, denotes a corresponding segmentation result.

5. The segmentation-guided self-supervised SAR image denoising method according to claim 1, characterized in that: the joint training loss is: wherein, denotes a self-supervised denoising loss, denotes a semantic segmentation loss, denotes a weight coefficient for controlling the semantic segmentation loss, .

6. The segmentation-guided self-supervised SAR image denoising method according to claim 2, characterized in that: the self-supervised de-noising loss is: wherein, denotes a self-supervised denoising network, denotes an input to the self-supervised denoising network a corresponding output, denotes an LI norm, denotes a self-supervised denoising network learnable parameters, denotes a designed self-supervised denoising loss.

7. The segmentation-guided self-supervised SAR image denoising method according to claim 6, characterized in that: the semantic segmentation loss is: wherein, and respectively represent a denoising-segmentation connection network using convolutional wavelet transform and a SAR image semantic segmentation network learnable parameters in the middle, denotes a SAR image with noise corresponding segmentation results, denotes a SAR image with noise corresponding semantic segmentation labels, denotes a cross-entropy calculation, denotes a semantic segmentation loss adopted.

8. The segmentation-guided self-supervised SAR image denoising method according to claim 2, characterized in that: the de-noised SAR image satisfies: wherein, denotes a pre-trained self-supervised denoising network, is the final obtained denoised SAR image.

Images