A remote sensing image bidirectional semantic segmentation method based on domain adaptation and super-resolution

Abstract

The disclosure discloses a remote sensing image bidirectional semantic segmentation method based on domain adaptation and super-resolution, comprising the following steps: collecting a high-resolution target domain remote sensing image to be segmented and a low-resolution source domain remote sensing image with a segmentation label; downsampling the target domain remote sensing image; inputting the source domain remote sensing image and the downsampled target domain remote sensing image into a bidirectional semantic segmentation model composed of a super-resolution image translation module and a domain adaptation semantic segmentation module, and outputting a segmentation result of the image to be segmented.

Description

TECHNICAL FIELD

[0001] The present disclosure belongs to the technical field of remote sensing image segmentation, and particularly relates to a remote sensing image bidirectional semantic segmentation method based on domain adaptation and super resolution. BACKGROUND

[0002] Deep neural networks trained by large data sets with pixel-level labels have promoted the development of remote sensing image semantic segmentation, but the labeling work of pixel-level labels requires a lot of time and manpower. Unsupervised domain adaptation (UDA) is one of the effective methods to solve the problem of insufficient labels, which reduces the feature difference between the source domain and the target domain, and migrates the semantic segmentation model learned from the source domain with labels to the target domain without labels. However, remote sensing images from different sources have different spatial resolutions, and are affected by different imaging areas, imaging conditions and imaging times, and often have different spatial resolutions and feature distributions. Most of the existing unsupervised domain adaptation methods are proposed for ordinary optical images, and since remote sensing images come from different sources, their spatial resolutions differ, and are affected by different imaging areas, imaging conditions and imaging times, they have large differences in spectral features. Therefore, directly applying unsupervised domain adaptation methods to remote sensing images often has poor results.

[0003] The unsupervised domain adaptation method for remote sensing images not only reduces the feature difference between the source domain and the target domain, but also solves the problem of different spatial resolutions. In the existing method, the spatial resolution of the source domain and the target domain can be unified by using a simple interpolation method, and then the feature distribution of the source domain data and the target data in the feature space is aligned to eliminate the domain difference. Although this method eliminates the image resolution difference to some extent, it is not perfect in processing detail information such as texture features, and the reconstruction result is biased towards smoothness.

[0004] In addition, the super resolution method based on convolutional neural network can greatly improve the super resolution effect, but since the source domain and the target domain of remote sensing images from different sources often have obvious visual style differences, unlike the general super resolution method which only needs to reconstruct the detail information of the low resolution source domain image itself, the super resolution method applied in the domain adaptation field needs to learn the features of the high resolution target domain image, so as to better realize the migration application of the model.

[0005] In recent years, significant progress has been made in combining image translation models based on generative adversarial networks (GANs) with deep domain adaptation methods. These methods use style translation to train semantic segmentation models using synthetic source-domain images with the style of the target domain. Adversarial domain adaptation methods are then used to further address the feature differences between real images (the target domain) and synthetic images (the source domain). However, there is currently no research combining super-resolution methods with these image translation methods. Summary of the Invention

[0006] In response to the shortcomings of the existing technology, the purpose of the present invention is to provide a bidirectional semantic segmentation method for remote sensing images based on domain adaptation and super-resolution. This method combines deep semantic segmentation models and domain adaptation technology to solve the semantic segmentation transfer learning task from low-resolution source domain data to high-resolution target domain data.

[0007] To achieve the above objectives, the present disclosure provides the following technical solutions:

[0008] A bidirectional semantic segmentation method for remote sensing images based on domain adaptation and super-resolution includes the following steps:

[0009] S100: Acquire a high-spatial-resolution target-domain remote sensing image to be segmented and a low-spatial-resolution source-domain remote sensing image with segmentation labels;

[0010] S200: downsampling the target domain remote sensing image;

[0011] S300: Inputting the source domain remote sensing image and the downsampled target domain remote sensing image into a bidirectional semantic segmentation model composed of a super-resolution image translation module and a domain adaptation semantic segmentation module, and outputting the segmentation result of the image to be segmented.

[0012] Preferably, the super-resolution image translation module includes:

[0013] Feature extraction network, used to extract features from source domain remote sensing images and downsampled target domain remote sensing images to obtain low-resolution feature maps;

[0014] High-resolution image generation network, used to restore low-resolution feature maps to high-resolution feature maps;

[0015] The pixel-level domain discriminator is trained through generative adversarial training with a high-resolution image generation network to generate high-resolution synthetic images in the source domain with the style of the target domain from high-resolution feature maps.

[0016] Preferably, the high-resolution image generation network includes: two sub-pixel convolution layers and an upsampling layer, and each sub-pixel convolution layer is connected to a ReLU activation function.

[0017] Preferably, the pixel-level domain discriminator includes: 4 two-dimensional convolution layers with a convolution kernel size of 4×4.

[0018] Preferably, the domain adaptation semantic segmentation module includes:

[0019] The semantic segmentation decoder network is used to output the low-resolution feature map extracted by the feature extraction network into the segmentation prediction result map of the source domain and target domain remote sensing images through two-dimensional convolution operation and softmax operation;

[0020] The output space-level discriminator is used to perform adversarial domain adaptation learning on the segmentation prediction result map output by the semantic segmentation decoder network.

[0021] Preferably, the semantic segmentation decoder network includes: two two-dimensional convolutional layers with 3x3 convolution kernels, one two-dimensional convolutional layer with 9x9 convolution kernel and stride 2, and a SoftMax classification layer.

[0022] Preferably, the output space-level discriminator includes: 5 two-dimensional convolutional layers with a convolution kernel size of 4×4, each convolutional layer is connected to a LeakyReLU activation function.

[0023] Preferably, the training process of the bidirectional semantic segmentation model includes:

[0024] S1. Pre-train the super-resolution image translation module. When the super-resolution image translation module converges and the loss is minimized and stable, the training is completed.

[0025] S2. Pre-train the domain adaptation semantic segmentation module based on the pre-trained super-resolution image translation module. When the domain adaptation semantic segmentation module converges and the loss is minimized and stable, the training is completed;

[0026] S3, based on the pre-trained domain adaptation semantic segmentation module, generates pseudo labels A for the target domain training set images T * ;

[0027] S4. Generate a correction label A for the source domain synthetic image through the synthetic image segmentation result and source domain label generated by the pre-trained semantic segmentation module. S ';

[0028] S5. Based on pseudo label A T * and correction label A S ' Calculate the cross entropy loss l for the segmentation results of the source domain synthetic image and the target domain image respectively cel ;

[0029] S6. Repeat steps S3 to S5 until the bidirectional semantic segmentation model converges. If it does not converge, adjust the parameters or increase the number of training times until it converges.

[0030] Preferably, the domain adaptation semantic segmentation module is pre-trained through a bidirectional learning method based on the pre-trained super-resolution image translation module.

[0031] Preferably, the two-way learning method includes positive promotion learning and negative promotion learning.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] (1) This paper designs a super-resolution image translation module for remote sensing image domain adaptation, which generates high-resolution synthetic images with the style of the target domain from low-resolution source domain images. The synthetic images with more complete detail information and easier transfer learning and the corrected synthetic image labels are then used to train the semantic segmentation module to address the problem that the semantic segmentation model trained on low spatial resolution data in the source domain cannot be well transferred to high-resolution target domain data.

[0034] (2) A bidirectional semantic segmentation method BSSM-SRDA combining super-resolution and domain adaptation is proposed. In the forward direction, FA-Loss uses high-resolution images with more detailed information to improve the performance of semantic segmentation; in the reverse direction, a new perceptual loss is proposed to correct the errors introduced by synthetic images through the label correction method, thereby promoting further learning of the super-resolution image translation module.

[0035] (3) A new bidirectional learning algorithm is designed for BSSM-SRDA by combining self-supervised learning SSL, so that the super-resolution image translation module and the domain adaptation semantic segmentation module can better incentivize each other to learn. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a flowchart of a method for bidirectional semantic segmentation of remote sensing images based on domain adaptation and super-resolution proposed in the present disclosure;

[0037] Figure 2 It is a structural diagram of the bidirectional semantic segmentation model;

[0038] Figure 3 is the source domain remote sensing image input to the super-resolution image translation module;

[0039] Figure 4 is the downsampled target domain remote sensing image that is input to the super-resolution image translation module;

[0040] Figure 5 It is the target domain high-resolution image output by the high-resolution image generation network;

[0041] Figure 6 is the high-resolution source domain synthetic image output by the super-resolution image translation module;

[0042] Figure 7 It is the source domain segmentation prediction result map;

[0043] Figure 8 It is the target domain segmentation prediction result map;

[0044] Figure 9 is the high-resolution target domain test set remote sensing image to be segmented;

[0045] Figure 10 is the label of the high-resolution target domain test set remote sensing image to be segmented;

[0046] Figure 11 It is the segmentation result obtained by AdaptSegNet;

[0047] Figure 12 is the segmentation result obtained using FCAN;

[0048] Figure 13 is the segmentation result obtained using BDL;

[0049] Figure 14 is the segmentation result obtained by SRDA;

[0050] Figure 15 is the segmentation result obtained using ScaleDA;

[0051] Figure 16 is the segmentation result obtained by the method described in this disclosure. DETAILED DESCRIPTION

[0052] The following will refer to the attached Figures 1 to 16 Specific embodiments of the present disclosure are described in detail. Although specific embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. Instead, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0053] It should be noted that certain words are used in the specification and claims to refer to specific components. Those skilled in the art should understand that technicians may use different nouns to refer to the same component. This specification and claims do not use the difference in nouns as a way to distinguish components, but use the difference in the functions of the components as the criterion for distinction. As mentioned throughout the specification and claims, "including" or "comprising" is an open term, so it should be interpreted as "including but not limited to". The subsequent description of the specification is a preferred embodiment of the present disclosure, but the description is based on the general principles of the specification and is not used to limit the scope of the present disclosure. The scope of protection of the present disclosure shall be as defined by the attached claims.

[0054] To facilitate understanding of the embodiments of the present disclosure, further explanation will be given below using specific embodiments as examples in conjunction with the accompanying drawings, and the accompanying drawings do not constitute a limitation on the embodiments of the present disclosure.

[0055] In one embodiment, if Figure 1 As shown, the present disclosure provides a method for bidirectional semantic segmentation of remote sensing images based on domain adaptation and super-resolution, comprising the following steps:

[0056] S100: Acquire a high-spatial-resolution target-domain remote sensing image to be segmented and a low-spatial-resolution source-domain remote sensing image with segmentation labels;

[0057] S200: Calculate the multiple relationship between the spatial resolutions of the source domain and the target domain remote sensing images to obtain a multiple value (since remote sensing datasets all provide the spatial resolution of the data, the multiple relationship between the spatial resolutions of the source domain and the target domain datasets can be obtained by simple division), use the multiple value as a downsampling factor, and downsample the target domain remote sensing image so that the spatial resolution of the target domain remote sensing image is the same as that of the source domain remote sensing image;

[0058] S300: Input the source domain remote sensing image and the downsampled target domain remote sensing image into Figure 2 The bidirectional semantic segmentation model shown, which is composed of a super-resolution image translation module and a domain adaptation semantic segmentation module, outputs the segmentation result of the remote sensing image to be segmented.

[0059] The method described in this embodiment combines a deep semantic segmentation model with domain adaptation technology to solve the task of semantic segmentation transfer learning from low-resolution source domain data to high-resolution target domain data. Furthermore, a bidirectional semantic segmentation model is constructed, consisting of a super-resolution image translation module and a domain adaptation semantic segmentation module. The former integrates super-resolution methods into image translation, while the latter combines adversarial domain adaptation methods to further narrow domain differences at the output level, thereby reducing the spatial resolution and feature distribution differences between the source and target domains. This allows the model trained on the source domain to better complete the semantic segmentation task of target domain images with different sources and resolutions.

[0060] In another embodiment, the super-resolution image translation module includes:

[0061] Feature extraction network, using void space convolution pooling pyramid network to extract source domain remote sensing images (such as Figure 3 ) and the downsampled target domain remote sensing image (as shown in Figure 4 (As shown in the figure, dilated convolutions with different sampling rates are used in parallel.) This captures image context at multiple scales, resulting in a larger receptive field than standard convolutions and better captures segmentation targets of varying sizes. The feature extraction network outputs a low-resolution feature map with 256 channels and a resolution half that of the source and downsampled target remote sensing images.

[0062] The high-resolution image generation network is used to obtain a channel number r by first performing two-dimensional convolution on the low-resolution feature map output by the feature extraction network. 2 The low-resolution feature map is then filled with information from different channels to obtain a high-resolution image magnified r times (such as Figure 5 As shown), where r is the upsampling factor (a high-resolution feature map can also be obtained by bilinear interpolation, which is achieved by calculating the weighted average of the four surrounding pixels and filling the blank pixels after the resolution is expanded).

[0063] The pixel-level domain discriminator generates a high-resolution source domain synthetic image with the target domain style (such as Figure 6 As shown). During training, the target domain high-resolution image is marked as a true sample (denoted as 1), and the generated source domain synthetic image is marked as a fake sample (denoted as 0). The parameters of the high-resolution image generation network are fixed unchanged, and the pixel-level discriminator is trained first, that is, given a generated image, the discriminator is made to determine whether it is true or false. The synthetic image generated at the beginning of model training can be easily identified as a fake sample by the discriminator, that is, the loss of the discriminator approaches 1. Therefore, after completing the training of the discriminator, the parameters of the pixel-level discriminator are fixed unchanged, and the high-resolution image generation network is trained (its parameters are updated according to the loss function of the network) to generate synthetic images with the characteristics of true samples. After completing the training of the generation network, its parameters will be fixed again to train the discriminator. The above process is repeated until the discriminator can no longer distinguish between true and false samples (at this time the loss of the discriminator will stabilize at around 0.5).

[0064] It should be noted that the pixel-level domain discriminator determines whether different regions of the synthesized image come from the source domain or the target domain. Through iterative training of the generator and the discriminator, when the discriminator cannot distinguish the domain class of the region, it means that the generated high-resolution synthesized image in the source domain has learned the style of the target domain.

[0065] In this embodiment, the feature extraction network includes three 2D convolutional layers with 3x3 kernels, a max pooling layer, and a dilated spatial convolutional pooling pyramid structure. Each 2D convolutional layer is followed by a LeakyReLU layer. The dilated spatial convolutional pooling pyramid structure consists of a 1x1 2D convolutional layer and three parallel 2D convolutional layers with dilation and fill rates of 1, 4, and 8, respectively. The feature maps obtained by these four convolution layers are concatenated and then passed through a 1x1 2D convolutional layer to obtain the final low-resolution feature map.

[0066] The two-dimensional convolution formula described in the high-resolution image generation network is as follows:

[0067]

[0068] Among them, X 2d is the input, Y 2d is the output, (x+h,y+w) is the position in the input data, and (x,y) is the position of the corresponding output data. Represents the two-dimensional convolution kernel K 2d The value at position (h,w), H and W represent the two-dimensional convolution kernel K 2d The height and width of , and h∈H,0≤h <H,w∈W,0≤w<W。

[0069] In another embodiment, the high-resolution image generation network includes: two sub-pixel convolution layers and an upsampling layer, and each sub-pixel convolution layer is connected to a ReLU activation function.

[0070] In another embodiment, the pixel-level domain discriminator includes: 4 two-dimensional convolution layers with a convolution kernel size of 4×4.

[0071] In this embodiment, the input is mapped into a matrix through four layers of two-dimensional convolution operations. Each value in the matrix represents the probability that a certain area in the image is a true sample. The values ​​in the final output matrix are averaged to obtain the final output of the pixel-level domain discriminator.

[0072] In another embodiment, the domain adaptation semantic segmentation module includes:

[0073] The semantic segmentation decoder network is used to output the low-resolution feature map extracted by the feature extraction network into a source domain segmentation prediction result map (such as Figure 7As shown) and the target domain segmentation prediction result diagram (as shown Figure 8 As shown), the resolution of the segmentation prediction result image is the same as the high spatial resolution target domain remote sensing image;

[0074] The output space-level discriminator is used to perform adversarial domain adaptation learning on the segmentation prediction result map output by the semantic segmentation decoder network to reduce the inter-domain feature differences between the source domain and the target domain.

[0075] It should be noted that when the output spatial-level discriminator cannot distinguish whether each pixel of the segmentation prediction result comes from the source domain or the target domain, it means that the segmentation network can extract the domain-invariant features of the source domain and the target domain.

[0076] In this embodiment, the two-dimensional convolution operation in the semantic segmentation decoder network is expressed as follows:

[0077]

[0078] Among them, X 2d Indicates input, Y 2d represents the output, (x+h,y+w) represents the position in the input data, and (x,y) represents the position of the corresponding output data. Represents the two-dimensional convolution kernel K 2d The value at position (h,w), H and W represent the two-dimensional convolution kernel K 2d The height and width of , and h∈H,0≤h <H,w∈W,0≤w<W。

[0079] The SoftMax function expression is as follows:

[0080] SoftMax(x i )=p(y=c|x i )

[0081]

[0082] Among them, x i Represents the input feature pixel, p(y=c|x i ) represents the probability that the pixel belongs to the cth class.

[0083] In another embodiment, the semantic segmentation decoder network includes: two two-dimensional convolutional layers with 3x3 convolution kernels, one two-dimensional convolutional layer with 9x9 convolution kernels and a stride of 2, and a SoftMax classification layer.

[0084] In another embodiment, the output spatial level discriminator includes: 5 two-dimensional convolutional layers with a convolution kernel size of 4×4, and each convolutional layer is connected to a LeakyReLU activation function.

[0085] In another embodiment, the training process of the bidirectional semantic segmentation model includes:

[0086] S1. Pre-train the super-resolution image translation module R. When the super-resolution image translation module converges and the loss is minimized and stable, the training is completed.

[0087] In this step, the input of the super-resolution image translation module R is the source domain image I S and the target domain image downsampled to the source domain resolution↓I T , the output is the source domain synthetic image I S ′, calculates the mean square error loss l widely used in the super-resolution field for the target domain with high and low resolution image pairs mse , calculate the perceptual loss l for the low-resolution original image and high-resolution synthetic image in the source domain per_nor , the loss l of the super-resolution image translation module R and the loss l of the pixel-level discriminator PDD For: l R =l mse (R(↓I T ),I T )+l per_nor (I S ′,↑I S )

[0088] Among them, D pdd is the pixel-level discriminator, p data 9*) represents the distribution of data, represents the expected value of the distribution function, log(D pdd (I T )) represents the probability that the discriminator judges the target domain data as true, log91-D pdd (I S ′)) represents the probability that the discriminator judges the synthesized image as fake, ↓ represents the downsampling operation, and ↑ represents the upsampling operation. The generator hopes that the probability of the synthesized image being judged as fake is as low as possible, and the probability of being judged as real is as high as possible, that is, to maximize the loss function l PDD , minimize the loss function l R .

[0089] S2. Pre-train the domain adaptation semantic segmentation module S based on the pre-trained super-resolution image translation module. When the domain adaptation semantic segmentation module converges and the loss is minimized and stable, the training is completed;

[0090] In this step, the input of the domain adaptation semantic segmentation module S is the source domain image I S , source domain synthetic image I S ′, source domain segmentation label A Sand target domain image I T , the output is the target domain segmentation prediction result. Before performing self-supervised learning on the target domain to obtain pseudo labels, the loss l of the domain adaptation semantic segmentation module S is the cross entropy loss l between the source domain original image and the synthesized image cel And the feature similarity loss l that plays a positive role in promoting learning fa , the loss function is:

[0091] l S =l cel (S(I S ),↑A S )+l cel (S(↓I S ′),↑A S )+l fa

[0092] Output space-level domain discriminator loss l ODD The function is:

[0093]

[0094] Among them, D odd It is the output space-level domain discriminator. First, the segmentation prediction results of the source domain and target domain images output by the segmentation module are used as the input of the discriminator. Then, the domain class (source domain or target domain) of the input sample is judged. If the sample comes from the target domain, z = 0; if the sample comes from the source domain, z = 1; h and w represent the height and width of the input image respectively.

[0095] S3. Based on step S2, generate pseudo labels for the target domain training set images through the pre-trained semantic segmentation module. Use the maximum probability threshold method to select pixels with higher confidence in the target domain segmentation prediction probability as pseudo labels A. T * .

[0096] S4. Based on step S2, the synthetic image segmentation result and source domain label generated by the pre-trained domain adaptation semantic segmentation module are used to generate the correction label A for the source domain synthetic image. S ′.

[0097] S5, the target domain pseudo label A generated by combining steps S3 and S4 T * And the source domain synthetic image correction label A S ′, calculate the cross entropy loss l for the segmentation results of the synthetic image and the target domain image respectively cel , to further train the bidirectional semantic segmentation model. At the same time, a new perceptual loss l is calculated for the synthetic image segmentation results and source domain labels. per_new, which plays a role in reversely promoting the learning of the super-resolution image translation module. At this time, the final loss function of the bidirectional semantic segmentation model is:

[0098]

[0099] Similar to the standard generative adversarial network, the ultimate goal of the bidirectional semantic segmentation model is to minimize the loss l of the domain adaptation semantic segmentation module S S and the loss l of the super-resolution image translation module R R , while maximizing the probability that the target domain is identified as the source domain in the discriminator (including the pixel-level discriminator PDD and the output space-level discriminator ODD). S for:

[0100] l S =l cel (S(I S ),↑A S )+e cel (S(↓I S ′),A S ′)+l cel (S(T ssl ),A T * )+l fa

[0101] l R for:

[0102] l R =l mse (R(↓I T ),I T )+l per_nor (I S ′,↑I S +l per_new (S(↓I S ′),↑A S )

[0103] S6. Repeat steps S3 to S5 until the bidirectional semantic segmentation model converges, that is, the total loss value of the model training is reduced to the minimum, and the gradient of the loss curve is 0 or close to 0 in the next period of time (the lowest loss value can be judged by the curve of the loss function listed in step S5. When the slope of the curve gradually slows down and its gradient value is close to 0 or is already 0, it is considered that the loss function has reached a minimum point). If it does not converge, adjust the parameters or increase the number of training times until convergence.

[0104] In another embodiment, the domain adaptation semantic segmentation module is pre-trained through a bidirectional learning method based on the pre-trained super-resolution image translation module.

[0105] In this embodiment, the feature semantic information extracted by the deep layer of the domain adaptation semantic segmentation module is rich in details but loses information, while the deep layer of the super-resolution image translation module can restore more detailed structural information. The synthesized image generated by the super-resolution image translation module based on the generative adversarial network inevitably introduces noise, and the super-resolution image translation module itself cannot provide much constraint information for its training process, but the domain adaptation semantic segmentation module can determine whether there is a content inconsistency phenomenon according to the segmentation result of the synthesized image and the source domain label image. Based on the above two points, the embodiment designs forward promotion learning and reverse promotion learning for the domain adaptation semantic segmentation module and the super-resolution image translation module.

[0106] wherein the forward promotion learning helps to train the semantic segmentation module by calculating the similarity of each pair of pixels on the feature maps of the first layer of the super-resolution image translation module and the first layer of the semantic segmentation decoder as a feature similarity loss to enhance the detailed structural information of the feature maps of the semantic segmentation decoder, and the specific steps include:

[0107] Step 1, taking the feature maps of the first layer of the super-resolution image translation module and the first layer of the domain adaptation semantic segmentation decoder, performing feature dimension transformation through a 1x1 convolution layer, a BatchNorm layer and a ReLU layer.

[0108] Step 2, the similarity matrix C of each feature map F is calculated ij , C ij represents the relationship between the i-th and j-th pixels on the feature map, and the calculation formula is as follows:

[0109]

[0110] Step 3, the purpose of the feature similarity loss L fa is to learn the distance between the similarity matrices of the two feature maps, and the calculation formula is:

[0111]

[0112] wherein W'xH' represents the spatial dimension, represents the similarity matrix of the feature map of the semantic segmentation module, represents the similarity matrix of the feature map of the super-resolution module.

[0113] The reverse promotion learning is based on the visual content of the source domain synthesized image generated by the generative adversarial network, which may not match the source domain label. In order to avoid interference to the training of the segmentation model, the reverse promotion learning includes a label correction strategy for the synthesized image and a new perception loss to constrain the further learning of the image translation module, and the specific steps include:

[0114] Step 1, taking the synthesized image IS ′ is sent to the domain adaptation semantic segmentation module to obtain the segmentation result, and the high-confidence pixels in the segmentation result are selected by the maximum probability threshold method as the pseudo label A of the synthetic image S *.

[0115] Step 2: Use the label correction strategy to change the source domain label A S Pixel labels with medium and low confidence are replaced with pseudo labels A S *The high confidence label in the image is used to achieve the purpose of label correction. Define the label A of the corrected composite image S ′={A S ' (i,j)}(1≤i≤H,1≤j≤W) is:

[0116]

[0117] Among them, P(*) is the probability map predicted by the segmentation model, k and k* represent A S (i,j) and A S *(i,j) The category index in , δ is the correction rate, H and W represent the height and width of the label map respectively.

[0118] In summary, the loss function of the domain adaptation semantic segmentation module is expressed as:

[0119] l S =l cel (S(I S ),↑A S )+l cel (S(↓I S ′),A S ′)+l cel (S(T ssl ),A T * )+l fa

[0120] The loss function of the super-resolution image translation module is expressed as:

[0121] l R =l mse (R(↓I T ),I T )+l per_nor (I S ′,↑I S )+l per_new 9S(↓I S ′),↑A S )

[0122] Step 3: Calculate the perceptual loss l between the segmentation result map of the synthetic image and the source domain pixel-level label mapper_new , which in turn further constrains the training of the super-resolution image translation module by minimizing the perceptual loss.

[0123] In this embodiment, first, the pre-trained super-resolution image translation module provides a well-translated synthetic image for the subsequent training of the semantic segmentation model. Then, the domain adaptation semantic segmentation module is trained based on the pre-trained super-resolution image translation module, and the domain adaptation semantic segmentation module is trained based on the pre-trained super-resolution image translation module. S The feature similarity loss FA-Loss is added to promote the learning of the domain adaptation semantic segmentation module using the well-trained R. Finally, the self-supervised learning is realized in an iterative way, and the bidirectional semantic segmentation model is trained N times. S Add the self-supervisory loss of the target domain to l R A new perceptual loss is added to the CNN. After each pseudo-label generation for the target domain self-supervised training, the source domain synthetic images are corrected. During the testing phase, the target domain test set data is fed into the trained bidirectional semantic segmentation model to obtain the final segmentation prediction results.

[0124] Below, the present disclosure describes the above scheme in detail through specific embodiments.

[0125] 1. Prepare the dataset

[0126] (1) Vaih-Pots: Two remote sensing datasets, Vaihingen and Potsdam, with six categories provided by ISPRS, are used for multi-class domain adaptation semantic segmentation experiments. The Vaihingen dataset, consisting of 33 images with a spatial resolution of 9 cm, is regarded as the source domain, and the Potsdam dataset, consisting of 38 images with a spatial resolution of 5 cm, is regarded as the target domain. In the experiment, all images of the source domain Vaihingen are used as the training set, and 19 images of the target domain Potsdam with labels from 2-10 to 5-12 are divided into the training set, and the other 19 images are used as the test set. Since the spatial resolution difference between the source domain and the target domain data is about 2 times, the Vaihingen dataset is cropped to 180×180 pixels and the Potsdam dataset is cropped to 360×360 pixels.

[0127] (2) Tibetan Plateau Dataset TPDS-TPDT: This self-produced Tibetan Plateau dataset uses remote sensing data from the Tibetan Plateau region captured by different sensors in 1990 and 2021. The source domain TPDS consists of two 8192×8192 pixel remote sensing images with a spatial resolution of 90m; the target domain TPD_T consists of a 7651×7781 pixel remote sensing image with a spatial resolution of 30m. We manually annotated the labels of this dataset, which are divided into four categories: vegetation, bare land, water bodies, and others. Since the spatial resolution difference between the source and target domains is three times, the source domain is cropped to 86×86 pixels and the target domain is cropped to 256×256 pixels.

[0128] 2. Experimental Setup

[0129] Experimented on NVIDIA GeForce RTX with 24GB of memory TM The algorithm is implemented on a 3090 GPU and the deep learning framework PyTorch. The Adam optimizer is used with a momentum of 0.9. In the actual training process, all samples are trained 20 times when pre-training R, and all samples are trained 30 times when pre-training BSSM-SRDA0. BSSM-SRDA is trained in a loop. i All samples will be trained 30 times, and the learning rate is set to 2×10 in the first half of each training round. 4 , and the learning rate in the second half decreases gradually until it reaches 0. In the experiment, mIoU and Macro-F1 suitable for multi-classification tasks are used as evaluation indicators.

[0130] 3. Experimental results and analysis

[0131] 3.1 Hyperparameter Experiments

[0132] This paper selects the hyperparameters mentioned in this method through experiments on the Vaih-Pots dataset, including the number of self-supervised learning iterations N, the confidence threshold ε for selecting pseudo labels, the number of iterations of the label correction strategy LCS, and the correction rate δ.

[0133] (1) Self-supervised learning iteration number N and confidence threshold ε

[0134] When the number of self-supervised learning iterations N = 1, the mIoU results obtained by training BSSM-SRDA1 using pseudo-labels at different confidence thresholds ε are shown in Table 1. When the threshold ε is lower than 0.9, some incorrect predictions may become pseudo-labels, interfering with the training of the semantic segmentation model. When the threshold ε is 0.95, the number of pseudo-label pixels that can be used is small, and the training effect of BSSM-SRDA1 is reduced. Therefore, in subsequent experiments, this paper sets the confidence threshold ε for selecting pseudo-labels to 0.9.

[0135] Table 1 Effects of different thresholds ε

[0136]

[0137] As the number of self-supervised learning iterations N increases, the proposed bidirectional semantic segmentation method for remote sensing images based on domain adaptation and super-resolution (BSSM-SRDA) assigns pseudo labels to more high-confidence pixels in the target domain to increase the number of training samples in the target domain, so that BSSM-SRDA can better transfer learning to the target domain data. Once there is a pseudo-labeled target domain data set T ssl When the confidence threshold ε stops increasing, it indicates that BSSM-SRDA has converged. When the confidence threshold ε is 0.9, the segmentation results of BSSM-SRDA (without label correction) on the Vaih-Pots dataset are shown in Table 2 as the number of iterations N increases. It can be seen that as N increases, BSSM-SRDA's performance improves, and the number of white pixels in the generated pseudo-labels decreases. When N = 3, the mIoU value increases very slowly, by only 0.1%. Considering that increasing N will increase the computational cost, the number of iterations N is set to 2 in subsequent experiments.

[0138] Table 2 The influence of the number of iterations N

[0139]

[0140] (2) Correction rate δ and number of iterations of label correction strategy

[0141] BSSM-SRDA uses a label correction strategy to determine the labels of synthesized images to avoid inconsistencies between the synthesized images and the original source domain labels, which could interfere with the learning of the semantic segmentation module. The correction rate δ is used to select pixels in the original source domain labels that need to be corrected. Table 3 shows the mIoU accuracy of BSSM-SRDA1 under different correction rates δ. It can be seen that δ is a relatively insensitive hyperparameter and is set to 0.3 in subsequent experiments in this paper.

[0142] Table 3 Effect of correction rate δ

[0143]

[0144] Table 4 shows that after the first round of iterative training, the label correction strategy improved the mIoU of BSSM-SRDA1 by 0.9%. After the second round of training, it improved the mIoU of BSSM-SRDA2 by 0.6%. After the third round of training, it only improved the mIoU of BSSM-SRDA3 by 0.1%. Experiments show that BSSM-SRDA has converged after the second round of training. This conclusion is consistent with the experimental results of the number of iterations N in self-supervised learning. Therefore, in subsequent experiments, the number of iterations of the label correction strategy is set to the same as N, which is 2.

[0145] Table 4. The impact of the number of iterations of the label correction strategy

[0146]

[0147] 3.2 Comparison with State-of-the-Art Methods on Two Datasets

[0148] In the field of adversarial domain adaptation semantic segmentation, this paper selects five comparable state-of-the-art methods including AdapSegNet, FCAN, BDL, SRDA and ScaleDA for comparative experiments: Table 5 shows the methods used in each comparative experiment. In terms of image translation, FCAN and BDL train the image translation network and the domain adaptation semantic segmentation network independently, while SRDA and BSSM-SRDA integrate the two into one network. BSSM-SRDA maintains the visual consistency of image translation through a new perceptual loss. In terms of self-supervised learning, BDL's self-supervised learning is only for the semantic segmentation method of the target domain. BSSM-SRDA combines self-supervised learning to design a bidirectional learning algorithm, and incorporates the label correction strategy of the source domain synthetic image into the self-supervised process to promote the bidirectional learning of semantic segmentation and image translation modules. In terms of the difference in spatial resolution between domains, SRDA uses simple bilinear interpolation in the model to restore the source domain to the target domain resolution size. ScaleDA reduces the difference in segmentation target scale caused by the spatial resolution difference by adding an inter-domain scale discriminator. BSSM-SRDA combines the super-resolution method into the image translation module to restore the resolution of the source domain image to the target domain size.

[0149] Table 5 Methods used in BSSM-SRDA and five comparative experiments

[0150]

[0151] (1)Vaih-Pots

[0152] Table 6 shows the comparison results between the latest methods and BSSM-SRDA on the Vaih-Pots remote sensing dataset. It can be seen that BSSM-SRDA achieves the highest performance (BSSM-SRDA's mIoU is 48.8% and Macro-F1 is 64.1%). As shown in Table 6, FCAN (mIoU is 42.0%), which adds an image translation model, achieves a 6.9% improvement in mIoU over AdapSegNet (mIoU is 35.1%). BDL (mIoU is 44.9%), which combines self-supervised learning to alternately train image translation models and semantic segmentation models, achieves a 2.9% improvement over FCAN. Compared with the previous best SRDA (mIoU is 46.0%), our BSSM-SRDA also combines image translation with semantic segmentation networks, but improves mIoU by 2.8% by combining super-resolution and bidirectional learning algorithms. ScaleDA (mIoU is 43.4%), which does not do image translation work, designed a scale discriminator and scale attention to address the spatial resolution difference of remote sensing images. It improved the mIoU by 8.3% compared to AdapSegNet, indicating that it is very necessary to solve the spatial resolution difference in the field of remote sensing image domain adaptation. For example, Figure 9 Given a high-resolution target domain test set remote sensing image to be segmented, Figure 10 Given the labels of the high-resolution target domain test set remote sensing images to be segmented, Figures 11 to 16 The segmentation results of BSSM-SRDA and five comparison methods are shown respectively. Figures 11 to 16 It can be observed that the segmentation results of BSSM-SRDA are better than those of AdaptSegnet, FCAN, BDL, SRDA and ScaleDA in both structure and details.

[0153] Table 6 Comparison results between the comparison method and BSSM-SRDA on the Vaih-Pots dataset

[0154]

[0155] (2) Tibetan Plateau Dataset TPDS-TPDT

[0156] Table 7 shows the comparison results between the latest methods and BSSM-SRDA on the TPDS-TPDT remote sensing dataset. BSSM-SRDA achieved the best results in all categories and comprehensive evaluation indicators, with an mIoU of 61.5% and a Macro-F1 of 73.1%, which is 3.2% higher than the previous best SRDA in mIoU and 3.9% higher in Macro-F1.

[0157] Table 7 Comparison results between the contrast method and TPDS-TPDT on the Vaih-Pots dataset

[0158]

[0159] Compared to the Vaih-Pots dataset, the TPDS and TPDT datasets exhibit more distinct visual style and feature differences. For example, the source domain TPDS data contains numerous small lakes and fewer vegetation types, while the target domain TPDT data contains almost no small lakes and more vegetation types. Because BSSM-SRDA incorporates super-resolution methods, the generated images have higher resolution and richer detail. Its bidirectional boost learning algorithm further improves the performance of semantic segmentation models.

[0160] The present disclosure has been introduced in detail above. Specific examples are used herein to illustrate the principles and implementation methods of the present disclosure. The description of the above embodiments is only used to help understand the method and core ideas of the present disclosure. At the same time, for those skilled in the art, according to the ideas of the present disclosure, there may be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present disclosure.

Claims

1. A bidirectional semantic segmentation method for remote sensing images based on domain adaptation and super-resolution, comprising the following steps: S100: Acquire a high-spatial-resolution target-domain remote sensing image to be segmented and a low-spatial-resolution source-domain remote sensing image with segmentation labels; S200: downsampling the target domain remote sensing image; S300: Inputting the source domain remote sensing image and the downsampled target domain remote sensing image into a bidirectional semantic segmentation model composed of a super-resolution image translation module and a domain adaptation semantic segmentation module, and outputting the segmentation result of the remote sensing image to be segmented; in, The super-resolution image translation module includes: Feature extraction network, used to extract features from source domain remote sensing images and downsampled target domain remote sensing images to obtain low-resolution feature maps; High-resolution image generation network, used to restore low-resolution feature maps to high-resolution feature maps; The pixel-level domain discriminator is trained against the high-resolution image generation network to generate high-resolution synthetic images in the source domain with the style of the target domain from high-resolution feature maps. The domain adaptation semantic segmentation module includes: The semantic segmentation decoder network is used to output the low-resolution feature map extracted by the feature extraction network into the segmentation prediction result map of the source domain and target domain remote sensing images through two-dimensional convolution operation and softmax operation; Output space-level discriminator for adversarial domain adaptation learning of the segmentation prediction result map output by the semantic segmentation decoder network; The training process of the bidirectional semantic segmentation model includes: S1. Pre-train the super-resolution image translation module. When the super-resolution image translation module converges and the loss is minimized and stable, the training is completed. S2. Pre-train the domain adaptation semantic segmentation module based on the pre-trained super-resolution image translation module. When the domain adaptation semantic segmentation module converges and the loss is minimized and stable, the training is completed; S3, based on the pre-trained domain adaptation semantic segmentation module, generates pseudo labels A for the target domain training set images T * ; S4. Generate a correction label A for the source domain synthetic image through the synthetic image segmentation result and source domain label generated by the pre-trained domain adaptation semantic segmentation module S '; S5. Based on pseudo label A T * and correction label A S ' Calculate the cross entropy loss for the segmentation results of the source domain synthetic image and the target domain image respectively S6. Repeat steps S3 to S5 until the bidirectional semantic segmentation model converges. If it does not converge, adjust the parameters or increase the number of training times until it converges.

2. The method according to claim 1, wherein The high-resolution image generation network includes: two sub-pixel convolution layers and an upsampling layer, and each sub-pixel convolution layer is connected to a ReLU activation function.

3. The method according to claim 1, wherein The pixel-level domain discriminator includes: 4 two-dimensional convolution layers with a convolution kernel size of 4×4.

4. The method according to claim 1, wherein The semantic segmentation decoder network includes: two two-dimensional convolutional layers with 3x3 convolution kernels, one two-dimensional convolutional layer with 9x9 convolution kernels and a stride of 2, and a SoftMax classification layer.

5. The method according to claim 1, wherein The output spatial level discriminator includes: 5 two-dimensional convolutional layers with a convolution kernel size of 4×4, and each convolutional layer is connected to a LeakyReLU activation function.

6. The method according to claim 1, wherein The domain adaptation semantic segmentation module is pre-trained based on the pre-trained super-resolution image translation module through a bidirectional learning method.

7. The method according to claim 6, wherein: The two-way learning method includes positive promotion learning and reverse promotion learning.

Images