Remote sensing image unsupervised domain adaptive semantic segmentation method based on two stages

Abstract

The invention discloses a remote sensing image unsupervised domain adaptive semantic segmentation method based on two stages. The method comprises the steps of preprocessing a data set, constructing a style conversion network, training the style conversion network, testing the style conversion network, constructing an unsupervised domain adaptive semantic segmentation network, training the unsupervised domain adaptive semantic segmentation network, testing the unsupervised domain adaptive semantic segmentation network and evaluating an output segmentation result graph. According to the method, the two-stage training steps of the style conversion network and the style conversion network are adopted, the style conversion network is used for generating a source domain image similar to a target domain in style, the style difference between the source domain and the target domain is reduced, learning of local context information and boundary information is enhanced, the balance of segmentation precision and calculation performance is achieved, and the segmentation accuracy is improved. The generated remote sensing image is consistent in content and clear in detail, the segmentation performance of the label-free target domain remote sensing image is improved, and the method is suitable for unsupervised domain adaptive semantic segmentation of the remote sensing image.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and particularly relates to unsupervised domain adaptation semantic segmentation of remote sensing images. Background Art

[0002] With the booming development of unmanned aerial vehicles and remote sensing technologies, a large number of high-resolution remote sensing images are widely used in fields such as urban planning, mineral exploration, and vegetation detection, providing data support for social development. Semantic segmentation aims to provide semantic category labels for each pixel in an image and is an important and challenging task in the remote sensing field.

[0003] Semantic segmentation often requires a large number of pixel-level labeled images, but pixel-level manual annotation of images requires a lot of time. In addition, due to differences in sensors used for shooting, shooting time, shooting area, etc., there are large inter-domain differences between remote sensing images from different sources, and it is difficult for a semantic segmentation model learned from labeled images to be effectively generalized to the semantic segmentation technology of unlabeled images from different sources.

[0004] Unsupervised domain adaptation can effectively alleviate the problems of insufficient pixel-level labels and large inter-domain differences between images from different sources. Among them, the unsupervised domain adaptation method based on self-training, such as DAFormer, predicts and generates pseudo-labels for the target domain through a segmentation model, and uses the pseudo-labels to learn the unique features of the target domain in order to better transfer to unlabeled target domain images. However, DAFormer ignores the large style differences between remote sensing images from different sources, and there are problems in remote sensing images such as large intra-class variance, small inter-class variance, and imbalance in the number of samples in different categories, and the domain adaptation segmentation performance needs to be improved. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to overcome the above-mentioned disadvantages of the prior art and provide a two-stage unsupervised domain adaptation semantic segmentation method for remote sensing images with high segmentation accuracy, fast segmentation speed, and clear details.

[0006] The technical solution adopted to solve the above technical problem consists of the following steps:

[0007] (1) Dataset preprocessing

[0008] Two groups of remote sensing images from different sources. One group of labeled remote sensing images is called source domain images, and the other group of unlabeled remote sensing images is called target domain images. Both the source domain images and the target domain images are cropped into images with a size of 512×512 pixels, and the source domain images and the target domain images are respectively divided into a training set and a test set according to a ratio of 7:3.

[0009] (2) Construct a style conversion network

[0010] The style transfer network is composed of a discriminative network, a generative network, and a segmentation network connected in series in sequence;

[0011] The described discriminative network is composed of convolutional layer 1, convolutional layer 2, convolutional layer 3, and convolutional layer 4 connected in series in sequence.

[0012] The described generative network is composed of a feature extraction module, a self-attention mechanism layer, a backbone network, and a deconvolutional layer connected in series in sequence.

[0013] The described segmentation network is composed of an encoder, a spatial decoder, and a detail decoder connected. One output end of the encoder is connected to the spatial decoder, and the other output end is connected to the detail decoder.

[0014] (3) Training the style transfer network

[0015] 1) Constructing the loss function of the generative network

[0016] Construct the loss function L of the generative network according to Equation (1) G :

[0017] L G =α adv L adv +α nce L nce +α per L per (1) where, L adv represents the adversarial loss function, L nce represents the contrastive mutual information loss function, L per represents the perceptual loss function, α adv represents the weight of the adversarial loss function, α nce represents the weight of the contrastive mutual information loss function, α per represents the weight of the perceptual loss function.

[0018] Construct the adversarial loss function L according to the following formula adv :

[0019] L adv =log(1 - D(G(X S )))

[0020] where, X S represents the source domain image input to the style transfer network, G(X S ) represents the source domain image with the style of the target domain image output by the generative network, D(G(X S )) represents the probability that the discriminative network judges as real.

[0021] Construct the contrastive mutual information loss function L according to Equation (2) nce :

[0022]

[0023] Among them, V represents the anchor sample, which is the image patch at a certain position in the generated image. V + represents the positive sample, which is the image patch at the corresponding position in the source domain image. V - represents the negative sample, which is several image patches at other positions in the source domain image. τ represents the temperature parameter, and τ ∈ (0, 1].

[0024] Construct the perceptual loss function L according to the following formula per :

[0025]

[0026] Among them, F represents the segmentation network. IN(F(X S )) n ) and IN(F(G(X S )) n ) represent the normalization operation. n represents the current pixel, N represents the number of pixels, and N takes the value of 512×512.

[0027] 2) Construct the loss function of the discriminative network

[0028] Construct the loss function of the discriminative network L according to the following formula D :

[0029] L D = -logD(X T ) - log(1 - D(G(X S )))

[0030] Among them, X T represents the target domain image.

[0031] 3) Train the style transfer network

[0032] Input the training sets of the source domain image and the target domain image into the style transfer network for training. The generation network is optimized by the adaptive moment estimation optimizer, and the initial learning rate is 2×10 -4 , and the discriminative network is optimized by the root mean square propagation gradient descent method, and the initial learning rate is set to 2×10 -4 . 2 images are read in each training batch, and the training is iterated 240,000 times until the loss functions of the generation network and the discriminative network converge to obtain the best weight file.

[0033] (4) Test the style transfer network

[0034] Input the training set of the source domain image into the trained style transfer network for testing, and output the generated remote sensing image.

[0035] (5) Construct an unsupervised domain adaptation semantic segmentation network

[0036] The unsupervised domain adaptation semantic segmentation network is composed of a Student segmentation network and a Teacher segmentation network in parallel.

[0037] The described Student segmentation network is composed of Transformer block 1, Transformer block 2, Transformer block 3, Transformer block 4, and MLP layer 1 connected in series in sequence.

[0038] The described Teacher segmentation network is composed of Transformer block 5, Transformer block 6, Transformer block 7, Transformer block 8, and MLP layer 2 connected in series in sequence.

[0039] The described Transformer block 1 is composed of an efficient self-attention mechanism layer, a hybrid feed-forward network, and an overlapping block merger connected in series in sequence. The structures of Transformer block 2, Transformer block 3, Transformer block 4, Transformer block 5, Transformer block 6, Transformer block 7, and Transformer block 8 are the same as the structure of Transformer block 1.

[0040] (6) Train the unsupervised domain adaptation semantic segmentation network

[0041] 1) Construct the loss function of the Student segmentation network

[0042] Construct the loss function L of the Student segmentation network according to the following formula S :

[0043]

[0044] Among them, represents the source domain segmentation loss function, L aff represents the affinity loss function, represents the target domain segmentation loss function, α aff represents the weight of the affinity loss function.

[0045] Construct the source domain segmentation loss function according to the following formula

[0046]

[0047] Among them, Y S represents the source domain label, P S represents the source domain segmentation prediction result output by the Student segmentation network.

[0048] Construct the affinity loss function \(L\) according to Equation (3) aff :[[]]

[0049]

[0050] where \(D\)[[]] KL represents the KL divergence, \(c\) represents the category, \(i\) represents the current pixel, \(i\in\{2\)[[]] 0 , 2[[]] 1 ,…, 2[[]] 18 \(\}\), \(j\) represents the neighboring pixel of \(i\), \(j\in\{1, 2, \ldots, 8\}\),[[]] represents that the \(i\)-th pixel in the source domain label is of category \(c\),[[]] represents that the \(i\)-th pixel in the source domain segmentation prediction result is of category \(c\), \(k\) represents the penalty parameter, \(k\in[0, 1]\).[[]]

[0051] Construct the target domain segmentation loss function according to the following formula[[]]

[0052]

[0053] where \(Y\)[[]] M represents the target domain segmentation prediction result output by the Teacher segmentation network, as the pseudo-label of the target domain, \(P\)[[]] M represents the target domain segmentation prediction result output by the Student segmentation network, \(r\)[[]] p represents the ratio of the number of reliable pixels in \(Y\)[[]] M to the total number of pixels.[[]]

[0054] 2) Train the unsupervised domain adaptation semantic segmentation network[[]]

[0055] Input the training sets of the generated images and the target domain images into the Student segmentation network, and input the training set of the target domain images into the Teacher segmentation network for training. The Student segmentation network is optimized by the Adam optimizer with weight decay. The encoder learning rate is \(6\times10\)[[]] -5 , and the decoder learning rate is \(6\times10\)[[]] -4 , and the weight decay is \(2\times10\)[[]] -2 . Read 2 images in each training batch, and train for 40000 iterations until the loss function of the Student segmentation network converges to obtain the best weight file of the Student segmentation network.[[]]

[0056] (7) Test the unsupervised domain adaptation semantic segmentation network[[]]

[0057] Input the test set of the target domain images into the trained Student segmentation network for testing, and output the semantic segmentation result map.[[]]

[0058] (8) Evaluate the output segmentation result map

[0059] Evaluate the semantic segmentation result map of the output target domain test set image using the intersection over union (IoU) and F1-score (F1-Score) according to the following formula;

[0060]

[0061] Among them, A represents the set of label maps, B represents the set of predicted segmentation results, and |·| represents calculating the number of pixels in the set.

[0062]

[0063] Among them, P represents the proportion of the number of correctly predicted positive classes to all predicted positive samples, and R represents the proportion of the number of correctly predicted positive classes to all positive samples.

[0064] In step (2) of constructing the style transfer network of the present invention, the feature extraction module is sequentially composed of a convolutional layer 6, a batch normalization layer 2, and a ReLU activation function layer 2 in series.

[0065] In step (2) of constructing the style transfer network of the present invention, the encoder is sequentially composed of a convolutional layer 5, a batch normalization layer 1, and a ReLU activation function layer 1 in series; the spatial decoder is composed of a dilated spatial pyramid pooling module and an upsampling layer 1 in series; the detail decoder is composed of a boundary enhancement module and an upsampling layer 2 in series.

[0066] In step (3) of training the style transfer network of the present invention, in 1) constructing the loss function of the generation network, the weight α adv The value range is 1 to 5, and the weight α nce The value range is 1 to 4, and the weight α per The value range is 2 to 6.

[0067] In step (3) of training the style transfer network of the present invention, in 1) constructing the loss function of the generation network, the weight α adv The optimal value is 2, and the weight α nce The optimal value is 3, and the weight α per The optimal value is 4.

[0068] In step (6) of training the unsupervised domain adaptation semantic segmentation network of the present invention, in 1) constructing the loss function of the Student segmentation network, the weight α aff The value range is 0.1 to 1.

[0069] In step (6) of training the unsupervised domain adaptation semantic segmentation network of the present invention, in 1) constructing the loss function of the Student segmentation network, the weight α affThe optimal value of

[0070] The present invention adopts a two-stage training step, uses a style conversion network to generate source domain images similar to the target domain style, so as to narrow the style difference between the source domain and the target domain, and uses the generated images for an unsupervised domain adaptation semantic segmentation network. In the training step, an adversarial loss, a contrastive mutual information loss, and a perceptual loss are combined to maintain the content consistency between the generated images and the source domain images. To enhance the key feature extraction ability, a self-attention mechanism layer is introduced in the generation network step; a dual-branch space and detail decoder is adopted in the segmentation network, which strengthens the learning of local context information and boundary information. An encoder of a parallel Student segmentation network and Teacher segmentation network is adopted, and global context information is learned through cascaded Transformer blocks. The decoder adopts an MLP layer, achieving a balance between segmentation accuracy and computational performance. Compared with the prior art, the remote sensing images generated by the present invention have consistent content and clear details, effectively improving the segmentation performance of unlabeled target domain remote sensing images, and are applicable to unsupervised domain adaptation semantic segmentation of remote sensing images. Brief Description of the Drawings

[0071] Figure 1 It is a flowchart of Embodiment 1 of the present invention.

[0072] Figure 2 It is a schematic structural diagram of the style conversion network.

[0073] Figure 3 It is a schematic structural diagram of the feature extraction module.

[0074] Figure 4 It is a schematic structural diagram of the unsupervised domain adaptation semantic segmentation network.

[0075] Figure 5 It is a schematic structural diagram of Transformer block 1.

[0076] Figure 6 It is a result diagram of Embodiment 1 and the comparative experiment. Detailed Description of the Invention

[0077] The present invention will be further described below in conjunction with the drawings and examples, but the present invention is not limited to the following examples.

[0078] Embodiment 1

[0079] Figure 1 A flowchart of Embodiment 1 of the present invention is given. In Figure 1 This embodiment's two-stage unsupervised domain adaptation semantic segmentation method for remote sensing images consists of the following steps:

[0080] (1) Dataset preprocessing

[0081] Two groups of remote sensing images with different sources. The group of remote sensing images with labels is called the source domain images, and the group of remote sensing images without labels is called the target domain images. Both the source domain images and the target domain images are cropped into images with a pixel size of 512×512, and the source domain images and the target domain images are divided into training sets and test sets according to 7:3 respectively.

[0082] (2) Construct a style transfer network

[0083] Figure 2 、 3 The structural schematic diagram of the style transfer network in this embodiment is given. In Figure 2 、 3 In this embodiment, the style transfer network is composed of a discriminant network, a generation network, and a segmentation network connected in series in turn.

[0084] The discriminant network of this embodiment is composed of convolutional layer 1, convolutional layer 2, convolutional layer 3, and convolutional layer 4 connected in series in turn.

[0085] The generation network of this embodiment is composed of a feature extraction module, a self-attention mechanism layer, a backbone network, and a deconvolution layer connected in series in turn.

[0086] In Figure 3 In this embodiment, the feature extraction module is composed of convolutional layer 6, batch normalization layer 2, and ReLU activation function layer 2 connected in series in turn.

[0087] The segmentation network of this embodiment is composed of an encoder, a spatial decoder, and a detail decoder connected. One output end of the encoder is connected to the spatial decoder, and the other output end is connected to the detail decoder. The encoder of this embodiment is composed of convolutional layer 5, batch normalization layer 1, and ReLU activation function layer 1 connected in series in turn. The spatial decoder of this embodiment is composed of an atrous spatial pyramid pooling module and an upsampling layer 1 connected in series. The detail decoder of this embodiment is composed of a boundary enhancement module and an upsampling layer 2 connected in series.

[0088] Since this embodiment uses a style transfer network to generate source domain images similar to the target domain style to reduce the style difference between the source domain and the target domain, the generated images are used in an unsupervised domain adaptation semantic segmentation network. The self-attention mechanism layer is introduced in the generation network step; a dual-branch spatial and detail decoder is adopted in the segmentation network, which strengthens the learning of local context information and boundary information. The encoders of the parallel Student segmentation network and Teacher segmentation network are adopted, and the global context information is learned through the series-connected Transformer block. The decoder adopts an MLP layer, achieving a balance between segmentation accuracy and computational performance.

[0089] (3) Train the style transfer network

[0090] 1) Construct the loss function of the generation network

[0091] Construct the loss function \(L\) of the generation network according to Equation (1) G :

[0092] \(L\) G =\(\alpha\) adv \(L\) adv +\(\alpha\) nce \(L\) nce +\(\alpha\) per \(L\) per (1)

[0093] Among them, \(L\) adv represents the adversarial loss function, \(L\) nce represents the contrastive mutual information loss function, \(L\) per represents the perceptual loss function, \(\alpha\) adv represents the weight of the adversarial loss function, \(\alpha\) nce represents the weight of the contrastive mutual information loss function, \(\alpha\) per represents the weight of the perceptual loss function, and the weight \(\alpha\) adv has a value range of 1 to 5, the weight \(\alpha\) nce has a value range of 1 to 4, the weight \(\alpha\) per has a value range of 2 to 6, and the weight \(\alpha\) in this embodiment adv has a value of 2, the weight \(\alpha\) nce has a value of 3, the weight \(\alpha\) per has a value of 4.

[0094] Construct the adversarial loss function \(L\) according to the following formula adv :

[0095] \(L\) adv =\(\log(1 - D(G(X S )))\)

[0096] Among them, \(X\) S represents the source domain image input to the style conversion network, \(G(X S )\) represents the source domain image with the style of the target domain image output by the generation network, \(D(G(X S ))\) represents the probability that the discriminator network judges as real.

[0097] Construct the contrastive mutual information loss function \(L\) according to Equation (2) nce :

[0098]

[0099] Among them, \(V\) represents the anchor sample, that is, an image patch at a certain position in the generated image, \(V\) + represents the positive sample, that is, the image patch at the corresponding position in the source domain image, \(V\) -Indicates negative samples, which are several image patches at other positions in the source domain image. τ represents the temperature parameter, τ ∈ (0, 1], and the value of τ in this embodiment is 0.5.

[0100] Construct the perceptual loss function L according to the following formula per :

[0101]

[0102] Among them, F represents the segmentation network, IN(F(X S )) n ) and IN(F(G(X S )) n ) represent the normalization operation, n represents the current pixel, N represents the number of pixels, and the value of N is 512×512.

[0103] 2) Construct the loss function of the discriminative network

[0104] Construct the loss function of the discriminative network L according to the following formula D :

[0105] L D =-logD(X T )-log(1 - D(G(X S )))

[0106] Among them, X T represents the target domain image.

[0107] 3) Train the style transfer network

[0108] Input the training sets of the source domain image and the target domain image into the style transfer network for training. The generation network is optimized by the adaptive moment estimation optimizer, and the initial learning rate is 2×10 -4 , and the discriminative network is optimized by the root mean square propagation gradient descent method, and the initial learning rate is set to 2×10 -4 . Two images are read in each training batch, and the training is iterated 240,000 times until the loss functions of the generation network and the discriminative network converge to obtain the best weight file.

[0109] (4) Test the style transfer network

[0110] Input the training set of the source domain image into the trained style transfer network for testing, and output the generated remote sensing image.

[0111] (5) Construct an unsupervised domain adaptation semantic segmentation network

[0112] Figure 4 The structural schematic diagram of the unsupervised domain adaptation semantic segmentation network in this embodiment is given. In Figure 4Among them, the unsupervised domain adaptation semantic segmentation network of this embodiment is composed of a Student segmentation network and a Teacher segmentation network connected in parallel. The Student segmentation network of this embodiment is successively composed of Transformer block 1, Transformer block 2, Transformer block 3, Transformer block 4, and MLP layer 1 connected in series. Among them, Transformer block 1, Transformer block 2, Transformer block 3, and Transformer block 4 are successively connected in series to form an encoder, and MLP layer 1 forms a decoder. The Teacher segmentation network of this embodiment is successively composed of Transformer block 5, Transformer block 6, Transformer block 7, Transformer block 8, and MLP layer 2 connected in series. Among them, Transformer block 5, Transformer block 6, Transformer block 7, and Transformer block 8 are successively connected in series to form an encoder, and MLP layer 2 forms a decoder.

[0113] Figure 5 Figure 4 shows the structural schematic diagram of Transformer block 1 of this embodiment. In Figure 5 Among them, Transformer block 1 of this embodiment is successively composed of an efficient self-attention mechanism layer, a hybrid feed-forward network, and an overlapping block merger connected in series. The structures of Transformer block 2, Transformer block 3, Transformer block 4, Transformer block 5, Transformer block 6, Transformer block 7, and Transformer block 8 are the same as those of Transformer block 1.

[0114] (6) Training the unsupervised domain adaptation semantic segmentation network

[0115] 1) Constructing the loss function of the Student segmentation network

[0116] The loss function L of the Student segmentation network is constructed according to the following formula S :

[0117]

[0118] Among them, represents the source domain segmentation loss function, L aff represents the affinity loss function, represents the target domain segmentation loss function, and α aff represents the weight of the affinity loss function. The weight α aff has a value range of 0.1 to 1, and the weight α aff of this embodiment takes a value of 0.5.

[0119] Construct the source domain segmentation loss function according to the following formula

[0120]

[0121] where Y S represents the source domain label, and P S represents the source domain segmentation prediction result output by the Student segmentation network.

[0122] Construct the affinity loss function L aff :

[0123]

[0124] where D KL represents the KL divergence, c represents the category, i represents the current pixel, i ∈ {2 0 , 2 1 , …, 2 18}, j represents the neighboring pixel of i, j ∈ {1, 2, …, 8}, represents that the i-th pixel in the source domain label is of category c, represents that the i-th pixel in the source domain segmentation prediction result is of category c, k represents the penalty parameter, k ∈ [0, 1], and the value of k in this embodiment is 0.5.

[0125] Construct the target domain segmentation loss function according to the following formula

[0126]

[0127] where Y M represents the target domain segmentation prediction result output by the Teacher segmentation network and serves as the pseudo label of the target domain, and P M represents the target domain segmentation prediction result output by the Student segmentation network, and r p represents the ratio of the number of reliable pixels in Y M to the total number of pixels.

[0128] 2) Train the unsupervised domain adaptation semantic segmentation network

[0129] Input the training sets of the generated images and the target domain images into the Student segmentation network, and input the training set of the target domain images into the Teacher segmentation network for training. The Student segmentation network is optimized by the Adam optimizer with weight decay. The learning rate of the encoder is 6×10 -5 , and the learning rate of the decoder is 6×10 -4 , and the weight decay is 2×10 -2, 2 images are read in each training batch, and the training is iterated 40,000 times until the loss function of the Student segmentation network converges, obtaining the best weight file of the Student segmentation network.

[0130] (7) Test the unsupervised domain adaptation semantic segmentation network

[0131] Input the test set of the target domain images into the trained Student segmentation network for testing, and output the semantic segmentation result map.

[0132] (8) Evaluate the output segmentation result map

[0133] Use the intersection over union IoU and F1 score F1-Score to evaluate the semantic segmentation result map of the output target domain test set images according to the following formula.

[0134]

[0135] Among them, A represents the set of label maps, B represents the set of predicted segmentation results, and |·| represents calculating the number of pixels in the set.

[0136]

[0137] Among them, P represents the proportion of the number of correctly predicted positive classes to all samples predicted as positive, and R represents the proportion of the number of correctly predicted positive classes to all positive samples.

[0138] Complete the two-stage unsupervised domain adaptation semantic segmentation method for remote sensing images.

[0139] Example 2

[0140] The two-stage unsupervised domain adaptation semantic segmentation method for remote sensing images in this example consists of the following steps:

[0141] (1) Dataset preprocessing

[0142] This step is the same as that in Example 1.

[0143] (2) Construct the style transfer network

[0144] This step is the same as that in Example 1.

[0145] (3) Train the style transfer network

[0146] 1) Construct the loss function of the generation network

[0147] Construct the loss function L of the generation network according to Equation (1) G :

[0148] L G = α adv Ladv +α nce L nce +α per L per (1)

[0149] Among them, L adv represents the adversarial loss function, L nce represents the contrastive mutual information loss function, L per represents the perceptual loss function, α adv represents the weight of the adversarial loss function, α nce represents the weight of the contrastive mutual information loss function, α per represents the weight of the perceptual loss function, the weight α adv has a value range of 1 to 5, the weight α nce has a value range of 1 to 4, the weight α per has a value range of 2 to 6, the weight α of this embodiment adv has a value of 1, the weight α nce has a value of 1, the weight α per has a value of 2.

[0150] The expression of the adversarial loss function L adv is the same as that in Embodiment 1. In this expression, the meanings, value ranges of the coefficients, parameters and variables are the same as those in Embodiment 1.

[0151] Construct the contrastive mutual information loss function L according to Equation (2) nce :

[0152] The expression of Equation (2) is the same as that in Embodiment 1.

[0153] In Equation (2), τ represents the temperature parameter, τ ∈ (0, 1], and the value of τ in this embodiment is 0.1. The meanings, value ranges of other coefficients, parameters and variables are the same as those in Embodiment 1.

[0154] The expression of the perceptual loss function L per is the same as that in Embodiment 1. In this expression, the meanings, value ranges of the coefficients, parameters and variables are the same as those in Embodiment 1.

[0155] 2) Construct the loss function of the discriminant network

[0156] This step is the same as that in Embodiment 1.

[0157] 3) Train the style transfer network

[0158] This step is the same as that in Embodiment 1.

[0159] (4) Test the style transfer network

[0160] This step is the same as that in Embodiment 1.

[0161] (5) Construct an unsupervised domain adaptation semantic segmentation network

[0162] This step is the same as that in Embodiment 1.

[0163] (6) Train the unsupervised domain adaptation semantic segmentation network

[0164] 1) Construct the loss function of the Student segmentation network

[0165] Construct the loss function L of the Student segmentation network according to the following formula S :

[0166]

[0167] Wherein, represents the source domain segmentation loss function, L aff represents the affinity loss function, represents the target domain segmentation loss function, α aff represents the weight of the affinity loss function. The weight α aff ranges from 0.1 to 1, and the weight α in this embodiment aff takes the value of 0.1.

[0168] The source domain segmentation loss function has the same expression as that in Embodiment 1. In this expression, the meanings, value ranges of the coefficients, parameters and variables are the same as those in Embodiment 1.

[0169] Construct the affinity loss function L according to formula (3) aff :

[0170]

[0171] In formula (3), k represents the penalty parameter, k ∈ [0,1], and the value of k in this embodiment is 0. The meanings, value ranges of other coefficients, parameters and variables are the same as those in Embodiment 1.

[0172] The target domain segmentation loss function has the same expression as that in Embodiment 1. In this expression, the meanings, value ranges of the coefficients, parameters and variables are the same as those in Embodiment 1.

[0173] 2) Train the unsupervised domain adaptation semantic segmentation network

[0174] This step is the same as that in Embodiment 1.

[0175] Other steps are the same as those in Embodiment 1. The unsupervised domain adaptation semantic segmentation method for remote sensing images based on two stages is completed.

[0176] Embodiment 3

[0177] The two-stage unsupervised domain adaptation semantic segmentation method for remote sensing images in this embodiment consists of the following steps:

[0178] (1) Dataset preprocessing

[0179] This step is the same as that in Embodiment 1.

[0180] (2) Construct a style transfer network

[0181] This step is the same as that in Embodiment 1.

[0182] (3) Train the style transfer network

[0183] 1) Construct the loss function of the generation network

[0184] Construct the loss function \(L\) of the generation network according to Equation (1) G :

[0185] \(L\) G =\(\alpha\) adv \(L\) adv +\(\alpha\) nce \(L\) nce +\(\alpha\) per \(L\) per (1)

[0186] Among them, \(L\) adv represents the adversarial loss function, \(L\) nce represents the contrastive mutual information loss function, \(L\) per represents the perceptual loss function, \(\alpha\) adv represents the weight of the adversarial loss function, \(\alpha\) nce represents the weight of the contrastive mutual information loss function, \(\alpha\) per represents the weight of the perceptual loss function, and the weight \(\alpha\) adv ranges from 1 to 5, the weight \(\alpha\) nce ranges from 1 to 4, the weight \(\alpha\) per ranges from 2 to 6. In this embodiment, the weight \(\alpha\) adv takes the value of 5, the weight \(\alpha\) nce takes the value of 4, and the weight \(\alpha\) per takes the value of 6.

[0187] The expression of the adversarial loss function \(L\) adv is the same as that in Embodiment 1. In this expression, the meanings, value ranges of the coefficients, parameters, and variables are the same as those in Embodiment 1.

[0188] Construct the contrastive mutual information loss function \(L\) nce according to Equation (2):

[0189] The expression of Equation (2) is the same as that in Embodiment 1.

[0190] In formula (2), τ represents the temperature parameter, τ ∈ (0, 1], and the value of τ in this embodiment is 1. The meanings, value ranges of other coefficients, parameters, and variables are the same as those in Embodiment 1.

[0191] Perceptual loss function L per has the same expression as that in Embodiment 1. In this expression, the meanings, value ranges of coefficients, parameters, and variables are the same as those in Embodiment 1.

[0192] 2) Construct the loss function of the discriminant network

[0193] This step is the same as that in Embodiment 1.

[0194] 3) Train the style transfer network

[0195] This step is the same as that in Embodiment 1.

[0196] (4) Test the style transfer network

[0197] This step is the same as that in Embodiment 1.

[0198] (5) Construct an unsupervised domain adaptation semantic segmentation network

[0199] This step is the same as that in Embodiment 1.

[0200] (6) Train the unsupervised domain adaptation semantic segmentation network

[0201] 1) Construct the loss function of the Student segmentation network

[0202] Construct the loss function L of the Student segmentation network according to the following formula S :

[0203]

[0204] where represents the source domain segmentation loss function, L aff represents the affinity loss function, represents the target domain segmentation loss function, and α aff represents the weight of the affinity loss function. The weight α aff has a value range of 0.1 to 1, and the weight α aff in this embodiment takes the value of 1.

[0205] Source domain segmentation loss function has the same expression as that in Embodiment 1. In this expression, the meanings, value ranges of coefficients, parameters, and variables are the same as those in Embodiment 1.

[0206] Construct the affinity loss function L according to formula (3) aff :

[0207]

[0208] In Equation (3), k represents the penalty parameter, where k ∈ [0, 1], and in this embodiment, k is taken as 1. The meanings, value ranges of other coefficients, parameters, and variables are the same as those in Embodiment 1.

[0209] Target domain segmentation loss function has the same expression as that in Embodiment 1. In this expression, the meanings, value ranges of the coefficients, parameters, and variables are the same as those in Embodiment 1.

[0210] 2) Training an unsupervised domain adaptation semantic segmentation network

[0211] This step is the same as that in Embodiment 1.

[0212] Other steps are the same as those in Embodiment 1. Thus, the two-stage unsupervised domain adaptation semantic segmentation method for remote sensing images is completed.

[0213] To verify the beneficial effects of the present invention, a computer simulation comparison experiment was conducted using the two-stage unsupervised domain adaptation semantic segmentation method of Embodiment 1 of the present invention and the semantic segmentation network AdaptSegNet based on adaptive adversarial training (hereinafter referred to as Comparative Experiment 1), the memory-based adaptive network MemoryAdaptNet (hereinafter referred to as Comparative Experiment 2), and the unsupervised domain adaptation semantic segmentation network DAFormer based on Transformer (hereinafter referred to as Comparative Experiment 3). The experimental results are shown in Figure 6 and Table 1.

[0214] Table 1 Results table of the present invention and comparative experiments

[0215] Experimental method Intersection over Union IoU (%) F1-Score F1-Score (%) Comparative experiment 1 22.71 33.70 Comparative experiment 2 49.30 61.11 Comparative experiment 3 64.17 74.92 Example 1 of the present invention 67.47 78.20

[0216] As can be seen from Figure 6 in the figure, (a) is the original image, (b) is the label map, (c) is the segmentation result map of Comparative Experiment 1, (d) is the segmentation result map of Comparative Experiment 2, (e) is the segmentation result map of Comparative Experiment 3, and (f) is the segmentation result map of Embodiment 1 of the present invention. The segmentation result map output by the method of Embodiment 1 of the present invention can effectively improve the technical problem that the shadow part and the categories with a relatively dense spatial distribution are misclassified.

[0217] As can be seen from Table 1, the intersection over union of the method of the present invention is increased by 44.76% and the F1-score is increased by 44.50% compared with Comparative Experiment 1, the intersection over union is increased by 18.17% and the F1-score is increased by 17.09% compared with Comparative Experiment 2, and the intersection over union is increased by 3.30% and the F1-score is increased by 3.28% compared with Comparative Experiment 3.

Claims

1. A two-stage unsupervised domain adaptation semantic segmentation method for remote sensing images, characterized in that It consists of the following steps: (1) Dataset preprocessing Two groups of remote sensing images with different sources. The group of remote sensing images with labels is called the source domain images, and the group of remote sensing images without labels is called the target domain images. Both the source domain images and the target domain images are cropped into images with a pixel size of 512×512. The source domain images and the target domain images are respectively divided into a training set and a test set according to a ratio of 7:3; (2) Construct a style transfer network The style transfer network is composed of a discriminative network, a generative network, and a segmentation network connected in series in sequence; The described discriminative network is composed of convolutional layer 1, convolutional layer 2, convolutional layer 3, and convolutional layer 4 connected in series in sequence; The described generative network is composed of a feature extraction module, a self-attention mechanism layer, a backbone network, and a deconvolution layer connected in series in sequence; The described segmentation network is composed of an encoder, a spatial decoder, and a detail decoder connected. One output end of the encoder is connected to the spatial decoder, and the other output end is connected to the detail decoder; (3) Train the style transfer network 1) Construct the loss function of the generative network Construct the loss function \(L\) of the generation network according to formula (1). G : L G = α adv L adv + α nce L nce + α per L per (1) Among them, L adv represents the adversarial loss function, L nce represents the contrastive mutual information loss function, L per represents the perceptual loss function, α adv represents the weight of the adversarial loss function, α nce represents the weight of the contrastive mutual information loss function, α per represents the weight of the perceptual loss function; Construct the adversarial loss function \(L\) according to the following formula adv :[[]] L adv = log(1 - D(G(X S ))) Among them, X S represents the source domain image of the input style conversion network, G(X S ) represents the source domain image with the style of the target domain image output by the generation network, and D(G(X S )) represents the probability that the discriminant network determines to be real; Construct the contrastive mutual information loss function \(L\) according to Equation (2) nce :[[]]END]] Among them, V represents the anchor sample, that is, the image patch at a certain position in the generated image, V + represents the positive sample, that is, the image patch at the corresponding position in the source domain image, V - represents the negative sample, that is, several image patches at other positions in the source domain image, and τ represents the temperature parameter, τ ∈ (0, 1]; Construct the perceptual loss function \(L\) according to the following formula per : Among them, F represents the segmentation network, IN(F(X S ) n ) and IN(F(G(X S )) n ) represent the normalization operation, n represents the current pixel, N represents the number of pixels, and N takes the value of 512×512; 2) Construct the loss function of the discriminative network Construct the loss function \(L\) of the discriminant network according to the following formula D :[[]]END]] L D = -logD(X T ) - log(1 - D(G(X S ))) Among them, X T represents the target domain image; 3) Train the style transfer network Input the training sets of the source domain image and the target domain image into the style transfer network for training. The generation network is optimized by the adaptive moment estimation optimizer with an initial learning rate of 2×10 -4 , and the discriminative network is optimized by the root mean square propagation gradient descent method with an initial learning rate set to 2×10 -4 . Read 2 images in each training batch, perform 240,000 training iterations, and train until the loss functions of the generation network and the discriminative network converge to obtain the best weight file; (4) Test the style transfer network Input the training set of the source domain images into the trained style transfer network for testing, and output the generated remote sensing images; (5) Construct an unsupervised domain adaptation semantic segmentation network The unsupervised domain adaptation semantic segmentation network is composed of a Student segmentation network and a Teacher segmentation network connected in parallel; The described Student segmentation network is composed of Transformer block 1, Transformer block 2, Transformer block 3, Transformer block 4, and MLP layer 1 connected in series in sequence; The described Teacher segmentation network is composed of Transformer block 5, Transformer block 6, Transformer block 7, Transformer block 8, and MLP layer 2 connected in series in sequence; The described Transformer block 1 is composed of an efficient self-attention mechanism layer, a hybrid feed-forward network, and an overlapping block merger connected in series in sequence. The structures of Transformer block 2, Transformer block 3, Transformer block 4, Transformer block 5, Transformer block 6, Transformer block 7, and Transformer block 8 are the same as the structure of Transformer block 1; (6) Train the unsupervised domain adaptation semantic segmentation network 1) Construct the loss function of the Student segmentation network Construct the loss function \(L\) of the Student segmentation network according to the following formula S : Among them, represents the source domain segmentation loss function, L aff represents the affinity loss function, represents the target domain segmentation loss function, α aff represents the weight of the affinity loss function; Construct the source domain segmentation loss function according to the following formula Among them, Y S represents the source domain label, and P S represents the source domain segmentation prediction result output by the Student segmentation network; Construct the affinity loss function \(L\) according to Equation (3). aff : Among them, D KL represents the KL divergence, c represents the category, i represents the current pixel, and i ∈ {2 0 , 2 1 , …, 2 18}, j represents the neighboring pixel of i, and j ∈ {1, 2, …, 8}, indicates that the i-th pixel in the source domain label is of category c, indicates that the i-th pixel in the source domain segmentation prediction result is of category c, and k represents the penalty parameter, where k ∈ [0, 1]; Construct the target domain segmentation loss function according to the following formula Among them, Y M represents the target domain segmentation prediction result output by the Teacher segmentation network, serving as the pseudo-label of the target domain, P M represents the target domain segmentation prediction result output by the Student segmentation network, r p represents the ratio of the number of reliable pixels in Y M to the total number of pixels; 2) Train the unsupervised domain adaptation semantic segmentation network The training sets of the generated images and the target domain images are input into the Student segmentation network, and the training set of the target domain images is input into the Teacher segmentation network for training. The Student segmentation network is optimized by the Adam optimizer with weight decay. The learning rate of the encoder is 6×10 -5 , and the learning rate of the decoder is 6×10 -4 , the weight decay is 2×10 -2 . Two images are read in each training batch, and the training is iterated 40,000 times until the loss function of the Student segmentation network converges, and the best weight file of the Student segmentation network is obtained; (7) Test the unsupervised domain adaptation semantic segmentation network Input the test set of the target domain images into the trained Student segmentation network for testing, and output the semantic segmentation result map; (8) Evaluate the output segmentation result map Use the intersection over union IoU and the F1 score F1-Score to evaluate the semantic segmentation result map of the output target domain test set images according to the following formula; Among them, A represents the set of label maps, B represents the set of predicted segmentation results, and |·| represents calculating the number of pixels in the set; Among them, P represents the proportion of the number of correctly predicted positive classes in all samples predicted as positive, and R represents the proportion of the number of correctly predicted positive classes in all positive samples.

2. The unsupervised domain adaptation semantic segmentation method for remote sensing images based on two stages according to claim 1, wherein: In the construction of the style conversion network in step (2), the feature extraction module is sequentially composed of a convolutional layer 6, a batch normalization layer 2, and a ReLU activation function layer 2 in series.

3. The unsupervised domain adaptation semantic segmentation method for remote sensing images based on two - stage according to claim 1, characterized in that: In the construction of the style conversion network in step (2), the encoder is sequentially composed of a convolutional layer 5, a batch normalization layer 1, and a ReLU activation function layer 1 in series; the spatial decoder is composed of an atrous spatial pyramid pooling module and an upsampling layer 1 in series; the detail decoder is composed of a boundary enhancement module and an upsampling layer 2 in series.

4. The unsupervised domain adaptation semantic segmentation method for remote sensing images based on two-stage according to claim 1, characterized in that: In the construction of the loss function of the generation network in step (3) of training the style conversion network, the weight α adv has a value range of 1 to 5, and the weight α nce has a value range of 1 to 4, and the weight α per has a value range of 2 to 6.

5. The two-stage unsupervised domain adaptation semantic segmentation method for remote sensing images according to claim 1 or 4, characterized in that: In the construction of the loss function of the generation network in step (3) of training the style conversion network, the weight α adv takes the value of 2, and the weight α nce takes the value of 3, and the weight α per takes the value of 4.

6. The unsupervised domain adaptation semantic segmentation method for remote sensing images based on two stages according to claim 1, characterized in that: In the construction of the loss function of the Student segmentation network in step (6) for training the unsupervised domain adaptation semantic segmentation network, the weight α aff has a value range of 0.1 to 1.

7. The unsupervised domain adaptation semantic segmentation method for remote sensing images based on two stages according to claim 1, characterized in that: In the construction of the loss function of the Student segmentation network in step (6) of training the unsupervised domain adaptation semantic segmentation network, the weight α aff takes a value of 0.5.

Images