Cross-domain remote sensing image semantic segmentation method based on iterative intra-domain adaptation and self-training

Abstract

The present invention discloses a cross-domain remote sensing image semantic segmentation method based on iterative intra-domain adaptation and self-training, including source-target domain inter-domain domain adaptation model training, target domain category segmentation probability and pseudo-label generation, target domain image segmentation probability credibility score ranking, target domain intra-domain iterative domain adaptation model training and target domain segmentation result generation. The present invention reduces the source-target domain inter-domain difference by using source-target domain inter-domain adaptation, and reduces the target domain intra-domain difference by using target domain intra-domain adaptation, thereby improving the accuracy of the cross-domain remote sensing image semantic segmentation model, further classifies and sorts the target domain image based on the segmentation probability credibility, thereby selecting the prediction result with good segmentation effect as the pseudo-label, and proposes a new pseudo-label screening strategy to remove the pixels in the pseudo-label that are very likely to be wrong, thereby avoiding the influence of the pseudo-label error during self-training in the target domain.

Description

Technical Field

[0001] The present invention belongs to the technical field of remote sensing image semantic segmentation, and in particular relates to a cross-domain remote sensing image semantic segmentation method based on iterative intra-domain adaptation and self-training. Background Art

[0002] With the continuous development of remote sensing technology, remote sensing equipment such as satellites and drones can collect a large number of remote sensing satellite images. For example, drones can capture a large number of high-spatial-resolution remote sensing images over cities and villages. Such a large amount of remote sensing data provides many application opportunities, such as urban monitoring, urban management, agriculture, automatic mapping, and navigation; in these applications, the key technologies are semantic segmentation or image classification of remote sensing images.

[0003] In recent years, convolutional neural networks (CNNs) have become the most commonly used technology for semantic segmentation and image classification. Several CNN-based models have shown their effectiveness in this task, such as FCN, SegNet, U-Net series, PSPNets, and Deeplab series. These models can get good semantic segmentation results when the training images and test images are from the same satellite or city, but when we use these models to classify remote sensing images acquired from different satellites or cities, the test results of the models become very poor and unsatisfactory because of the different data distributions between different satellite and city images (domain shift). In some related literature, this problem is called domain adaptation; in the field of remote sensing, domain shift is usually caused by different atmospheric conditions when remote sensing equipment is imaging, acquisition differences (these differences will change the spectral characteristics of the object), differences in the spectral characteristics of the sensor, or differences in different types of spectral bands (for example, some images may be red, green, and blue bands, while others may be near-infrared, red, and green bands).

[0004] In a typical domain adaptation problem, training images and test images are usually specified as source domain and target domain. A common solution to handle domain adaptation is to create a new semantically labeled dataset on the target domain and train the model on it. Since it would be time-consuming and expensive to collect a large number of pixel-labeled images for the target city, this solution is very expensive and impractical. In order to reduce the workload of manual pixel-by-pixel classification, some solutions have been proposed, such as synthesizing data from weakly supervised labels. However, these methods still have limitations because they also require a lot of manual labor.

[0005] In order to improve the generalization ability of CNN-based semantic segmentation models, another common method is to perform data augmentation by randomly changing colors, such as gamma correction and image brightness conversion, which has been widely used in remote sensing. Nevertheless, when there are significant differences between data distributions, the above data augmentation methods cannot achieve good results in cross-domain semantic segmentation. Using this simple augmentation method, it is impossible to apply a model from a domain containing red, green, and blue bands to another domain containing near-infrared, red, and green channels. To overcome this limitation, Generative Adversarial Network (GAN) [I.Goodfellow, J.Pouget-Abadie, M.Mirza, B.Xu, D.Warde-Farley, S.Ozair, A.Courville, and Y.Bengio.Generative adversarial nets[C].Proceedingsof the international conference on Neural Information Processing Systems(NIPS).2014:2672~2680] is used to generate pseudo-target domain images with similar data distribution to the target domain images. These generated pseudo-target domain images can be used to train classifiers on the target domain.At the same time, some works based on adversarial learning [Y.-H.Tsai, W.-C.Hung, S.Schulter, K.Sohn, M.-H.Yang, and M.Chandraker.Learning toadapt structured output space for semantic segmentation[C]." Proceedings ofthe international conference on computer vision and pattern recognition(CVPR).2018:7472~7481] and self-training [Y.Zou, Z.Yu, B.Kumar, and J.Wang.Unsuperviseddomain adaptation for semantic segmentation via class-balanced self-training[C].Proceedings of the international conference on European conference oncomputer vision (ECCV). 2018: 289-305] have also been proposed by researchers to solve the domain adaptation problem. Although these methods have achieved good results on natural images, there are still some problems in directly applying these methods to remote sensing images. The most important point is that these methods ignore the differences in the target domain images themselves. For example, the styles and shapes of buildings in the same city can also be quite different.

[0006] Due to the differences in the target domain images themselves, the segmentation effects of the inter-domain semantic segmentation model migrated from the source domain to the target domain on all target domain images will also be different, that is, a relatively accurate segmentation result can be obtained on some target domain images, but the segmentation results obtained on other target domain images will become very poor. Therefore, how to further adapt the target domain images to the domain within the domain to reduce the differences within the target domain, so that the cross-domain semantic segmentation model can achieve good segmentation effects on all target domain images, is an important problem facing the semantic segmentation of cross-domain remote sensing images. Secondly, because the target domain images do not have corresponding labels, the current common method is to use self-training technology, use the semantic segmentation results generated by the trained cross-domain semantic segmentation model as the pseudo-label of the target domain image, and then use the pseudo-label to continue training the cross-domain semantic segmentation model to obtain the final target domain semantic segmentation model. The training effect of this self-training model based on pseudo-labels depends on the quality of pseudo-labels. When the quality of pseudo-labels is poor, the training effect of the model will be greatly weakened, and the semantic segmentation ability of the model will also be greatly weakened. Therefore, how to select image results with good model segmentation effects as pseudo-labels and how to improve the quality of pseudo-labels is also an important issue in self-training technology. Summary of the invention

[0007] In view of the above, the present invention provides a cross-domain remote sensing image semantic segmentation method based on iterative intra-domain adaptation and self-training, which can migrate the semantic segmentation model trained on remote sensing images of one domain to remote sensing images of other domains, and perform further intra-domain adaptation training in the target domain remote sensing images, thereby reducing the inter-domain differences between the source domain and the target domain while reducing the intra-domain differences in the target domain, thereby further improving the performance and robustness of the cross-domain remote sensing image semantic segmentation model.

[0008] A cross-domain remote sensing image semantic segmentation method based on iterative intra-domain adaptation and self-training includes the following steps:

[0009] (1) Using the source domain image x s , source domain label y s , source domain semantic segmentation model F S And the target domain image x t Train the source domain-target domain semantic segmentation model F inter ;

[0010] (2) The target domain image x t Input to the source domain-target domain inter-domain semantic segmentation model F inter In the target domain image x, we get t The category segmentation probability P t , and then use the category segmentation probability P t Calculate the segmentation probability confidence S t and the target domain pseudo-label

[0011] (3) All target domain images x t According to the segmentation probability confidence S t The size is sorted in descending order, and then all the target domain images x are sorted according to the sorting order. t Divide into K target domain image subsets on average K is a natural number greater than 1;

[0012] (4) Using a subset of target domain images with the highest segmentation probability confidence and its corresponding pseudo-label subset And the source domain-target domain inter-domain semantic segmentation model F inter and a subset of target domain images Iteratively train the semantic segmentation model F in the target domain intra ;

[0013] (5) The target domain image x t Input to the target domain semantic segmentation model F intra In the above example, we can get the target domain image x t The final category segmentation probability P and segmentation result map.

[0014] Furthermore, the specific implementation process of step (1) is as follows:

[0015] 1.1 Using the source domain image x s and the source domain label y s Train the source domain semantic segmentation model F S ;

[0016] 1.2 Using the source domain image x s And the target domain image x t Training a source-target domain image bidirectional converter, which includes an image converter in the source→target direction and an image converter in the target→source direction;

[0017] 1.3 For all the intermediate saved models of the image converters generated in the above training process, a set of optimal results is selected as the image converter G in the source→target direction S→T and the image converter G in the target→source direction T→S ;

[0018] 1.4 Using Image Converter G S→T The source domain image x s Convert from the source domain to the target domain to obtain the pseudo-target domain image G S→T (x s );

[0019] 1.5 Using the pseudo target domain image G S→T (x s) and the source domain label y s Train the source domain-target domain semantic segmentation model F inter .

[0020] Furthermore, the segmentation probability credibility S in step (2) t The calculation expression is as follows:

[0021]

[0022] Where: H and W are the target domain image x t The length and width of C is the target domain image x t The number of segmentation categories, Denotes the target domain image x t The pixel point with coordinates (h,w) corresponds to category c i The segmentation probability, c i represents the i-th category, i is a natural number and 1≤i≤C, and θ() is a function used to measure the likelihood between the segmentation probabilities of each category of pixel points.

[0023] Furthermore, the target domain pseudo label in step (2) The calculation expression is as follows:

[0024]

[0025] in: represents the target domain pseudo label The category of the pixel point with coordinates (h, w), P t (h,w,c) Denotes the target domain image x t The segmentation probability of the pixel point with coordinates (h, w) corresponding to category c, μ c is the segmentation probability threshold corresponding to category c, Denotes the target domain image x t The pixel point with coordinates (h,w) corresponds to category c i The segmentation probability, c i represents the i-th category, i is a natural number and 1≤i≤C, C is the target domain image x t The number of segmentation categories, Denotes the target domain image x t The segmentation probability chaos of the pixel with coordinates (h, w), v is the segmentation probability chaos threshold.

[0026] Furthermore, the segmentation probability disorder The calculation expression is as follows:

[0027]

[0028] Among them: δ() is a function used to measure the degree of confusion between the segmentation probabilities of different categories of pixel points.

[0029] Furthermore, the specific implementation process of step (4) is as follows:

[0030] 4.1 Initially, a set of target domain image subsets with the highest segmentation probability confidence and its corresponding pseudo-label subset As a training set and its corresponding label set The source domain-target domain semantic segmentation model F inter As a target domain semantic segmentation model

[0031] 4.2 Using the training set Label Set Target domain semantic segmentation model and a subset of target domain images Train a semantic segmentation model in the target domain k is a natural number and 2≤k≤K; the training process is similar to step (1);

[0032] 4.3 Subsetting the Target Domain Images Input to the target domain semantic segmentation model In the above example, we get the corresponding category segmentation probability Then use the category segmentation probability Calculate the target domain image subset The pseudo-label subset

[0033] 4.4 Subsetting the Target Domain Images and its pseudo-label subset Add to the training set and label sets middle;

[0034] 4.5 Let k = k + 1;

[0035] 4.6 Repeat steps 4.2 to 4.5 until k = K, and train the semantic segmentation model in the target domain. That is, as the semantic segmentation model F in the target domain intra .

[0036] The method of the present invention is a complete cross-domain remote sensing image semantic segmentation framework, which includes domain adaptation model training between source domain and target domain, target domain category segmentation probability and pseudo-label generation, target domain image segmentation probability credibility score ranking, target domain iterative domain adaptation model training and target domain segmentation result generation.

[0037] The present invention proposes an iterative domain adaptation training network in a target domain. When training the iterative domain adaptation training network, the present invention uses a commonly used self-training learning technology, and uses the part of the image with better segmentation effect and its segmentation result as pseudo labels to guide the training of the target domain segmentation model, so that the target domain model can also obtain better segmentation results on the part of the image with originally poor segmentation effect.

[0038] In addition, in order to cope with the complex and diverse distribution characteristics within the target domain, the present invention also proposes to divide the target domain into multiple subdomains and perform iterative in-domain adaptation training on the multiple subdomains; in order to divide the target domain into multiple subdomains, the present invention proposes a segmentation probability credibility calculation method, which sorts and classifies the target domain images according to the good or bad scores of the segmentation results of the target domain model, thereby selecting the part of the target domain images with good segmentation effect and their pseudo-labels to further optimize the target domain model.

[0039] In the process of obtaining pseudo labels, the present invention proposes a method combining a segmentation probability threshold and a segmentation probability confusion threshold to remove pixels with poor segmentation results in the pseudo labels, thereby avoiding low-quality pseudo labels from interfering with the target domain model training.

[0040] Based on the iterative domain adaptation training framework, the present invention realizes domain adaptation training within the target domain. After obtaining the migration model from the source domain to the target domain and the target domain segmentation result, the iterative domain adaptation training framework adopted by the present invention performs further domain adaptation training on the target domain model, and obtains the final target domain model and semantic segmentation result, thereby improving the accuracy of semantic segmentation of cross-domain remote sensing images. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 Schematic diagram of the steps of the cross-domain remote sensing image semantic segmentation method of the present invention.

[0042] Figure 2 It is a schematic diagram of the specific implementation process of the cross-domain remote sensing image semantic segmentation method of the present invention. DETAILED DESCRIPTION

[0043] In order to describe the present invention more specifically, the technical solution of the present invention is described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0044] like Figure 1 and Figure 2 As shown, the cross-domain remote sensing image semantic segmentation method based on iterative intra-domain adaptation and self-training of the present invention comprises the following steps:

[0045] (1) Using the source domain image x s , source domain label y s , source domain semantic segmentation model FS And the target domain image x t Train the source domain-target domain semantic segmentation model F inter .

[0046] In this implementation, there is no source domain semantic segmentation model F S When the source domain image x s and the source domain label y s After training, the model network structure can adopt the commonly used deeplab, U-net, etc. The loss function adopts the cross entropy loss with K categories, and the corresponding formula is as follows:

[0047]

[0048] Where: x s is the source domain image, y s is the source domain image label, K is the number of label categories, and F S is the semantic segmentation model on the source domain, is the indicator function (when k = y s hour, When k≠y s hour, Indicator function reference—Zhou Zhihua. Machine Learning [M]. Beijing: Tsinghua University Press, 2016. Main symbol table), represents the mathematical expectation function, For x s Input to Model F S The result of the kth category in the output results obtained.

[0049] This implementation uses Potsdam city images with building labels as the source domain, and crops them to 512*512 pixels, retaining RGB 3 channels. The number of images and corresponding building labels is 4000 each. The model network structure can use deeplabV3+, and the learning rate is 10 -4 , the optimization algorithm is adam, and 900 epochs are trained to obtain the semantic segmentation model F on the Potsdam domain S .

[0050] Common domain adaptation training from source domain to target domain includes image conversion and adversarial learning. This example uses the GAN-based image conversion method as an example, but it is not limited to the image conversion method. The image conversion method first needs to train a bidirectional image conversion model between the source domain and the target domain. The bidirectional image conversion model contains the source domain image x s To the target domain image x t Image Converter G S→T , target domain image x t To the source domain image xs Image Converter G T→S And the source domain discriminator D S and the target domain discriminator D T ,The training loss functions include cycle consistent loss function, semantic consistent loss function, self-loss function and adversarial loss function.

[0051] The equation expression of the cycle-consistent loss function is as follows:

[0052]

[0053] Where: x s is the source domain image, x t is the target domain image, G S→T is the source domain image x s To the target domain image x t Image converter, G T→S is the target domain image x t To the source domain image x s Image Converter, is the mathematical expectation function, ‖‖1 is the L1 norm.

[0054] The equation expression of the semantic consistency loss function is as follows:

[0055]

[0056] Where: x s is the source domain image, x t is the target domain image, G S→T is the source domain image x s To the target domain image x t Image converter, G T→S is the target domain image x t To the source domain image x s Image Converter, is the mathematical expectation function, F T is the semantic segmentation model on the target domain, F S is the semantic segmentation model on the source domain, and KL(||) is the KL divergence between the two distributions.

[0057] The equation expression of the adversarial loss function is as follows:

[0058]

[0059] Where: x s is the source domain image, x t is the target domain image, G S→T is the source domain image x s To the target domain image x t Image converter, G T→Sis the target domain image x t To the source domain image x s Image Converter, is the mathematical expectation function, D S is the source domain discriminator, D T is the target domain discriminator.

[0060] The equation expression of the self-loss function is as follows:

[0061]

[0062] Where: x s is the source domain image, x t is the target domain image, G S→T is the source domain image x s To the target domain image x t Image converter, G T→S is the target domain image x t To the source domain image x s Image Converter, is the mathematical expectation function, ‖·‖1 is the L1 norm.

[0063] In this implementation, Potsdam city images are used as source domains and Vaihingen city images are used as target domains. The image size is 512*512 pixels and 3 channels. There are 832 Potsdam city images (source domain) and 845 Vaihingen city images (target domain). Both images contain buildings. The image conversion model uses GAN, including Potsdam images x s To Vaihingen Image x t Image Converter G S→T 、Vaihingen image x t To Potsdam image x s Image Converter G T→S And the Potsdam domain discriminator D S and Vaihingen domain discriminator D T The generator network structure is a 9-layer ResNet, the discriminator network structure is a 4-layer CNN, the training loss function includes cycle consistency loss function, semantic consistency loss function, adversarial loss function and self-loss function, and the learning rate is 10 -4 The optimization algorithm is adam, and the training is stopped after 100 epochs. After the training, the image converter G in the Potsdam-Vaihingen direction is obtained. S→T and 10 image converters G in the Vaihingen-Potsdam direction T→S Then use the converter G S→TConvert 4000 512*512 pixel, 3-channel Potsdam satellite images from the Potsdam domain to the Vaihingen domain to obtain the quasi-Vaihingen image G S→T (x s ). Then use the quasi-Vaihingen (target domain) image G S→T (x s ) and Potsdam (source domain) label y s Train a quasi-Vaihingen (target domain) semantic segmentation model F inter .

[0064] The model network structure can adopt the commonly used deeplab, U-net, etc., and the loss function adopts the cross entropy loss with K categories. The corresponding formula is as follows:

[0065]

[0066] Where: x s is the source domain image, y s is the source domain image label, K is the number of label categories, and F inter is the semantic segmentation model on the target domain, is the indicator function (when k = y s hour, When k≠y s hour, represents the mathematical expectation function, G S→T (x s ) is the target domain image, G S→T (x s ) is input to model F inter The result of the kth category in the output results obtained.

[0067] This embodiment uses 4000 512*512 pixel, 3-channel pseudo-Vaihingen domain images G generated in step (1) S→T (x s ) and the source domain label y s Train the semantic segmentation model F on the Vaihingen domain inter ; The model network structure uses deeplabV3+, and the learning rate is 10 -4 , the optimization algorithm is adam, and the semantic segmentation model F on the quasi-Vaihingen domain is obtained after training for 100 epochs. inter .

[0068] (2) The target domain image x t Input to the source domain-target domain inter-domain semantic segmentation model F interIn the target domain image x, we get t The category segmentation probability P t , and use the category segmentation probability P t Calculate the segmentation probability confidence S T and the target domain pseudo-label

[0069] In this embodiment, 500 512*512 pixel, 3-channel Vaihingen domain images are x t Input to the source domain-target domain inter-domain semantic segmentation model F inter In the target domain image x, we get t The category segmentation probability P t , and use the category segmentation probability P t Calculate the segmentation probability confidence S T and the target domain pseudo-label Calculate the segmentation probability confidence S t The calculation method is as follows:

[0070]

[0071] Where: ∑ represents the mathematical summation symbol, ∏ represents the mathematical product symbol, and H is the target domain image x t The length of W is the target domain image x t The width of C is the target domain image x t The number of classification categories, P t To transform the target domain image x t Input semantic segmentation model F inter The obtained category segmentation probability (whose size is a matrix of H×W×C), P t (h,w,c) is the category segmentation probability P t The category segmentation probability of the pixel point with coordinates (h, w) and category c,

[0072] ∏ c P t (h,w,c) To calculate the product of the category segmentation probabilities corresponding to each category c of the pixel point with coordinates (h, w).

[0073] Using the category segmentation probability P t Get the target domain pseudo label The way is as follows:

[0074]

[0075] Among them: argmax is the maximum value function, is the category segmentation probability P tThe category with the highest probability of segmentation among the pixels with coordinates (h, w) μ c is the segmentation probability threshold used to generate pseudo labels for category c, is the target domain image x t The segmentation probability chaos of the pixel at coordinates (h, w), v is the segmentation probability chaos threshold used to generate pseudo labels. The calculation method is as follows:

[0076]

[0077] Where: ∏ represents the mathematical product symbol, H is the target domain image x t The length of W is the target domain image x t The width of C is the target domain image x t The number of classification categories, ∏ c P t (h,w,c) To calculate the product of the category segmentation probabilities corresponding to each category c of the pixel point with coordinates (h, w).

[0078] (3) 500 Vaihingen (target) domain images x t The segmentation probability confidence S t Sort in descending order according to the value size, and then sort according to the reliability S of the sorted segmentation probability t The target domain image x t Divide equally into 4 target domain image sets

[0079] (4) Using the Vaihingen (target) domain image subset with the highest segmentation probability confidence and its corresponding pseudo-label subset Source domain-target domain inter-domain semantic segmentation model F inter and a subset of target domain images Iterative training obtains the semantic segmentation model F in the target domain intra .

[0080] The single domain adaptation method in the domain adopted in this embodiment is described by the method based on adversarial learning, but is not limited to the method based on adversarial learning. The method based on adversarial learning requires the semantic segmentation model F in the domain. intra and the discriminator D intra ,The training loss function includes semantic segmentation loss function and adversarial loss function.

[0081] The equation expression of the semantic segmentation loss function is as follows:

[0082]

[0083] Where: X i is the target domain image subset of the i-th part, y i For x i The corresponding pseudo-label subset, K is the number of label categories, F intra is the semantic segmentation model on the target domain, is the indicator function (when k = Y i hour, When k≠Y i hour, Indicator function reference—Zhou Zhihua. Machine Learning [M]. Beijing: Tsinghua University Press, 2016. Main symbol table), represents the mathematical expectation function, For X i Input to Model F intra The result of the kth category in the output results obtained.

[0084] The equation expression of the adversarial loss function is as follows:

[0085]

[0086] Where: X i is the target domain image subset of the i-th part, is the mathematical expectation function, D intra is the target domain discriminator.

[0087] This implementation requires three iterations of in-domain adaptation. First, in the first iteration, a subset of 125 target domain images is and its corresponding pseudo-label subset Add the originally empty training set separately And the corresponding label set Then use the 125-image training set And the corresponding label set and a subset of 125 target domain images Perform adversarial training and use the source domain-target domain semantic segmentation model F inter As the initial target domain semantic segmentation model The segmentation model network structure uses deeplabV3+, the discriminator network structure is a 4-layer CNN, and the learning rate is 10 -4 , the optimization algorithm is adam, and the training stops after 100 epochs. After the training, we get A subset of 125 target domain images Input to the target domain semantic segmentation model In the above example, we get the category segmentation probability And according to the segmentation probability Get the target domain image subset The pseudo-label subset Subset the target domain images and the corresponding pseudo-label subset Add to training set and the corresponding label set Then use the 250-image training set And the corresponding label set and a subset of 125 target domain images And the in-domain semantic segmentation model For adversarial training, the segmentation model network structure uses deeplabV3+, the discriminator network structure is a 4-layer CNN, and the learning rate is 10 -4 , the optimization algorithm is adam, and the training stops after 100 epochs. After the training, we get A subset of 125 target domain images Input to the target domain semantic segmentation model In the above example, we get the category segmentation probability And according to the segmentation probability Get the target domain image subset The pseudo-label subset Subset the target domain images and the corresponding pseudo-label subset Add to training set and the corresponding label set Then use the 375-image training set And the corresponding label set and a subset of 125 target domain images And the in-domain semantic segmentation model For adversarial training, the segmentation model network structure uses deeplabV3+, the discriminator network structure is a 4-layer CNN, and the learning rate is 10 -4 The optimization algorithm is adam, and the training stops after 100 epochs. After the training, the final semantic segmentation model F in the target domain is obtained. intra

[0088] (5) The target domain image x t Input to the target domain semantic segmentation model F intra In the target domain image x, we get t The final segmentation result map.

[0089] Table 1 shows the results obtained by relevant experiments before migration, histogram matching (traditional method), GAN-based inter-domain domain adaptation method, single intra-domain domain adaptation and the iterative intra-domain domain adaptation strategy of the present invention, as well as the precision, recall, F1 and IoU indicators calculated by the true value of the label.

[0090] Table 1

[0091] Before Migration Histogram matching Inter-domain Adaptation Intra-domain Adaptation Iterative intra-domain adaptation precision 0.8387 0.4184 0.8920 0.8899 0.8884 recall 0.1548 0.2847 0.3704 0.4033 0.4226 F1 0.2614 0.3389 0.5234 0.5551 0.5728 IoU 0.1503 0.2040 0.3545 0.3841 0.4013

[0092] It can be seen from the above experimental results that this implementation effectively improves the IoU index of semantic segmentation compared with that before migration, and the improvement reaches 0.2510. At the same time, compared with simple histogram matching, the IoU index of this implementation is also improved by 0.1973; comparing the single intra-domain domain adaptation with the inter-domain domain adaptation, the IoU index of the single intra-domain domain adaptation is improved by 0.0296, which shows that the intra-domain domain adaptation can reduce the intra-domain differences. At the same time, compared with the single intra-domain domain adaptation, the IoU index of the iterative intra-domain domain adaptation is further improved by 0.0172, which shows that the iterative intra-domain domain adaptation can further reduce the intra-domain differences. Therefore, the present invention is of great help in improving the performance of semantic segmentation of cross-satellite remote sensing images.

[0093] The above description of the embodiments is to facilitate the understanding and application of the present invention by those skilled in the art. It is obvious that those skilled in the art can easily make various modifications to the above embodiments and apply the general principles described herein to other embodiments without creative work. Therefore, the present invention is not limited to the above embodiments. Improvements and modifications made by those skilled in the art to the present invention based on the disclosure of the present invention should be within the protection scope of the present invention.

Claims

1. A cross-domain remote sensing image semantic segmentation method based on iterative in-domain adaptation and self-training, comprising the following steps: (1) Using the source domain image x s , source domain label y s , source domain semantic segmentation model F S And the target domain image x t Train the source domain-target domain semantic segmentation model F inter The specific implementation process is as follows: 1.1 Using the source domain image x s and the source domain label y s Train the source domain semantic segmentation model F S ; 1.2 Using the source domain image x s And the target domain image x t Training a source-target domain image bidirectional converter, which includes an image converter in the source→target direction and an image converter in the target→source direction; 1.3 For all the intermediate saved models of the image converters generated in the above training process, a set of optimal results is selected as the image converter G in the source→target direction S→T and the image converter G in the target→source direction T→S ; 1.4 Using Image Converter G S→T The source domain image x s Convert from the source domain to the target domain to obtain the pseudo-target domain image G S→T (x s ); 1.5 Using the pseudo target domain image G S→T (x s ) and the source domain label y s Train the source domain-target domain semantic segmentation model F inter ; (2) The target domain image x t Input to the source domain-target domain inter-domain semantic segmentation model F inter In the target domain image x, we get t The category segmentation probability P t , and then use the category segmentation probability P t The segmentation probability confidence S is calculated by the following formula t and the target domain pseudo-label in: H and W are the target domain image x t The length and width of C is the target domain image x t The number of segmentation categories, Denotes the target domain image x t The pixel point with coordinates (h,w) corresponds to category c i The segmentation probability, c i represents the i-th category, i is a natural number and 1≤i≤C, θ() is a function used to measure the likelihood between the segmentation probabilities of each category of pixel points, represents the target domain pseudo label The category of the pixel with coordinates (h,w), Denotes the target domain image x t The segmentation probability of the pixel point with coordinates (h, w) corresponding to category c, μ c is the segmentation probability threshold corresponding to category c, Denotes the target domain image x t The segmentation probability chaos of the pixel with coordinates (h, w), v is the segmentation probability chaos threshold, and δ() is a function used to measure the degree of chaos between the segmentation probabilities of different categories of pixels; (3) All target domain images x t According to the segmentation probability confidence S t The size is sorted in descending order, and then all the target domain images x are sorted according to the sorting order. t Divide into K target domain image subsets on average K is a natural number greater than 1; (4) Using a subset of target domain images with the highest segmentation probability confidence and its corresponding pseudo-label subset And the source domain-target domain inter-domain semantic segmentation model F inter and a subset of target domain images Iteratively train the semantic segmentation model F in the target domain intra The specific implementation process is as follows: 4.1 Initially, a set of target domain image subsets with the highest segmentation probability confidence and its corresponding pseudo-label subset As a training set and its corresponding label set The source domain-target domain semantic segmentation model F inter As a target domain semantic segmentation model 4.2 Using the training set Label Set Target domain semantic segmentation model and a subset of target domain images Train a semantic segmentation model in the target domain k is a natural number and 2≤k≤K; 4.3 Subsetting the Target Domain Images Input to the target domain semantic segmentation model In the above example, we get the corresponding category segmentation probability. Then use the category segmentation probability Calculate the target domain image subset The pseudo-label subset 4.4 Subsetting the Target Domain Images and its pseudo-label subset Add to the training set and label sets middle; 4.5 Let k = k + 1; 4.6 Repeat steps 4.2 to 4.5 until k = K, and train the semantic segmentation model in the target domain. That is, as the semantic segmentation model F in the target domain intra ; (5) The target domain image x t Input to the target domain semantic segmentation model F intra In the above example, we can get the target domain image x t The final category segmentation probability P and segmentation result map.

Images