Self-training domain adaptation method for remote sensing image semantic segmentation based on pseudo label quality

Abstract

The application belongs to the technical field of semantic segmentation of remote sensing images, and proposes a self-training domain adaptation remote sensing image semantic segmentation method based on pseudo label quality, comprising the following steps: data set preparation, including division of source domain and target domain, and data preprocessing in the same domain; semantic segmentation model initialization; semi-supervised model integration; cross-domain image mixing; local pseudo label quality calculation; and self-training domain adaptation of remote sensing image semantic segmentation. The application can solve the problem of cross-domain prediction performance decline existing in the deep learning-based remote sensing image semantic segmentation algorithm, reduce the influence of false pseudo labels in the self-training method, stabilize the generation process of pseudo labels in the self-training, and finally improve the performance of the neural network model in the data set not contacted.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of semantic segmentation of remote sensing images, and particularly relates to a self-training domain adaptation remote sensing image semantic segmentation method based on pseudo-label quality. BACKGROUND

[0002] Remote sensing image semantic segmentation technology aims to automatically analyze the pixel-level content of acquired remote sensing images and divide them into regions with different land cover class labels, such as roads, vegetation, farmland, and buildings. This technology can extract the distribution of the land cover class of interest and has high practical value in land resource investigation, ecological environment detection, and disaster assessment. In recent years, with the rapid development of deep learning, neural network image semantic segmentation algorithms based on deep learning have gradually become the dominant method for remote sensing image semantic segmentation and have achieved excellent performance.

[0003] Currently, the performance of neural network technology based on supervised learning depends heavily on the size and quality of the training data set. In actual use, there are often differences in the feature distribution of the test data set and the training data set in the field of remote sensing. Specifically, when acquiring remote sensing images, due to factors such as light, weather, and camera settings, different remote sensing image data sets are prone to phenomena such as different objects having similar or identical spectral characteristics and similar objects having different spectral characteristics. The phenomenon of different objects having similar or identical spectral characteristics refers to the fact that different land covers have similar or identical spectral characteristics in certain spectral bands, such as clouds and water bodies having similar spectral characteristics in certain spectral bands. The phenomenon of similar objects having different spectral characteristics refers to the fact that the spectral characteristics of the same land cover change under different environments, such as the spectral characteristics of objects changing under different lighting conditions. This phenomenon, in which the feature distribution of the prediction target differs between different remote sensing data sets due to external conditions, is referred to as a domain gap. Since deep learning relies on the features of the data for learning and prediction, the domain gap phenomenon increases the difficulty of semantic segmentation in different domains for deep learning, thereby affecting the generalization ability of the deep learning model and reducing the prediction performance of deep learning in uncontacted test data sets.

[0004] To alleviate the performance degradation caused by the domain gap, domain adaptation (DA) techniques are proposed. Among them, domain represents a set of images with similar features: in deep learning, the training set and the test set can be regarded as different domains; in the field of remote sensing, different remote sensing datasets can be regarded as different domains. Domain adaptation technology is a technology that improves the generalization ability of the model in the unknown domain by processing the differences between different domains during the training and inference of the deep learning model. At present, domain adaptation technology mainly falls into the following three categories: generative-based domain adaptation technology, adversarial-learning (AL) domain adaptation technology and self-training (ST) domain adaptation technology. In order to realize domain adaptation and knowledge transfer, both the generative-based method and the adversarial-learning method need to introduce an additional network. The former introduces a generation network to translate images, while the latter introduces a discriminant network to assist the learning of the semantic segmentation network, so these two methods are heavily dependent on the performance of the auxiliary network and lack stability. At the same time, the additional neural network not only puts higher requirements on hardware resources, but also leads to the extension of the training time.

[0005] Unlike the first two domain adaptation methods, the self-training domain adaptation method does not rely on any auxiliary network. It follows the principle of "from easy to difficult" in the training process of the neural network, gradually learns the different feature distributions between the two domains, and thus completes the transfer and development of knowledge. Specifically, when using a neural network to predict an unlearned domain, if the confidence of a prediction result is higher than a certain threshold, the self-training domain adaptation technology assumes that the result is correct and can be regarded as a pseudo-label instead of a real label. In order to complete the transfer and development of knowledge, the pseudo-label generated in this round of training will be added to the training set in the next iteration, so that the model learns the knowledge in different domains. Repeat this process until all or most of the data are learned, and the self-training domain adaptation process is complete. However, the current self-learning still has the following two problems:

[0006] 1. How to further reduce the impact of false pseudo-labels: Even if the prediction confidence of some pixels is high enough, it cannot be guaranteed that it is correct, so a suitable mechanism needs to be introduced to distinguish or reduce the weight of false pseudo-labels.

[0007] 2、How to further stabilize the generation process of pseudo labels: self-training domain adaptation technology relies on the generation of pseudo labels, however, with the iterative training of the semantic segmentation network, different outputs may be generated even for the same input, thus introducing noise into the dataset, leading to a large performance change of the model during the training process, i.e., the loss function oscillation phenomenon, therefore a suitable mechanism needs to be introduced to stabilize the generation process of pseudo labels. SUMMARY

[0008] The purpose of the present application is to provide a self-training domain adaptation remote sensing image semantic segmentation method based on the quality of pseudo labels, which aims to solve the problem of cross-domain prediction performance decline existing in deep learning-based remote sensing image semantic segmentation algorithms, while reducing the influence of false pseudo labels in the self-training method, and finally stabilizing the generation process of pseudo labels in self-training, and finally improving the performance of the neural network model in the dataset that has not been contacted.

[0009] The present application solves its technical problems, and the technical scheme adopted is:

[0010] The self-training domain adaptation remote sensing image semantic segmentation method based on the quality of pseudo labels comprises the following steps:

[0011] Data set preparation, including the division of source domain and target domain, and data preprocessing within the same domain;

[0012] Semantic segmentation model initialization;

[0013] Semi-supervised model integration;

[0014] Cross-domain image mixing;

[0015] Local pseudo label quality calculation.

[0016] Semantic segmentation of remote sensing image sub-training domain adaptation.

[0017] As a further optimization, in the division process of the source domain and the target domain, the remote sensing image dataset with existing labels is taken as the source domain, and the unlabeled remote sensing image dataset to be predicted is taken as the target domain, respectively denoted as and Where x R H×W×3 represents an image, y R H×W×C represents the label corresponding to the image, H and W represent the length and width of the image, the number of image channels is 3, and the number of label channels is C, representing a total of C categories, and the subscripts S and T of the image and the label represent that they belong to the source domain and the target domain, respectively.

[0018] As a further optimization, the pixel at the spatial position (h, w) of the source domain image is denoted as and its corresponding label is The label is a one-hot encoding vector, with C bits, only 0 and 1 two states, at any time only one bit is valid bit 1, the rest are 0, and if the label of the pixel is the k-th class, then the rest where c∈[1,...,C] satisfies c≠k.

[0019] As a further optimization, the data preprocessing in the same field includes the following steps:

[0020] The images and labels in the two fields are divided into slices using a sliding window method, with a step size of 512 and a window size of 512x512.

[0021] The images and labels in the two fields are randomly scaled and bilinearly enlarged, and are cropped at a random position to obtain image slices with a resolution of 512x512 and their corresponding labels.

[0022] The slices are randomly subjected to data augmentation operations such as padding, inversion, and image distortion, and are finally normalized. The mean values used when normalizing the three channels are 123.675, 116.28, and 103.53, respectively, and the variances are 58.395, 57.12, and 57.375, respectively.

[0023] As a further optimization, the semantic segmentation model initialization includes the following steps:

[0024] Selecting a semantic segmentation model: the semantic segmentation model uses an image semantic segmentation model in deep learning, which includes DeepLabV3+, U-Net, SegNet, CCNet based on convolutional neural networks, or SegFormer based on Transformer.

[0025] Selecting a pre-trained dataset: there are Potsdam and Vaihingen datasets for unmanned aerial vehicle type remote sensing image datasets, and satellite image datasets such as Gaofen 2 satellite remote sensing dataset. If remote sensing image datasets cannot be used, the ImageNet dataset in the field of computer vision is used for pre-training.

[0026] Pre-training the semantic segmentation model: using the training set in the pre-training dataset to supervise the learning of the semantic segmentation model, and completing the preparation and initialization of the semantic segmentation model after reaching the expected performance indicators in the test set.

[0027] As a further optimization, in the semi-supervised model integration process, the average teacher algorithm in semi-supervised learning is used to integrate the model parameters, and the average teacher strategy contains two semantic segmentation models, which are denoted as student model gθ and teacher model h φ The two models have the same structure, and the difference lies in that the student model g θ is responsible for training, and its model parameters θ participate in the back propagation calculation of the gradient, while the model parameters φ of the teacher model h φ do not participate in the back propagation calculation, but are obtained by weighted average of the parameters θ of the student model.

[0028] As a further optimization, the semi-supervised model ensemble specifically includes the following steps:

[0029] Step 1, record the training round number of the current round as t, and the last round as t-1, where t≥1;

[0030] Step 2, let t=0, and initialize the student model g θ and the teacher model h φ to the pre-trained semantic segmentation model, and then let t=1;

[0031] Step 3, use the student model g θ to train on the source domain , and record the model parameters before gradient update as θ t-1 , and the model parameters after gradient update as θ t ;

[0032] Step 4, update the teacher model parameter φ t using the updated student model parameter θ t , and the specific formula is φ t ←αφ t-1 +(1-α)θ t , where α is a hyperparameter representing the importance of the past teacher model parameter, and is set to 0.99;

[0033] Step 5, use the updated teacher model θ t to complete the subsequent steps of cross-domain image mixing, local pseudo label quality calculation, and remote sensing image semantic segmentation sub-training domain adaptation calculation tasks;

[0034] Step 6, if the training round number t reaches the specified target round number 4000, the training is completed, otherwise, let t←t+1, and repeat steps 3 to 5.

[0035] As a further optimization, the cross-domain image mixing specifically includes the following steps:

[0036] Use the student model g θ to make predictions in the source domain , and obtain the semantic segmentation prediction result , that is

[0037] Using the teacher model h φ In the target domain Make a prediction to get the pseudo label p T Where (·) c Represents the c-th element of the vector in the parentheses, and the calculation formula is as follows:

[0038]

[0039] Semantic segmentation prediction results of the source domain And the class set of the pseudo label p T Take the union of the two class sets to get the set This set represents all the classes that exist in the prediction result and the pseudo label class set, and then randomly select half of the classes to get a new set That is, it satisfies

[0040] According to the set Calculate the mask M, M is a mask with the same size as the image x S , x T Each element is 0 or 1, determined by the label y S of the source domain and the selected class set If the true class of the source domain training set label y S at the (i, j) position belongs to the set , then set the corresponding position of the mask M to 1, otherwise to 0, the specific calculation formula is as follows:

[0041]

[0042] Calculate the mixed image x mix and label y mix , the calculation formula is as follows:

[0043] x mix =M⊙x S +(1-M)⊙x T

[0044] y mix =M⊙y S +(1-M)⊙p T

[0045] Where ⊙ represents the dot product of the matrix, that is, the corresponding positions of the matrix are multiplied.

[0046] As a further optimization, the local pseudo label quality calculation specifically includes the following steps:

[0047] Given the pseudo label quality threshold λ∈(0,1), calculate the high-quality pseudo label indicator function q(h,w), the calculation formula is as follows:

[0048]

[0049] wherein the threshold value λ takes 0.7;

[0050] Given the depth K of the adjacent pseudo-label range, the quality coefficient f(h, w) of each pseudo-label is calculated, and the calculation formula is The depth K takes 3.

[0051] As a further optimization, the semantic segmentation sub-training field of the remote sensing image is adapted, specifically including the following steps:

[0052] Input the source domain image x S , the source domain label y S , the student model g θ , and calculate the loss function L S (x S , y S ) in the source domain, and the calculation formula is as follows:

[0053]

[0054] Input the target domain image x T , the pseudo label p T , the pseudo label quality coefficient f, and the student model g θ , and calculate the loss function L T (x T , p T ) in the target domain, and the calculation formula is as follows:

[0055]

[0056] The sum of the two loss functions L S (x S , y S )+L T (x T , p T ) is gradient backpropagation, and the parameters of the semantic segmentation model are updated. The optimizer is selected as AdamW optimizer, and the hyperparameter settings are as follows: The baseline learning rate is set to 1×10 -4 , the weight decay coefficient is set to 0.01, the exponential decay rate coefficient β1=0.9, β2=0.999, the whole training process is 4000 iterations, and the batch size of each iteration is 8, containing 4 source domain images and 4 target domain images.

[0057] The beneficial effects of the present application are: through the above-mentioned self-training domain adaptation remote sensing image semantic segmentation method based on pseudo label quality, firstly, data set preparation, including the division of source domain and target domain, and data preprocessing in the same domain; secondly, semantic segmentation model initialization; then, semi-supervised model integration; then, cross-domain image mixing; then, local pseudo label quality calculation; finally, semantic segmentation of remote sensing image sub-training domain adaptation. It can solve the problem of cross-domain prediction performance decline existing in the deep learning based remote sensing image semantic segmentation algorithm, at the same time, it can reduce the influence of false pseudo labels in the self-training method, finally, it can stabilize the generation process of pseudo labels in self-training, and finally improve the performance of neural network model in the data set which has not been contacted. BRIEF DESCRIPTION OF DRAWINGS

[0058] Figure 1 The flowchart of the self-training domain adaptation remote sensing image semantic segmentation method based on pseudo label quality of the embodiment of the present application. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical scheme and advantages of the embodiment of the present application clearer, the technical scheme in the embodiment of the present application will be described clearly and completely below in combination with the drawings in the embodiment of the present application. Obviously, the described embodiment is a part of the embodiments of the present application, not all the embodiments. The components of the embodiment of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0060] EMBODIMENT

[0061] The present embodiment provides a self-training domain adaptation remote sensing image semantic segmentation method based on pseudo label quality, which can include the following steps:

[0062] A, data set preparation: responsible for preprocessing the remote sensing image data set, inputting two remote sensing image data sets, and outputting the preprocessed source domain data set and target domain data set.

[0063] B, semantic segmentation model initialization: responsible for the selection and initialization of the semantic segmentation model, inputting a semantic segmentation model and a data set for pre-training, and obtaining a pre-trained semantic segmentation model.

[0064] C, semi-supervised model integration: responsible for stabilizing the generation of pseudo labels, introducing a student model and a teacher model, inputting the weight of the semantic segmentation model after the previous round of training and the image of the target domain, and outputting the pseudo label of the target domain.

[0065] D, cross-domain image mixing: responsible for generating more rich target domain images, inputting the image and label of the source domain, the image of the target domain, the student model, and the teacher model, and outputting the mixed image.

[0066] E. Local pseudo label quality calculation: responsible for completing the calculation and weighting of the pseudo label, inputting the pseudo label of the target domain, and outputting the pseudo label weight after considering the local pseudo label quality.

[0067] F. Self-training domain adaptation technology integration: responsible for completing the self-training domain adaptation process of the remote sensing image semantic segmentation, inputting the image and label in the source domain dataset, the image in the target domain dataset, the pre-trained semantic segmentation model, the pseudo label of the target domain, and the pseudo label weight of the target domain, and outputting the semantic segmentation model applicable to the target domain image without accessing the label of the target domain.

[0068] The training of the final semantic segmentation model needs to be obtained through multiple rounds of iterative updates. The flowchart of each round of iteration is as shown in Figure 1 Two loss values and are obtained after each round of iteration, and the gradient is back-propagated to update the parameters of the semantic segmentation model.

[0069] In a specific application, the specific steps of the self-training domain adaptation remote sensing image semantic segmentation method based on the pseudo label quality provided by the embodiment are as follows:

[0070] A. Dataset preparation:

[0071] The preparation of the dataset mainly includes two parts: the division of the source domain and the target domain, and the data preprocessing in the same domain.

[0072] A1. Division of the source domain and the target domain:

[0073] In the domain adaptation task, the data with known labels is usually called the source domain, and the data without known labels is called the target domain. The source domain and the target domain have the same task, but the image feature distributions are different. In actual use, the remote sensing image dataset with labels can be used as the source domain, and the remote sensing image dataset to be predicted without labels can be used as the target domain, which are denoted as and wherein x e R H×W×3 represents an image, y e R H×W×C represents the label corresponding to the image, H and W represent the length and width of the image, the number of image channels is 3, and the number of label channels is C, representing a total of C categories, and the subscripts S and T of the image and the label represent that they belong to the source domain and the target domain, respectively. It should be noted that the ground resolution of the image in the source domain dataset needs to be as close as possible to the ground resolution of the image in the target domain.

[0074] Here, the pixel at the spatial position (h, w) of the source domain image is denoted as and the corresponding label is denoted as The label is a one-hot encoding vector, which has C bits and only two states of 0 and 1. At any time, only one bit is 1 and the rest are 0. Let (·) c be the c-th element of the vector in the parentheses. If the label of the pixel is the k-th class, then the rest where c [1,..., C] satisfies c≠k.

[0075] A2, data preprocessing in the same domain:

[0076] Data augmentation is a common image preprocessing method in computer vision. It can generate more data based on limited data, increase the amount and diversity of training samples, and reduce overfitting and enhance the generalization of the model. The same image preprocessing method is used for images in the same domain in this embodiment:

[0077] The images and labels in the two domains are divided by sliding window, with a step size of 512 and a window size of 512x512.

[0078] Randomly perform bilinear enlargement on the images and labels in the two domains, and crop at a random position to obtain image slices with a resolution of 512x512 and their corresponding labels.

[0079] Randomly use padding, inversion, image distortion, etc. Data augmentation operations are performed on these slices, and finally normalized. The mean values used when normalizing the three channels are 123.675, 116.28, and 103.53, respectively, and the variances are 58.395, 57.12, and 57.375, respectively.

[0080] B, semantic segmentation model initialization:

[0081] Pre-training technology is a common transfer technology in deep learning, which trains the model on a large dataset first, and then fine-tunes it on a small dataset of downstream tasks to improve the performance of the model on the small dataset. The advantage of pre-training technology is that it does not require the source domain and the target domain to have the same task. For example, a daily image dataset can be used for pre-training to improve the performance of the semantic segmentation model on the remote sensing image dataset. At the same time, even when training on a large dataset, pre-training technology has positive significance. Compared with randomly initializing and training the network from scratch, using a pre-trained model for initialization can greatly speed up the convergence of the neural network and reduce the possibility of fluctuations in the training process.

[0082] The specific implementation steps of the semantic segmentation model initialization are as follows:

[0083] selecting a semantic segmentation model, the semantic segmentation model can use an image semantic segmentation model commonly used in deep learning, such as DeepLabV3+, U-Net, SegNet, CCNet based on a convolutional neural network, or SegFormer based on a Transformer;

[0084] selecting a pre-trained dataset, the type of the pre-trained dataset is the same as that of the source domain and the target domain, which is the best case. At present, among the public datasets, there are Potsdam, Vaihingen, etc. Datasets for unmanned aerial vehicle type remote sensing image datasets, satellite image datasets such as Gaofen No. 2 satellite remote sensing dataset, if the remote sensing image dataset cannot be used, the ImageNet dataset commonly used in the field of computer vision can be used for pre-training;

[0085] pre-training the semantic segmentation model, using the training set in the pre-training dataset to perform the usual supervised learning on the semantic segmentation model, and completing the preparation and initialization of the semantic segmentation model after reaching a high performance index (such as IoU, PA, F1-score) in the test set.

[0086] C, semi-supervised model integration:

[0087] In each round of training, the semantic segmentation model first updates the parameters according to the training results of the last round, and then uses the new parameters to predict all the pseudo labels in this round. In order to prevent the noise data in each round from disturbing the generation of pseudo labels too much and causing network fluctuations, affecting the results of network training, the Mean Teacher algorithm in semi-supervised learning is used to integrate the model parameters in this embodiment, thereby enhancing the robustness of the model. The Mean Teacher strategy includes two semantic segmentation models, denoted as student model g θ and teacher model h φ The structures of the two models are exactly the same, and the difference lies in that the student model g θ is responsible for training, and its model parameters θ participate in the back propagation calculation of the gradient, while the model parameters φ of the teacher model h φ do not participate in the back propagation calculation, but are obtained by weighted averaging the parameters θ of the student model.

[0088] Therefore, the semi-supervised model integration can specifically include the following steps:

[0089] Step 1, let the training round number of the current round be t, the last round is t-1, where t≥1;

[0090] Step 2, let t=0, initialize the student model g θ and the teacher model h φ to the pre-trained semantic segmentation model, and then let t=1;

[0091] Step 3, using the student model g θ In the source domain , the model parameters before gradient update are denoted as θ t-1 , and the model parameters after gradient update are denoted as θ t ;

[0092] Step 4, using the updated student model parameter θ t to update the teacher model parameter φ t , the specific formula is φ t ← αφ t-1 + (1-α)θ t , where α is a hyperparameter representing the importance of the past teacher model parameters, and is set to 0.99;

[0093] Step 5, using the updated teacher model θ t to complete subsequent steps of cross-domain image mixing, local pseudo label quality calculation, and remote sensing image semantic segmentation sub-training domain adaptation computing tasks;

[0094] Step 6, if the training round t reaches the specified target round 4000, the training is completed, otherwise, let t <- t+1, and repeat steps 3 to 5.

[0095] D. Cross-domain image mixing:

[0096] In the domain adaptation task, cross-domain image mixing helps to further increase the diversity of data, thereby improving the generalization of the model.

[0097] In this embodiment, the cross-domain image mixing can specifically include the following steps:

[0098] Using the student model g θ to make predictions in the source domain , obtaining semantic segmentation prediction results , i.e.

[0099] Using the teacher model h φ to make predictions in the target domain , obtaining pseudo labels p T , where (·) c represents the c-th element of the vector in the parentheses, and the calculation formula is as follows:

[0100]

[0101] Taking the union of the class sets of the semantic segmentation prediction results of the source domain and the pseudo labels p T of the target domain, the set The set represents all categories existing in the two categories of prediction results and pseudo labels, and then a new set is obtained by randomly selecting half of the categories That is, satisfying

[0102] According to the set The mask M is calculated, M is a mask with the same size as the image x S , x T , and each element is 0 or 1, which is determined by the label y S of the source domain and the selected category set If the true category of the source domain training set label y S at the (i, j) position belongs to the set , the mask M is set to 1 at the corresponding position, otherwise it is set to 0, and the specific calculation formula is as follows:

[0103]

[0104] The mixed image x mix and label y mix are calculated, and the calculation formula is as follows:

[0105] x mix =M⊙x S +(1-M)⊙x T

[0106] y mix =M⊙y S +(1-M)⊙p T

[0107] Where represents the dot product of the matrix, that is, the corresponding positions of the matrix are multiplied.

[0108] E. Local pseudo label quality calculation

[0109] In this embodiment, the main idea of local pseudo label quality is as follows: low-quality pseudo labels are usually located at the junction of ground objects, where the texture is more complex, and the label may also be wrong, and it is usually easy to get discrete pseudo labels, so the pseudo label quality should be relatively low; high-quality pseudo labels usually appear in the interior of ground objects, where the texture is similar to the surrounding, and the label is not easy to be wrong, and it is easy to be predicted as aggregated pseudo labels of the same kind, so the pseudo label quality should be relatively high. The specific calculation idea is to determine the pseudo label quality of each pixel point from the proportion of high-quality pseudo labels of the surrounding pixels of the pixel point, and the result can be efficiently obtained by convolution operation.

[0110] It should be pointed out that in this embodiment, the local pseudo label quality calculation can specifically include the following steps:

[0111] Given the pseudo-label quality threshold λ∈(0, 1), calculate the high-quality pseudo-label indicator function q(h, w), the calculation formula is as follows:

[0112]

[0113] Wherein, in actual use, the threshold λ can be 0.7;

[0114] Given the depth K of the adjacent pseudo-label range, calculate the quality coefficient f(h, w)∈[0, 1] of each pseudo label, the calculation formula is In actual use, the depth K can be 3.

[0115] F, self-training domain adaptation technology integration:

[0116] This step uses self-training technology, combined with the aforementioned semi-supervised model integration, cross-domain image mixing, local pseudo label quality, etc. Without access to the target domain label, it is responsible for training the semantic segmentation model suitable for the target domain image.

[0117] It should be pointed out that in the present embodiment, the semantic segmentation sub-training domain adaptation of remote sensing images can specifically include the following steps:

[0118] Input source domain image x S , source domain label y S , student model g θ , calculate the loss function L S (x S , y S ) in the source domain, the calculation formula is as follows:

[0119]

[0120] Input target domain image x T , pseudo label p T , pseudo label quality coefficient f, student model g θ , calculate the loss function L T (x T , p T ) in the target domain, the calculation formula is as follows:

[0121]

[0122] The sum of the two loss functions L S (x S , y S )+L T (x T , p T) Gradient return is performed, parameters of the semantic segmentation model are updated, and the AdamW optimizer is selected as the optimizer, and the hyperparameter settings are as follows: the base learning rate is set to 1x10 -4 , the weight decay coefficient is set to 0.01, the exponential decay rate coefficient β1=0.9, β2=0.999, the whole training process has 4000 iterations, and the batch size of each iteration is 8, containing 4 source domain images and 4 target domain images.

[0123] The above are only preferred embodiments of the present application and are not used to limit the present application. For those skilled in the art, the present application can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A self-training domain adaptation method for semantic segmentation of remote sensing images based on pseudo-label quality, characterized in that, Includes the following steps: Dataset preparation includes dividing the source and target domains, as well as data preprocessing within the same domain; Semantic segmentation model initialization; Semi-supervised model ensemble; Cross-domain image blending; Local pseudo-label quality calculation; Semantic segmentation of remote sensing images: training domain adaptation; In the process of dividing the source and target domains, the labeled remote sensing image dataset is used as the source domain, and the unlabeled remote sensing image dataset to be predicted is used as the target domain, denoted as […]. and ,in Representative image, The labels represent the corresponding images. H and W represent the length and width of the image. The image has 3 channels, while the label has C channels, representing a total of C categories. The subscripts S and T of the image and label represent that they belong to the source domain and the target domain, respectively. Describe the spatial location of the source domain image The pixel at that location is Its corresponding tag is , The label is a one-hot encoded vector with C bits, having only two states: 0 and 1. At any given time, only one bit is a valid 1, and the rest are 0. Let be the value of the label. Let c be the c-th element of the vector within the parentheses. If the pixel... If the label is of class k, then ,the remaining ,in satisfy .

2. The self-trained domain-adaptive remote sensing image semantic segmentation method based on pseudo-label quality according to claim 1, characterized in that, The data preprocessing within the same domain includes the following steps: The image and label in the two domains are segmented using a sliding window method with a step size of 512 and a window size of 512×512. The images and labels in the two domains are subjected to bilinear scaling at random ratios, and cropped at random positions to obtain image slices with a resolution of 512×512 and their corresponding labels. Data augmentation operations such as padding, inversion, and image distortion were randomly applied to these slices, and finally normalized. The mean values ​​used for normalizing the three channels were 123.675, 116.28, and 103.53, and the variances were 58.395, 57.12, and 57.375, respectively.

3. The self-trained domain-adaptive remote sensing image semantic segmentation method based on pseudo-label quality according to claim 1, characterized in that, The initialization of the semantic segmentation model includes the following steps: Selecting a semantic segmentation model: The semantic segmentation model uses an image semantic segmentation model in deep learning, including DeepLabV3+, U-Net, SegNet, CCNet based on convolutional neural networks, or SegFormer based on Transformer. Choose a pre-training dataset: For drone-type remote sensing image datasets, there are Potsdam and Vaihingen datasets; for satellite image datasets, there are satellite remote sensing datasets such as Gaofen-2. If a remote sensing image dataset is not available, use the ImageNet dataset from the field of computer vision for pre-training. Pre-training the semantic segmentation model: Supervised learning of the semantic segmentation model is performed using the training set in the pre-training dataset. After the expected performance metrics are achieved in the test set, the preparation and initialization of the semantic segmentation model are completed.

4. The self-trained domain-adaptive remote sensing image semantic segmentation method based on pseudo-label quality according to claim 1, characterized in that, In the semi-supervised model integration process, the average teacher algorithm in semi-supervised learning is used to integrate the model parameters. The average teacher strategy includes two semantic segmentation models, denoted as the student model. Teacher Model The two models have the same structure; the difference lies in the student model. Responsible for the model parameters during training. It will participate in the backpropagation calculation of the gradient, while the teacher model Model parameters It does not participate in backpropagation calculations, but is determined by the parameters of the student model. The weighted average is obtained.

5. The self-trained domain-adaptive remote sensing image semantic segmentation method based on pseudo-label quality according to claim 4, characterized in that, The semi-supervised model ensemble specifically includes the following steps: Step 1: Let t be the number of training rounds in the current round and t-1 be the number of the previous round, where t≥1; Step 2: Let t=0, and set the student model... Teacher Model Initialize with a pre-trained semantic segmentation model, then set t=1; Step 3: Use the student model In the source domain The model parameters before gradient update are denoted as follows: The model parameters after gradient update are denoted as ; Step 4: Use the updated student model parameters Update teacher model parameters The specific formula is as follows ,in The hyperparameter, representing the importance of past teacher model parameters, is set to 0.99; Step 5: Use the updated teacher model The subsequent steps include cross-domain image mixing, local pseudo-label quality calculation, and semantic segmentation of remote sensing images, which are computational tasks for training domain adaptation. Step 6: If the number of training rounds t reaches the specified target number of rounds 4000, then the training is complete; otherwise, let... Repeat steps 3 through 5.

6. The self-trained domain-adaptive remote sensing image semantic segmentation method based on pseudo-label quality according to claim 4, characterized in that, The cross-domain image fusion specifically includes the following steps: Using the student model In the source domain The semantic segmentation prediction result is obtained by performing prediction in the middle. ,Right now ; Using the teacher model In the target domain Prediction is performed to obtain pseudo-labels. ,in The c-th element of the vector within the parentheses is calculated using the following formula: ； Semantic segmentation prediction results of the source domain pseudo tags of the target domain Union of the category sets yields the set This set represents all categories that exist in both the prediction result and pseudo-label category sets. Then, a new set is obtained by randomly selecting half of the categories from this set. That is, satisfying ; According to the set Calculate the mask M, where M is a representation of the image. Masks of the same size, each element being either 0 or 1, determined by the labels of the source domain. and the selected category set Decision, if the source domain training set labels exist The actual category at the location belongs to the set. If the value is true, the corresponding position of mask M is set to 1; otherwise, it is set to 0. The specific calculation formula is as follows: ； Calculate the blended image and tags The calculation formula is as follows: in This represents the dot product of matrices, which is the multiplication of corresponding elements in the matrix.

7. The self-trained domain-adaptive remote sensing image semantic segmentation method based on pseudo-label quality according to claim 1, characterized in that, The local pseudo-label quality calculation specifically includes the following steps: Given a pseudo-label quality threshold Calculate the high-quality pseudo-label indicator function The calculation formula is as follows: Among them, threshold Take 0.7; Given the depth K of the neighboring pseudo-label range, calculate the quality coefficient of each pseudo-label. The calculation formula is: The depth K is 3.

8. The self-trained domain-adaptive remote sensing image semantic segmentation method based on pseudo-label quality according to claim 1, characterized in that, The semantic segmentation training domain adaptation of the remote sensing image specifically includes the following steps: Input source domain image Source domain tags Student Model Calculate the loss function in the source domain. The calculation formula is as follows: ； Input target domain image Pseudo-labels False label quality coefficient Student Model Calculate the loss function in the target domain. The calculation formula is as follows: ； The sum of two loss functions Gradient backpropagation is performed to update the parameters of the semantic segmentation model. The optimizer chosen is the AdamW optimizer, with its hyperparameters set as follows: the baseline learning rate is set to... The weight decay coefficient is set to 0.01, and the exponential decay rate coefficient is... The entire training process consists of 4000 iterations, with a batch size of 8 for each iteration, containing 4 source domain images and 4 target domain images.

Images