Remote sensing unsupervised domain adaptive semantic segmentation method for multi-task collaborative learning

By utilizing the hybrid consistency learning of cross-task feature correlation learning and pseudo-label guidance in the multi-task collaborative learning method in the remote sensing field, the problems of semantic ambiguity and confusing classification in the cross-domain segmentation of remote sensing are solved, and the remote sensing data set is significantly improved Segmentation accuracy.

CN118968064AActive Publication Date: 2024-11-15CHINA UNIV OF GEOSCIENCES (WUHAN)

Patent Information

Application Number
CN202411027659.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-30
Publication Date
2024-11-15
Estimated Expiration
2044-07-30

AI Technical Summary

Technical Problem

The prior art cannot accurately segment in the cross-domain segmentation task in the field of remote sensing, especially when the amount of optical image information is insufficient and the phenomenon of heterogeneous objects and heterogeneous objects are common, resulting in semantic ambiguity and confusing classification.

Method used

A remote sensing unsupervised domain adaptive semantic segmentation method for multi-task collaborative learning is proposed, and the pseudo-label-guided class-level alignment is optimized through the cross-task feature correlation learning of student model and teacher model, mixed consistency learning of pseudo-label-guided and entropy-guided category-level alignment. Quality and domain adaptive strategies to improve the cross-domain segmentation accuracy of the model.

Benefits of technology

By closely linking elevation features and semantic features, improving the accuracy of cross-domain segmentation tasks is significantly improved, and segmentation accuracy of challenging scenarios in remote sensing datasets, such as shadowed areas and similar spectral regions with significant height differences.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118968064A_ABST
    Figure CN118968064A_ABST
Patent Text Reader

Abstract

The invention relates to the field of remote sensing image processing, and discloses a multi-task collaborative learning-based remote sensing unsupervised domain self-adaptive semantic segmentation method, which comprises the following steps of: sending task-specific semantic features and elevation features into a cross-task feature association learning module, potentially learning correlation among different tasks, and obtaining enhanced task-specific feature representation. And the quality of the pseudo-tag is improved by using a pseudo-tag guided hybrid consistency learning module, and global domain alignment is promoted. And meanwhile, the separability of categories difficult to classify is improved through entropy-guided category level alignment. According to the method, multi-task learning and remote sensing domain self-adaptive semantic segmentation are combined, elevation information is fully utilized to assist cross-domain ground object classification of remote sensing, and the cross-domain segmentation precision of difficult-to-classify areas of remote sensing images is remarkably improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of remote sensing image processing, and in particular to a remote sensing unsupervised domain adaptive semantic segmentation method based on multi-task collaborative learning. Background Art

[0002] Many tasks in the field of remote sensing use optical images as input sources, such as image classification, target detection, semantic segmentation, elevation estimation, edge estimation, etc. By setting different tasks and supervision information, deep learning models can learn different types of features to generate specific prediction results. For example, in the semantic segmentation task, the neural network aims to establish the correlation between semantic features and object category information, thereby generating pixel-level object classification results. In the elevation estimation task, the neural network pays more attention to the correlation between elevation features and elevation information, aiming to predict the height value of the object. Therefore, there are differences in the focus of neural networks in different tasks. By effectively combining different tasks, cross-task feature representations can be implicitly learned to improve the performance of the model. Existing research usually uses multi-task learning technology to combine different representations of multiple tasks. Simply put, multi-task learning is to concentrate different visual tasks on one model for training and prediction, so as to better learn the correlation between different tasks in a collaborative way. In this way, different tasks can learn from each other's useful experience and knowledge, thereby improving the overall performance of the model.

[0003] Due to the insufficient amount of information provided by optical images, the phenomenon of different objects with the same spectrum and different spectra of the same object is common in various remote sensing datasets. Neural networks rely heavily on optical images when learning semantic features, so this phenomenon can easily cause semantic ambiguity, and it is more likely to cause confusion in the classification of different objects in cross-domain segmentation tasks. Simply adjusting the convolutional layer structure of the network or enhancing the feature learning ability of deep neural networks cannot fundamentally solve the limitations of the data itself. To overcome this limitation, there are two strategies to consider: one is to use additional information to reduce the negative impact of different objects with the same spectrum and different spectra of the same object; the other is to promote the semantic category-level alignment of objects and enhance classification guidance. In order to obtain additional information to make up for the shortcomings of single data, existing semantic segmentation research usually uses multi-task learning methods to obtain and fuse implicit feature representations in supervision information, and further explore the correlation between different tasks to improve model performance, which provides many theoretical foundations and practical cases. However, this type of method is rarely used in cross-domain segmentation of remote sensing, ignoring the challenging scenes in remote sensing datasets, and ignoring the effective combination of multi-task learning technology and domain adaptation strategy. Summary of the invention

[0004] The purpose of the present invention is to propose a remote sensing unsupervised domain adaptive semantic segmentation method based on multi-task collaborative learning, so as to solve the technical problem that the prior art cannot accurately segment the remote sensing images when performing cross-domain segmentation for specific classification scenarios.

[0005] A remote sensing unsupervised domain adaptive semantic segmentation method based on multi-task collaborative learning, comprising the following steps:

[0006] S1: Obtain remote sensing image datasets of the source domain and the target domain, crop the remote sensing image datasets of the source domain and the target domain to obtain elevation images, remote sensing images, and semantic label images.

[0007] S2: Build a semantic segmentation model, which includes a student model and a teacher model. Both the student model and the teacher model include an initial feature extraction module, a cross-task feature association learning module, and a final classifier.

[0008] S3: Obtain remote sensing images and elevation images of mixed samples;

[0009] Input the target domain remote sensing image into the teacher model to obtain the labels of the target domain samples and the pseudo labels of the mixed samples;

[0010] Input the source domain remote sensing image and the enhanced mixed remote sensing image into the student model to obtain the final elevation estimation results of the source domain samples and the final elevation estimation results of the mixed samples, the final semantic segmentation results of the source domain samples and the final semantic segmentation results of the mixed samples;

[0011] S4: Construct a domain adaptation module, which includes a pseudo-label-guided hybrid consistency learning module and an entropy-guided category-level alignment module. The pseudo-label-guided hybrid consistency learning module uses the pseudo-labels generated above to supervise the student model's learning of mixed samples; the entropy-guided category-level alignment module weights the entropy map and the mean entropy of each category into the segmentation result to guide fine-grained category-level alignment;

[0012] S5: The semantic segmentation loss of the source domain samples is calculated based on the source domain sample segmentation results and the semantic label image. The mixed pseudo-label is used as the supervision information. The semantic segmentation loss and weighted cross entropy loss of the mixed samples are calculated using the semantic segmentation results of the mixed samples, and the overall semantic segmentation loss is calculated.

[0013] S6: Calculate the berHu loss by adding the elevation estimation value and the elevation truth value of the source domain sample and the mixed sample to obtain the elevation estimation loss of the source domain sample and the mixed sample, and calculate the overall elevation estimation loss;

[0014] S7: Weight the entropy map and the mean entropy of each class to the prediction results of the source domain samples and the mixed samples to promote the semantic alignment of the data distribution of the source domain and the target domain; use the source domain true value and the mixed pseudo-label as supervision information, and calculate the entropy loss with the semantic segmentation result after entropy value weighting;

[0015] S8: Train the model according to the overall loss and update the model parameters. Input the remote sensing image dataset of the target domain into the trained student model to obtain the final semantic segmentation result.

[0016] A storage medium stores instructions and data for implementing a remote sensing unsupervised domain adaptive semantic segmentation method for multi-task collaborative learning.

[0017] A remote sensing unsupervised domain adaptive semantic segmentation device for multi-task collaborative learning comprises: a processor and the storage medium; the processor loads and executes instructions and data in the storage medium to implement a remote sensing unsupervised domain adaptive semantic segmentation method for multi-task collaborative learning.

[0018] The beneficial effects provided by the present invention are: using cross-task feature association learning, fully mining and optimizing cross-task feature representation through continuous cross-feature enhancement units, closely associating elevation features and semantic features to improve the accuracy of cross-domain segmentation tasks using implicit elevation information. In addition, the present invention optimizes the quality of pseudo-labels, and further uses pseudo-label-guided hybrid consistency learning and entropy-guided category-level alignment to narrow the domain gap between the source domain and the target domain, promote good transfer of source domain knowledge, and greatly enhance the cross-domain capabilities of the model. The present invention greatly improves the segmentation accuracy of challenging scenes in remote sensing datasets, such as shadow areas, boundary areas, and similar spectral areas with significant height differences. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a schematic flow chart of the method of the present invention;

[0020] Figure 2 It is a schematic diagram of the structure of the cross-task feature association learning module of the present invention;

[0021] Figure 3 is a schematic diagram of the attention fusion module in the cross-feature enhancement unit;

[0022] Figure 4 It is a working schematic diagram of the hardware device of the present invention. DETAILED DESCRIPTION

[0023] To make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0024] Before formally describing the present invention, the scheme of the present invention is first generally described for easy understanding.

[0025] Please refer to Figure 1 , Figure 1 It is a schematic flow diagram of the method of the present invention;

[0026] The present invention provides a remote sensing unsupervised domain adaptive semantic segmentation method based on multi-task collaborative learning, comprising the following steps:

[0027] S1: Obtain remote sensing image datasets of the source domain and the target domain, crop the remote sensing image datasets of the source domain and the target domain to obtain elevation images, remote sensing images, and semantic label images.

[0028] It should be noted that, specifically, the remote sensing image datasets used in the source domain and the target domain are the ISPRS2D semantic segmentation benchmark datasets, where the source domain dataset is the Potsdam IRRG dataset, and the target domain dataset is the VaihingenIRRG dataset. The semantic labels contain six types of objects, namely buildings, impervious surfaces, trees, low vegetation, clutter, and cars.

[0029] Crop the elevation images and high-resolution remote sensing images of the source and target domains, and crop the semantically labeled images of the source domain;

[0030] According to actual needs, the present invention crops the remote sensing images, elevation images, and semantic label images of the Potsdam IRRG dataset with a fixed step size of 512 pixels, and obtains 4598 images with a pixel size of 512×512;

[0031] The present invention crops the remote sensing images, elevation images and semantic label images of the Vaihingen IRRG dataset with a fixed step size of 256 pixels, and its semantic labels are only used to evaluate the model performance.

[0032] S2: Build a semantic segmentation model, which includes a student model and a teacher model. Both the student model and the teacher model include an initial feature extraction module, a cross-task feature association learning module, and a final classifier.

[0033] It should be noted that, in step S2:

[0034] The initial feature extraction module includes: a parameter sharing encoder, an elevation estimation bottleneck, a semantic segmentation bottleneck, an initial elevation classifier and an initial semantic classifier; the initial feature extraction module is used to extract initial features and generate initial semantic segmentation results and initial elevation estimation prediction results;

[0035] As an embodiment, the parameter sharing encoder adopts the ResNet101 network, and the semantic segmentation bottleneck and the elevation estimation bottleneck are both composed of two consecutive residual blocks, including two 1×1 convolutional layers and one 3×3 convolutional layer; the initial elevation classifier and the initial semantic classifier are composed of 1×1 convolutional layers.

[0036] Specifically, the initial semantic features and initial elevation features are obtained through the above-mentioned initial feature extraction module, and the initial classifier is used to generate the corresponding initial semantic segmentation prediction results and initial elevation estimation prediction results.

[0037] Please refer to Figure 2-Figure 3 , Figure 2 It is a schematic diagram of the structure of the cross-task feature association learning module of the present invention. Figure 3 It is a schematic diagram of the attention fusion module in the cross-feature enhancement unit.

[0038] It should be noted that the cross-task feature association learning module consists of two consecutive cross-feature enhancement units and two task-specific gating structures; the cross-task feature association learning module is used to generate adaptive fusion features based on the initial features;

[0039] The initial feature is input into two consecutive cross-feature enhancement units to obtain the intermediate feature and the final feature in turn. The initial feature F init and the intermediate feature F mid Merge through channels and add to the final feature F last The original features and complementary features are tightly integrated to obtain the adaptive fusion feature F. fuse The fusion operation can be expressed as follows:

[0040]

[0041] Among them, conv 1×1 (g) represents the channel dimensionality reduction operation, which reduces the number of channels to half of the original number of channels, [g] represents the channel merging operation, and sigmoid(g) is the activation function.

[0042] The cross-feature enhancement unit consists of two task-specific ASPPs and a cross-attention fusion block. The cross-attention fusion block includes a channel attention module, a spatial attention module, and a nested fusion gating structure. This module is used to cross-integrate task features, task salient features, and cross-task salient features to fully enhance the feature representation of different tasks. For the semantic segmentation task, the process can be expressed as:

[0043]

[0044] where conv 1×1 (g) represents the channel dimensionality reduction operation, which reduces the number of channels to half of the original number, [g] represents the channel merging operation, and sigmoid(g) is the activation function. Indicates significant elevation features, Represents the initial semantic features.

[0045] The final classifier is used to generate final semantic segmentation and elevation estimation prediction results based on the adaptive fusion features.

[0046] As an embodiment, the decoder of the Deeplabv2 network is used as the final classifier.

[0047] S3: Obtain remote sensing images and elevation images of mixed samples;

[0048] Input the target domain remote sensing image into the teacher model to obtain the labels of the target domain samples and the pseudo labels of the mixed samples; input the source domain remote sensing image and the enhanced mixed remote sensing image into the student model to obtain the final elevation estimation results of the source domain samples and the final elevation estimation results of the mixed samples, the final semantic segmentation results of the source domain samples and the final semantic segmentation results of the mixed samples;

[0049] It should be noted that step S3 is specifically as follows:

[0050] S31: Generate a mixed mask using the ClassMix method, and select the corresponding areas of the source domain and target domain samples according to the mask to generate a remote sensing image of the mixed sample;

[0051] S32: Input the target domain remote sensing image into the teacher model to generate pseudo labels; combine the mixed mask, the target domain pseudo labels and the source domain label truth values ​​to generate pseudo labels for the mixed samples;

[0052] S33: Data augmentation for mixed samples, including random horizontal flipping, vertical flipping, color jittering, and Gaussian noise;

[0053] S34: inputting the remote sensing images of the source domain and the target domain into the initial feature extraction module, generating initial features through the parameter sharing encoder, and generating initial semantic segmentation and elevation estimation prediction results using the elevation estimation bottleneck, the semantic segmentation bottleneck and the initial semantic classifier;

[0054] S35: The initial features are fed into the cross-task feature association learning module, and the final semantic and elevation adaptive fusion features are generated through the cross-feature enhancement unit and the task-specific gating structure;

[0055] S36: Input the adaptive fusion features into the final classifier to generate the final semantic segmentation and elevation estimation prediction results.

[0056] S4: Construct a domain adaptation module, which includes a pseudo-label-guided hybrid consistency learning module and an entropy-guided category-level alignment module. The pseudo-label-guided hybrid consistency learning module uses the pseudo-labels generated above to supervise the student model's learning of mixed samples; the entropy-guided category-level alignment module weights the entropy map and the mean entropy of each category into the segmentation result to guide fine-grained category-level alignment;

[0057] It should be noted that step S4 is specifically as follows:

[0058] S41: Construct a pseudo-label-guided hybrid consistency learning module to balance the model's feature representation capability and pseudo-label quality; the pseudo-label-guided hybrid consistency learning module adopts an incremental pseudo-label screening strategy, dynamically adjusts the current threshold according to the given minimum threshold, maximum threshold, total number of iterations and current number of iterations, and screens pseudo-labels that meet the conditions;

[0059] As an embodiment, the specific calculation process of step S41 is as follows:

[0060]

[0061] in is the pseudo label after filtering, Y soft is the original pseudo label of the target domain, σ(Y soft ) represents the pseudo label Y soft Prediction probability plot.

[0062] S42: Construct an entropy-guided category-level alignment module; calculate the entropy map based on the final semantic segmentation prediction results and labels, and use the entropy map to calculate the average entropy value of each category, followed by normalization.

[0063] As an example, the calculation process of the entropy map and the mean entropy of each category in step S42 is as follows:

[0064]

[0065] Where P represents the predicted probability map, W, H and C are the length, width and number of channels of the probability map P respectively, and Y S Represents a semantic tag, num c Represents label Y S The number of pixels belonging to the cth class in .

[0066] S5: The semantic segmentation loss of the source domain samples is calculated based on the source domain sample segmentation results and the semantic label image. The mixed pseudo-label is used as the supervision information. The semantic segmentation loss and weighted cross entropy loss of the mixed samples are calculated using the semantic segmentation results of the mixed samples, and the overall semantic segmentation loss is calculated.

[0067] It should be noted that step S5 is specifically as follows:

[0068] S51: Calculate the semantic segmentation loss of the source domain sample according to the initial semantic segmentation result of the source domain sample, the final semantic segmentation result and the source domain semantic label image;

[0069] The semantic segmentation loss of source domain samples is as follows:

[0070]

[0071] in represents the segmentation loss of the source domain, and Represents the initial segmentation prediction and final segmentation prediction results of the source domain samples.

[0072] S52: Calculate the semantic segmentation loss of the mixed sample according to the initial semantic segmentation result, the final semantic segmentation result and the mixed pseudo-label of the mixed sample;

[0073] The semantic segmentation loss of mixed samples is as follows:

[0074]

[0075] in, is the pseudo label of the mixed sample, and Represents the initial segmentation prediction and final segmentation prediction results of the mixed sample.

[0076] S53: Calculate the penalty weight according to the adaptation difficulty of the elevation task, and further calculate the weighted cross entropy loss of the mixed samples.

[0077] The weighted cross entropy loss of mixed samples is as follows:

[0078]

[0079] Where W M is the highly adaptive weight of the mixed sample, X M is a mixed sample, is the pseudo label of the mixed sample, P M is the predicted probability map of the mixed sample, W, H and C are the probability maps P M The length, width and number of channels, G S () and G T () represent the elevation estimation classifiers of the source domain and the target domain respectively, E M Represents the true value of the elevation of the mixed sample.

[0080] S6: Calculate the berHu loss by adding the elevation estimation value and the elevation truth value of the source domain sample and the mixed sample to obtain the elevation estimation loss of the source domain sample and the mixed sample, and calculate the overall elevation estimation loss;

[0081] It should be noted that step S6 is specifically as follows:

[0082] Get the elevation estimation values ​​of the source domain samples and the mixed samples, use the elevation estimation values ​​and the true elevation values ​​of the source domain samples and the mixed samples to calculate the berHu loss, get the elevation estimation loss of the source domain samples and the mixed samples, sum the elevation estimation losses of all source domain samples and the mixed samples to get the overall elevation estimation loss.

[0083] S7: Weight the entropy map and the mean entropy of each class to the prediction results of the source domain samples and the mixed samples to promote the semantic alignment of the data distribution of the source domain and the target domain; use the source domain true value and the mixed pseudo-label as supervision information, and calculate the entropy loss with the semantic segmentation result after entropy value weighting;

[0084] Step S7 is specifically as follows:

[0085] S71: Obtain prediction results of source domain samples and mixed samples; perform weighted processing on the prediction results of source domain samples and mixed samples, the calculated entropy map and the mean entropy of each class;

[0086] As an example, the calculation process of weighted processing is as follows:

[0087]

[0088] Where P represents the semantic segmentation prediction probability map, W, H and C are the length, width and number of channels of the probability map P respectively, and Y S Represents a tag value.

[0089] S72: Use the true value label of the source domain and the pseudo label of the mixed sample as supervision information, compare them with the semantic segmentation result after entropy weighting, and calculate the entropy loss.

[0090] It should be noted that the calculation process of step S72 is as follows:

[0091]

[0092] where Y S and are the semantic labels of source domain samples and the pseudo labels of mixed samples, P S and P M Representing the semantic segmentation prediction probability maps of source domain samples and mixed samples respectively.

[0093] S8: Train the model according to the overall loss and update the model parameters. Input the remote sensing image dataset of the target domain into the trained student model to obtain the final semantic segmentation result.

[0094] It should be noted that step S8 is specifically as follows:

[0095] S81: According to the overall loss, the loss function is used to update the semantic segmentation model and the model is trained until convergence; the overall loss is as follows:

[0096]

[0097] in and Represents the Berhu loss of source domain samples and mixed samples, ω ele and ω ent are the weighted values ​​of the losses respectively.

[0098] S82: Obtain the trained student model and remove the domain adaptation module;

[0099] S83: Input the remote sensing image of the target domain into the trained model, obtain the semantic segmentation prediction result of the target domain through the final classifier, and achieve accurate object classification.

[0100] As an embodiment, the present invention provides 5 comparative examples for comparison with the present invention, including the SourceOnly method and four most advanced UDA methods based on multi-task learning: the deep perception domain adaptation network DADA, the cross-task relationship network CTRL, the mutual information domain adaptation network MIDA and the correlation perception domain adaptation network CorDA. See Table 1.

[0101] Table 1 Quantization results with Potsdam IRRG as source domain and Vaihingen IRRG as target domain

[0102]

[0103]

[0104] The evaluation indicators are F1_Score and IoU, and the detailed formula is:

[0105]

[0106] Where TP represents true positives, FP represents false positives, and FN represents false negatives. Precision represents the ratio of correctly predicted positive examples to all predicted positive examples, and recall represents the ratio of correctly predicted positive examples to all true positive examples. F1_Score represents the weighted average of precision and recall. P represents the prediction mask, G represents the ground truth mask, and IoU represents the intersection over union of the prediction result and the ground truth.

[0107] Table 1 is a comparison of the experimental results of the present invention and other domain adaptive semantic segmentation methods based on multi-task learning on potsdam IRRG->vaihingenIRRG, evaluating the performance of the methods in different geographical environments and when the appearance of similar objects is quite different. Compared with Source Only, all domain adaptive methods have achieved higher accuracy. Overall, the average F1_score and mIoU of the present invention are 35.22% and 33.76% higher than those of Source Only, respectively. It is worth noting that the present invention has achieved the best accuracy in all object categories, significantly outperforming other advanced methods, with an average F1_score and mIoU of 15.33% and 13.95% higher than the best advanced method "CorDA". In addition, compared with all methods, the present invention greatly improves the prediction accuracy of clutter / background classes and reduces the confusion classification of objects with large inter-class differences with other objects. The results show that the method proposed by the present invention is an effective combination of existing multi-task learning methods and domain adaptive semantic segmentation tasks, verifying that the method proposed by the present invention has stronger feature learning ability and object classification ability.

[0108] See also Figure 4 , Figure 4 4 is a schematic diagram of the working of the hardware device of an embodiment of the present invention, and the hardware device specifically includes: a remote sensing unsupervised domain adaptive semantic segmentation device 401 for multi-task collaborative learning, a processor 402 and a storage medium 403.

[0109] A remote sensing unsupervised domain adaptive semantic segmentation device 401 for multi-task collaborative learning: The remote sensing unsupervised domain adaptive semantic segmentation device 401 for multi-task collaborative learning implements the remote sensing unsupervised domain adaptive semantic segmentation method for multi-task collaborative learning.

[0110] Processor 402: The processor 402 loads and executes the instructions and data in the storage medium 403 to implement the remote sensing unsupervised domain adaptive semantic segmentation method of multi-task collaborative learning.

[0111] Storage medium 403: The storage medium 403 stores instructions and data; the storage medium 403 is used to implement the remote sensing unsupervised domain adaptive semantic segmentation method of multi-task collaborative learning.

[0112] The beneficial effects of the present invention are: using cross-task feature association learning, fully mining and optimizing cross-task feature representation through continuous cross-feature enhancement units, closely associating elevation features and semantic features to improve the accuracy of cross-domain segmentation tasks using implicit elevation information. In addition, the present invention optimizes the quality of pseudo-labels, and further uses pseudo-label-guided hybrid consistency learning and entropy-guided category-level alignment to narrow the domain gap between the source domain and the target domain, promote good transfer of source domain knowledge, and greatly enhance the cross-domain capabilities of the model. The present invention greatly improves the segmentation accuracy of challenging scenes in remote sensing datasets, such as shadow areas, boundary areas, and similar spectral areas with significant height differences.

[0113] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A remote sensing unsupervised domain adaptive semantic segmentation method based on multi-task collaborative learning, characterized by: Includes the following: S1: Obtain remote sensing image datasets of the source domain and the target domain, crop the remote sensing image datasets of the source domain and the target domain to obtain elevation images, remote sensing images, and semantic label images; S2: Build a semantic segmentation model, which includes a student model and a teacher model. Both the student model and the teacher model include an initial feature extraction module, a cross-task feature association learning module, and a final classifier. S3: Obtain remote sensing images and elevation images of mixed samples; Input the target domain remote sensing image into the teacher model to obtain the labels of the target domain samples and the pseudo labels of the mixed samples; Input the source domain remote sensing image and the enhanced mixed remote sensing image into the student model to obtain the final elevation estimation results of the source domain samples and the final elevation estimation results of the mixed samples, the final semantic segmentation results of the source domain samples and the final semantic segmentation results of the mixed samples; S4: Construct a domain adaptation module, which includes a pseudo-label-guided hybrid consistency learning module and an entropy-guided category-level alignment module. The pseudo-label-guided hybrid consistency learning module uses the pseudo-labels generated above to supervise the student model's learning of mixed samples; the entropy-guided category-level alignment module weights the entropy map and the mean entropy of each category into the segmentation result to guide fine-grained category-level alignment; S5: The semantic segmentation loss of the source domain samples is calculated based on the source domain sample segmentation results and the semantic label image. The mixed pseudo-label is used as the supervision information. The semantic segmentation loss and weighted cross entropy loss of the mixed samples are calculated using the semantic segmentation results of the mixed samples, and the overall semantic segmentation loss is calculated. S6: Calculate the berHu loss by adding the elevation estimation value and the elevation truth value of the source domain sample and the mixed sample to obtain the elevation estimation loss of the source domain sample and the mixed sample, and calculate the overall elevation estimation loss; S7: Weight the entropy map and the mean entropy of each class to the prediction results of the source domain samples and the mixed samples to promote the semantic alignment of the data distribution of the source domain and the target domain; use the source domain true value and the mixed pseudo-label as supervision information, and calculate the entropy loss with the semantic segmentation result after entropy value weighting; S8: Train the model according to the overall loss and update the model parameters. Input the remote sensing image dataset of the target domain into the trained student model to obtain the final semantic segmentation result.

2. The remote sensing unsupervised domain adaptive semantic segmentation method of multi-task collaborative learning as claimed in claim 1, characterized in that: In step S2: The initial feature extraction module includes: a parameter sharing encoder, an elevation estimation bottleneck, a semantic segmentation bottleneck, an initial elevation classifier and an initial semantic classifier; the initial feature extraction module is used to extract initial features and generate initial semantic segmentation results and initial elevation estimation prediction results; The cross-task feature association learning module consists of two consecutive cross-feature enhancement units and two task-specific gating structures; the cross-task feature association learning module is used to generate adaptive fusion features based on the initial features; The final classifier is used to generate final semantic segmentation and elevation estimation prediction results based on the adaptive fusion features.

3. The remote sensing unsupervised domain adaptive semantic segmentation method of multi-task collaborative learning as claimed in claim 2, characterized in that: Step S3 is specifically as follows: S31: Generate a mixed mask using the ClassMix method, and select the corresponding areas of the source domain and target domain samples according to the mask to generate a remote sensing image of the mixed sample; S32: Input the target domain remote sensing image into the teacher model to generate pseudo labels; combine the mixed mask, the target domain pseudo labels and the source domain label truth values ​​to generate pseudo labels for the mixed samples; S33: Data augmentation for mixed samples, including random horizontal flipping, vertical flipping, color jittering, and Gaussian noise; S34: inputting the remote sensing images of the source domain and the target domain into the initial feature extraction module, generating initial features through the parameter sharing encoder, and generating initial semantic segmentation and elevation estimation prediction results using the elevation estimation bottleneck, the semantic segmentation bottleneck and the initial semantic classifier; S35: The initial features are fed into the cross-task feature association learning module, and the final semantic and elevation adaptive fusion features are generated through the cross-feature enhancement unit and the task-specific gating structure; S36: Input the adaptive fusion features into the final classifier to generate the final semantic segmentation and elevation estimation prediction results.

4. The remote sensing unsupervised domain adaptive semantic segmentation method of multi-task collaborative learning as claimed in claim 1, characterized in that: Step S4 is specifically as follows: S41: Construct a pseudo-label-guided hybrid consistency learning module to balance the model's feature representation capability and pseudo-label quality; the pseudo-label-guided hybrid consistency learning module adopts an incremental pseudo-label screening strategy, dynamically adjusts the current threshold according to the given minimum threshold, maximum threshold, total number of iterations and current number of iterations, and screens pseudo-labels that meet the conditions; S42: Construct an entropy-guided class-level alignment module; The entropy map is calculated based on the final semantic segmentation prediction results and labels, and the average entropy value of each class is calculated using the entropy map, followed by normalization.

5. The remote sensing unsupervised domain adaptive semantic segmentation method of multi-task collaborative learning as claimed in claim 1, characterized in that: Step S5 is specifically as follows: S51: Calculate the semantic segmentation loss of the source domain sample according to the initial semantic segmentation result of the source domain sample, the final semantic segmentation result and the source domain semantic label image; S52: Calculate the semantic segmentation loss of the mixed sample according to the initial semantic segmentation result, the final semantic segmentation result and the mixed pseudo-label of the mixed sample; S53: Calculate the penalty weight according to the adaptation difficulty of the elevation task, and further calculate the weighted cross entropy loss of the mixed samples.

6. The remote sensing unsupervised domain adaptive semantic segmentation method of multi-task collaborative learning as claimed in claim 1, characterized in that: Step S6 is specifically as follows: Get the elevation estimation values ​​of the source domain samples and the mixed samples, use the elevation estimation values ​​and the true elevation values ​​of the source domain samples and the mixed samples to calculate the berHu loss, get the elevation estimation loss of the source domain samples and the mixed samples, sum the elevation estimation losses of all source domain samples and the mixed samples to get the overall elevation estimation loss.

7. The remote sensing unsupervised domain adaptive semantic segmentation method of multi-task collaborative learning as claimed in claim 1, characterized in that: Step S7 is specifically as follows: S71: Obtain prediction results of source domain samples and mixed samples; perform weighted processing on the prediction results of source domain samples and mixed samples, the calculated entropy map and the mean entropy of each class; S72: Use the true value label of the source domain and the pseudo label of the mixed sample as supervision information, compare them with the semantic segmentation result after entropy weighting, and calculate the entropy loss.

8. The remote sensing unsupervised domain adaptive semantic segmentation method of multi-task collaborative learning as claimed in claim 1, characterized in that: Step S8 is specifically as follows: S81: Based on the overall loss, use the loss function to update the semantic segmentation model and train the model until convergence; S82: Obtain the trained student model and remove the domain adaptation module; S83: Input the remote sensing image of the target domain into the trained model, obtain the semantic segmentation prediction result of the target domain through the final classifier, and achieve accurate object classification.

9. A storage medium, characterized in that: The storage medium stores instructions and data for implementing a remote sensing unsupervised domain adaptive semantic segmentation method for multi-task collaborative learning as described in any one of claims 1 to 8.

10. A remote sensing unsupervised domain adaptive semantic segmentation device with multi-task collaborative learning, characterized by: include: Processor and storage medium; the processor loads and executes instructions and data in the storage medium to implement a multi-task collaborative learning remote sensing unsupervised domain adaptive semantic segmentation method as described in any one of claims 1 to 8.

Citation Information

Patent Citations

  • Feature adaptive alignment unsupervised domain adaptive remote sensing image semantic segmentation method

    CN113378906A

  • Unsupervised domain adaptive remote sensing image semantic segmentation method based on course learning

    CN114943965A

  • Unsupervised domain adaptive semantic segmentation method and system

    CN115631337A

  • Local-to-global domain adaptive remote sensing image semantic segmentation method based on pseudo tag generation

    CN115830597A

  • Self-training domain adaptive remote sensing image semantic segmentation method based on pseudo label quality

    CN116863136A

Cited By

  • Open set cross-domain hyperspectral image classification method and network based on evaluation reflection learning

    CN119992199A

  • Semantic association-based remote sensing fire burnout area segmentation method and system

    CN120912631A

  • A semantic correlation-based remote sensing fire burn area segmentation method and system

    CN120912631B

  • Multi-task lymphoma segmentation method based on PET-CT image fusion

    CN121213560A