Cross-domain remote sensing image semantic segmentation method based on multi-scale distribution correction under condition of few labeled samples

Abstract

The invention discloses a cross-domain remote sensing image semantic segmentation method based on multi-scale distribution correction under a few-labeled sample. The method comprises the following steps: acquiring a plurality of source domain images and a category label of each image in each source domain image; performing feature extraction on each source domain image by using a preset feature extractor to obtain a feature map of each source domain image; based on the feature map of each source domain image and the category label of each image in each source domain image, determining a semantic prototype of each category in the plurality of source domain images; and continuously updating the semantic prototype of each category in the plurality of source domain images through the iteration round of the extraction process of the feature extractor to obtain the finally updated semantic prototype of each category. The cross-domain remote sensing image semantic segmentation method solves the technical problems that cross-domain remote sensing image semantic segmentation in the prior art needs a large amount of label data, only uses the same size features, is large in sample semantic distribution difference and is low in segmentation precision.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a cross-domain remote sensing image semantic segmentation method based on multi-scale distribution correction under few labeled samples. Background Art

[0002] Few-shot semantic segmentation uses a small number of labeled samples to train the model and achieve pixel-level classification of images. Existing few-shot image segmentation methods generally use base class samples with the same distribution as the new class for meta-training, but it is often difficult to find training samples with the same distribution as the new class in real scenarios. Based on this, the few-shot image segmentation involved in the present invention aims to use samples with different distributions for cross-domain meta-training, and promote meta-knowledge from fields with sufficient training labels to low-resource fields. Existing few-shot image segmentation methods ignore the multi-scale characteristics of remote sensing images, and there is a problem of overfitting biased distribution. Based on this, the few-shot image segmentation involved in the present invention integrates semantic information of different scales, and designs a regularization mechanism to alleviate the overfitting problem. The above-mentioned technology plays a key role in actual high-complexity and highly dynamic scene applications such as urban planning, precision agriculture, and forest inspections.

[0003] With the development of deep learning, combined with the transfer effect of meta-learning and the few-shot adaptability of metric learning, semantic segmentation technology under few labeled samples has also made progress. For example, Z. Wang et al. designed a few-shot image segmentation algorithm based on prototype queue learning in the document "Zichao Wang, Zhiyu Jiang, and Yuan Yuan. Prototype Queue Learning for Multi-Class Few-Shot Semantic Segmentation. In IEEE International Conference on Image Processing, 2022, pp. 1721-1725". However, this technology often uses a large number of training samples of the same distribution but different categories for transfer training, which are not easy to obtain in real scenarios. In recent years, some attempts have also emerged to solve the problem of cross-domain few-shot segmentation. S. Lei et al. designed a few-shot image segmentation algorithm based on prototype queue learning in the document "Shuo Lei, Xuchao Zhang, Jianfeng He, Fanglan Chen, Bowen Du

[0004] ,andChang-TienLu.Cross-domainFew-shotSemanticSegmentation.InEuropeanConferenceonComputerVision,2022,pp.73-90." The difficulty of cross-domain tasks was evaluated and a pyramid-based module was proposed to convert domain-specific features into domain-agnostic features. Y.Wang et al. in the paper "YixinWang,ZheXu,JiangTian,JieLuo,ZhongchaoShi,YangZhang,JianpingFan,andZhiqiangHe.Cross-domainFew-shotLearningforRare-diseaseSkinLesionSegmentation.InIEEEInternationalConferenceonAcoustics,SpeechandSignalProcessing,2022,pp.1086-1090." The problem of limited samples of rare cases explores the few-shot image segmentation algorithm from natural domain to medical images, and designs two decoders as feature knowledge and general knowledge learners to update alternately and mine common information that can be transferred across domains. H. Chen et al. used bidirectional 3D convolution in the paper "Hao Chen, Yonghan Dong, Zheming Lu, Yunlong Yu, and Jungong Han. Pixel Matching Network for Cross-domain Few-shot Segmentation. In IEEE Winter Conference on Applications of Computer Vision, 2024, pp. 978-987." to capture the relationship between pixel-to-pixel and pixel-to-image blocks between the support set and the query set, and fixed parameters to complete domain-independent pixel-level matching using feature extractors. The above methods explore the cross-domain application of few-shot image segmentation, but they are mainly for natural images and medical images, and are mainly limited to the segmentation of the foreground and background of the image, without further exploration of multi-category segmentation.

[0005] In addition, existing few-shot image segmentation methods lack the use of multi-scale characteristics of remote sensing images, and have weak prediction effects on objects with large scale differences. The model designed by X. Zhang et al. in the paper "Xiaolin Zhang, Yunchao Wei, Yi Yang, and Thomas S. Huang. SG-One: Similarity Guidance Network for One-Shot Semantic Segmentation. IEEE Transactions on Cybernetics, 50(9): 3855-3865, 2020" only extracts semantic prototypes of a single scale to complete prediction, ignoring the differences in category semantics at different scales. C. Zhang et al. used the ASPP module in the document "Chi Zhang, Guosheng Lin, Fayao Liu, Rui Yao, and Chunhua Shen. Canet: Class-agnostic Segmentation Networks with Iterative Refinement and Attentive Few-shot Learning. In IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 5217-5226." to extract the multi-scale information of the test (query) image in the decoding stage, but lacked the utilization of the multi-scale information of the training (support) images. Summary of the invention

[0006] The embodiment of the present invention provides a cross-domain remote sensing image semantic segmentation method based on multi-scale distribution correction under sparsely labeled samples, so as to at least solve the technical problems that the semantic segmentation model in the prior art requires a large amount of labeled data, has no constraints on intra-class differences, leads to semantic aliasing, and has low multi-classification segmentation accuracy.

[0007] According to one aspect of an embodiment of the present invention, a cross-domain remote sensing image semantic segmentation method based on multi-scale distribution correction under sparsely labeled samples is provided. The method may include: obtaining multiple source domain images and the category label of each image in each source domain image; using a preset feature extractor to perform feature extraction on each source domain image respectively to obtain a feature map of each source domain image; based on the feature map of each source domain image and the category label of each image in each source domain image, determining the semantic prototype of each category in the multiple source domain images; continuously updating the semantic prototype of each category in the multiple source domain images through iterative rounds of the feature extractor extraction process to obtain the final updated semantic prototype of each category; based on the final updated semantic prototype of each category, the final updated semantic prototypes of all categories and each source domain image, determining the semantic prototype of each category in the multiple source domain images; The method comprises the following steps: using a feature map of a domain image to determine a probability map of a semantic prototype of each category of multiple source domain images after final updating; obtaining predicted segmentation results corresponding to the multiple source domain images based on the probability map of the semantic prototype of each category of the multiple source domain images after final updating; in the process of obtaining the predicted segmentation results of the multiple source domain images, tuning the parameters of the image segmentation network framework through a cross-domain migration correction module, a multi-scale prototype guidance mechanism and a prototype reverse guidance module to obtain the final image segmentation network framework; obtaining a target domain image to be detected, inputting the target domain image to be detected into the final image segmentation network framework, and obtaining the segmentation result of the target domain image to be detected.

[0008] Optionally, after determining the semantic prototype of each category after the multiple source domain images are finally updated, obtain multiple target domain images and the category label of each image in each target domain image, divide the multiple target domain images into equal parts to obtain a first part of target domain images and a second part of target domain images, use a feature extractor to extract features of each target domain image in the first part of target domain images and the second part of target domain images, respectively, to obtain a feature map of each target domain image in the first part of target domain images and a feature map of each target domain image in the second part of target domain images; determine the semantic prototype of each category in the first part of target domain images based on the feature map of each target domain image in the first part of target domain images and the category label of each image in each target domain image; use a metric function to calculate the initial prototype similarity between the semantic prototype of each category in the first part of target domain images and the semantic prototype of each category after the multiple source domain images are finally updated; determine the prototype comparison loss function based on the initial prototype similarity and preset hyperparameters.

[0009] Optionally, the cross-domain migration correction module also includes: after determining the probability map of the semantic prototype of each category of the multiple source domain images after the final update, determining the first target segmentation loss function based on the probability map of the semantic prototype of each category of the multiple source domain images after the final update and the category label of each image in each source domain image.

[0010] Optionally, the cross-domain migration correction module also includes: after determining the probability map of the semantic prototype of each category of the multiple source domain images after the final update, using the semantic prototype of each category of the multiple source domain images after the final update to segment the feature map of each target domain image of the second part of the target domain images, to obtain the probability map of the semantic prototype of each category of the second part of the target domain images; based on the probability map of the semantic prototype of each category of the second part of the target domain images, obtaining the predicted segmentation result corresponding to the second part of the target domain images; based on the probability map of the semantic prototype of each category of the second part of the target domain images and the category label of each image in each target domain image of the second part of the target domain images, determining the migration alignment loss function.

[0011] Optionally, the cross-domain migration correction module also includes: after determining the probability map of the semantic prototype of each category of the multiple source domain images after the final update, using the semantic prototype of each category of the multiple source domain images after the final update to segment the feature map of each target domain image of the first part of the target domain images, and obtain the probability map of the semantic prototype of each category in the first part of the target domain images; processing the probability map of the semantic prototype of each category in the first part of the target domain images through the first function to obtain the prediction result of each category in each target domain image of the first part of the target domain images; performing feature extraction on each target domain image of the first part of the target domain images through a preset feature extractor to obtain the feature map of each target domain image of the first part of the target domain images, and inputting the feature map of each target domain image of the first part of the target domain images into the decoder to obtain the prediction probability map corresponding to each category in the first part of the target domain images, wherein each category corresponds to a prediction probability map; based on the probability map of the semantic prototype of each category in the first part of the target domain images and the prediction result of the category of each image in each target domain image of the first part of the target domain images, determine the second target segmentation loss function.

[0012] Optionally, the cross-domain migration correction module further includes: determining a total target segmentation loss function of the cross-domain migration correction module based on a prototype contrast loss function, a first target segmentation loss function, a migration alignment loss function and a second target segmentation loss function.

[0013] Optionally, the multi-scale prototype guidance mechanism includes: using feature extractors of four different spatial resolution scales to extract features from each source domain image respectively, to obtain feature maps of four different spatial resolution scales for each source domain image; determining the semantic prototype of each category at each spatial resolution in multiple source domain images based on the feature maps of each different spatial resolution scale of each source domain image and the category label of each image in each source domain image; continuously updating the semantic prototype of each category at each spatial resolution in multiple source domain images through iterative rounds of extraction processes using feature extractors of four different spatial resolution scales, to obtain the finally updated semantic prototype of each category at each spatial resolution; based on the finally updated semantic prototype of each category at each spatial resolution and the finally updated semantic prototype of all four spatial resolutions The method adopts the semantic prototypes of multiple categories and the feature maps of four different spatial resolution scales of each source domain image to determine the probability map of the semantic prototype of each category in each spatial resolution after the final update of multiple source domain images; the size of the probability map of the semantic prototype of each category in each spatial resolution after the final update of multiple source domain images is up-sampled to make the size of the probability map of the semantic prototype of each category in each spatial resolution after the final update the same; the probability maps of the semantic prototype of each category in each spatial resolution with the same size after the final update are fused to obtain the target probability map; based on the probability map of the semantic prototype of each category in each spatial resolution after the final update of multiple source domain images and the category label of each image in each source domain image, the target segmentation loss function of the multi-scale prototype guidance mechanism is determined.

[0014] Optionally, the prototype reverse guidance module includes: obtaining multiple target domain images, wherein each target domain image is an unlabeled image; using a preset feature extractor to extract features from each unlabeled target domain image respectively to obtain a feature map of each unlabeled target domain image; using the semantic prototype of each category in the first part of the target domain images to segment the feature map of each unlabeled target domain image to obtain a probability map of the semantic prototype of each category of the multiple unlabeled target domain images; processing the probability map of the semantic prototype of each category of the multiple unlabeled target domain images by a first function to obtain a pseudo label of each category in each unlabeled target domain image in the multiple unlabeled target domain images; based on each unlabeled target domain image in the multiple unlabeled target domain images The pseudo label of each category in the unlabeled target domain images and the feature map of each unlabeled target domain image are used to obtain the semantic prototype of each category in multiple unlabeled target domain images; the semantic prototype of each category in multiple unlabeled target domain images is used to guide the segmentation of the first part of the target domain images and the second part of the target domain images to obtain the unlabeled prototype-guided probability map of the first part of the target domain images and the unlabeled prototype-guided probability map of the second part of the target domain images; according to the unlabeled prototype-guided probability map of the first part of the target domain images, the unlabeled prototype-guided probability map of the second part of the target domain images, the true label of each image in the first part of the target domain images and the true label of each image in the second part of the target domain images, the target segmentation loss function of the prototype reverse guidance module is determined.

[0015] Beneficial effects of the present invention:

[0016] The present invention proposes a cross-domain remote sensing image semantic segmentation method based on multi-scale distribution correction under few labeled samples. The present invention replaces samples of the same distribution used for meta-training in previous methods with samples of different distributions, and corrects the category semantic distribution respectively at the feature level and the prototype level; performs feature extraction and semantic guidance on images at different scales; uses the original sample semantic prototype to predict other samples to obtain labels, and then extracts other sample prototypes to reversely predict the original samples; a multi-scale prototype extraction module and a prototype reverse guidance module further optimize the segmentation result, and solves the technical problems in the prior art that the semantic segmentation of cross-domain remote sensing images requires a large amount of label data, only uses features of the same size, the sample semantic distribution gap is large, and the segmentation accuracy is low, and achieves the technical effect of requiring a small amount of label data, making full use of the multi-scale characteristics of remote sensing images, reducing the sample semantic distribution gap, and improving the classification accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0018] Figure 1 It is a flowchart of a cross-domain remote sensing image semantic segmentation method based on multi-scale distribution correction under few labeled samples according to an embodiment of the present invention;

[0019] Figure 2 Schematic diagram of semantic segmentation results generated by the present invention (CDPLNet) and the comparison algorithm. DETAILED DESCRIPTION

[0020] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only embodiments of a part of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0021] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0022] Example 1

[0023] According to an embodiment of the present invention, a cross-domain remote sensing image semantic segmentation method based on multi-scale distribution correction with few labeled samples is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system comprising at least one set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in an order different from that shown here.

[0024] Figure 1 is a flowchart of a cross-domain remote sensing image semantic segmentation method based on multi-scale distribution correction under few labeled samples according to an embodiment of the present invention. Figure 1 As shown, the method may include the following steps:

[0025] Step S101, obtaining multiple source domain images and a category label of each image in each source domain image.

[0026] In the technical solution provided in the above step S101 of the present invention, multiple source domain images and the category label of each image in each source domain image are obtained. For example, if a source domain image contains a person, a car, and a building, the label of the person is 1, the label of the car is 2, and the label of the building is 3.

[0027] Step S102: Use a preset feature extractor to perform feature extraction on each source domain image to obtain a feature map of each source domain image.

[0028] In the technical solution provided in the above step S102 of the present invention, a feature extractor (a feature extractor with a resolution) is used to extract features from each source domain image to obtain a feature map of each source domain image. .

[0029] Step S103, determining a semantic prototype of each category in the plurality of source domain images based on the feature map of each source domain image and the category label of each image in each source domain image.

[0030] In the technical solution provided in step S103 of the present invention, the feature map of each source domain image and the category label of each image in each source domain image are calculated to obtain the expression of the semantic prototype of each category in the multiple source domain images:

[0031]

[0032] in, is the semantic prototype of each category in multiple source domain images, [M s (x,y)=c] is the category label of each image in each source domain image, x and y are the indexes of the spatial position, 1[.] represents the indicator function, which outputs a value of 1 when the internal judgment statement is correct, otherwise it outputs 0, F s (x,y) is the feature map of each source domain image.

[0033] Step S104, the semantic prototype of each category in the multiple source domain images is continuously updated through iterative rounds of the feature extractor extraction process to obtain a final updated semantic prototype of each category.

[0034] In the technical solution provided in the above step S104 of the present invention, the semantic prototype of each category in the multiple source domain images is continuously updated through iterative rounds of the feature extractor extraction process to obtain the semantic prototype of each category after the final update. For example, 30 rounds of epoachs, each epoachs includes 64 batch-sizes, that is, the semantic prototype of each category after the final update is the semantic prototype of each category of the last batch-size of the 30th epoachs, wherein the expression of the semantic prototype of each category after the final update is:

[0035]

[0036] in, is the semantic prototype of each category after the final update, is the semantic prototype before the final updated semantic prototype of each category, l is the iteration round, and β is the weight factor, which is set to 0.8.

[0037] Step S105 , based on the finally updated semantic prototype of each category, the finally updated semantic prototypes of all categories and the feature map of each source domain image, determine the probability map of the semantic prototype of each category of the multiple source domain images after the final update.

[0038] In the technical solution provided in step S105 of the present invention, the semantic prototype of each category after the final update, the semantic prototypes of all categories after the final update, and the feature map of each source domain image are calculated to obtain the expression of the probability map of the semantic prototype of each category of the multiple source domain images after the final update:

[0039]

[0040] in, is the probability map of the semantic prototype of each category of multiple source domain images after the final update, F s (x,y) is the feature map of each source domain image, is the semantic prototype of all categories after the final update, p j is the semantic prototype of each category after the final update.

[0041] Step S106, based on the probability map of the semantic prototype of each category of the multiple source domain images after the final update, obtain the predicted segmentation results corresponding to the multiple source domain images.

[0042] In the technical solution provided in the above step S106 of the present invention, the predicted segmentation results corresponding to the multiple source domain images are obtained according to the probability map of the semantic prototype of each category after the multiple source domain images are finally updated.

[0043] Step S107, in the process of obtaining the predicted segmentation results of multiple source domain images, the parameters of the image segmentation network framework are tuned through the cross-domain migration correction module, the multi-scale prototype guidance mechanism and the prototype reverse guidance module to obtain the final image segmentation network framework.

[0044] In the technical solution provided in the above step S107 of the present invention, in the process of obtaining the predicted segmentation results of multiple source domain images, the parameters of the image segmentation network framework are tuned through the cross-domain migration correction module, the multi-scale prototype guidance mechanism and the prototype reverse guidance module to obtain the final image segmentation network framework.

[0045] Step S108, obtaining the target domain image to be detected, inputting the target domain image to be detected into the final image segmentation network framework, and obtaining the segmentation result of the target domain image to be detected.

[0046] In the technical solution provided in the above step S108 of the present invention, the target domain image to be detected is obtained, and the target domain image to be detected is input into the final image segmentation network framework to obtain the segmentation result of the target domain image to be detected.

[0047] The above method of this embodiment is further introduced below.

[0048] As an optional implementation method, step S106, the cross-domain migration correction module includes: after determining the semantic prototype of each category after the multiple source domain images are finally updated, obtaining multiple target domain images and the category label of each image in each target domain image, dividing the multiple target domain images into equal parts to obtain a first part of target domain images and a second part of target domain images, using a feature extractor to perform feature extraction on each target domain image in the first part of the target domain images and the second part of the target domain images, respectively, to obtain a feature map of each target domain image in the first part of the target domain images and a feature map of each target domain image in the second part of the target domain images; based on the feature map of each target domain image in the first part of the target domain images and the category label of each image in each target domain image, determining the semantic prototype of each category in the first part of the target domain images; using a metric function to calculate the initial prototype similarity between the semantic prototype of each category in the first part of the target domain images and the semantic prototype of each category after the multiple source domain images are finally updated; based on the initial prototype similarity and the preset hyperparameters, determining the prototype comparison loss function.

[0049] In this embodiment, a feature extractor is used to extract features from each target domain image in the first part of the target domain image and the second part of the target domain image, respectively, to obtain a feature map of each target domain image in the first part of the target domain image. And the feature map of each target domain image in the second part of the target domain image

[0050] Based on the feature map of each target domain image in the first part of the target domain images and the category label of each image in each target domain image, the expression of the semantic prototype of each category in the first part of the target domain images is determined as follows:

[0051]

[0052] in, is the semantic prototype of each category in the target domain image in the first part, is the feature map of each target domain image in the first part of the target domain images, is the category label of each image in each target domain image in the first part.

[0053] The expression for calculating the initial prototype similarity between the semantic prototype of each category in the first part of the target domain image and the semantic prototype of each category after the final update of multiple source domain images using the metric function is:

[0054]

[0055] Among them, ProSim is the initial prototype similarity between the semantic prototype of each category in the first part of the target domain images and the semantic prototype of each category after the final update of multiple source domain images, N is the total number of the same categories in the source domain and the first part of the target domain images, d is the measurement function, and β is used to ensure that the similarity is a positive number, which is set to 1 in this section.

[0056] Based on the initial prototype similarity and preset hyperparameters, the expression of the prototype comparison loss function is determined as:

[0057]

[0058] Among them, L pro is the prototype contrast loss function, α is a hyperparameter, set to 1×10 -5 .

[0059] As an optional implementation method, step S106, the cross-domain migration correction module also includes: after determining the probability map of the semantic prototype of each category of the multiple source domain images after the final update, based on the probability map of the semantic prototype of each category of the multiple source domain images after the final update and the category label of each image in each source domain image, determining the first target segmentation loss function.

[0060] In this embodiment, based on the probability map of the semantic prototype of each category of the multiple source domain images after the final update and the category label of each image in each source domain image, the expression of the first target segmentation loss function is determined as follows:

[0061]

[0062] Among them, L seg is the first target segmentation loss function.

[0063] As an optional implementation method, step S106, the cross-domain migration correction module also includes: after determining the probability map of the semantic prototype of each category of the multiple source domain images after the final update, using the semantic prototype of each category of the multiple source domain images after the final update to segment the feature map of each target domain image of the second part of the target domain images, to obtain the probability map of the semantic prototype of each category of the second part of the target domain images; based on the probability map of the semantic prototype of each category of the second part of the target domain images, obtaining the predicted segmentation result corresponding to the second part of the target domain images; based on the probability map of the semantic prototype of each category of the second part of the target domain images and the category label of each image in each target domain image of the second part of the target domain images, determine the migration alignment loss function.

[0064] In this embodiment, the feature map of each target domain image of the second part of the target domain images is segmented using the semantic prototype of each category after the final update of the multiple source domain images, and the expression of the probability map of the semantic prototype of each category of the second part of the target domain images is obtained as follows:

[0065]

[0066] in, For the second part, the probability map of the semantic prototype of the target domain image in each category, is the semantic prototype of each category after the final update of multiple source domain images, It is the feature map of each target domain image in the second part of target domain images.

[0067] Based on the probability map of the semantic prototype of each category of the second part of the target domain image and the category label of each image in each target domain image of the second part of the target domain image, the expression of the transfer alignment loss function is determined as:

[0068]

[0069] Among them, L supp is the transfer alignment loss function, is the category label of each image in each target domain image of the second part, The second part is the probability map of the semantic prototype of each category of the target domain image.

[0070] As an optional implementation method, step S106, the cross-domain migration correction module also includes: after determining the probability map of the semantic prototype of each category of the multiple source domain images after the final update, using the semantic prototype of each category of the multiple source domain images after the final update to segment the feature map of each target domain image of the first part of the target domain images, to obtain the probability map of the semantic prototype of each category in the first part of the target domain images; the probability map of the semantic prototype of each category in the first part of the target domain images is processed by the first function to obtain the prediction of each category in each target domain image of the first part of the target domain images. Results: Feature extraction is performed on each target domain image of the first part of target domain images through a preset feature extractor to obtain a feature map of each target domain image of the first part of target domain images, and the feature map of each target domain image of the first part of target domain images is input into a decoder to obtain a prediction probability map corresponding to each category in the first part of target domain images, wherein each category corresponds to a prediction probability map; based on the probability map of the semantic prototype of each category in the first part of target domain images and the prediction result of the category of each image in each target domain image of the first part of target domain images, the second target segmentation loss function is determined.

[0071] In this embodiment, the probability map of the semantic prototype of each category in the first part of the target domain images is processed by the first function, and the expression of the prediction result of each category in each target domain image of the first part of the target domain images is obtained as follows:

[0072]

[0073] in, is the prediction result of each category in each target domain image of the first part, The second part is the probability map of the semantic prototype of the target domain image in each category, and the first function is the argmax function.

[0074] The feature map of each target domain image in the first part of the target domain image Input into the decoder to obtain the predicted probability map corresponding to each category in the first part of the target domain image

[0075] Based on the probability map of the semantic prototype of each category in the first part of the target domain image and the prediction result of the category of each image in each target domain image of the first part of the target domain image, the expression of the second target segmentation loss function is determined as follows:

[0076]

[0077] Among them, L query is the second target segmentation loss function, P tj (x, y) is the prediction result of the category of each image in each target domain image of the first part of the target domain images.

[0078] As an optional implementation method, in step S106, the cross-domain migration correction module also includes: determining the total target segmentation loss function of the cross-domain migration correction module based on the prototype contrast loss function, the first target segmentation loss function, the migration alignment loss function and the second target segmentation loss function.

[0079] In this embodiment, based on the prototype contrast loss function, the first target segmentation loss function, the migration alignment loss function and the second target segmentation loss function, the expression of the total target segmentation loss function of the cross-domain migration correction module is determined as follows:

[0080] Loss = L seg +a×L supp +b×L query +c×L pro

[0081] Among them, Loss is the total target segmentation loss function of the cross-domain migration correction module, a, b, c are hyperparameters, which are set to 0.5, 0.4 and 0.2 respectively in this section.

[0082] As an optional implementation method, step S106, a multi-scale prototype guidance mechanism, includes: using feature extractors of four different spatial resolution scales to perform feature extraction on each source domain image respectively, to obtain feature maps of four different spatial resolution scales for each source domain image; based on the feature maps of each different spatial resolution scale of each source domain image and the category label of each image in each source domain image, determining the semantic prototype of each category at each spatial resolution in multiple source domain images; continuously updating the semantic prototype of each category at each spatial resolution in multiple source domain images through iterative rounds of extraction processes using feature extractors of four different spatial resolution scales, to obtain the finally updated semantic prototype of each category at each spatial resolution; based on the finally updated semantic prototype of each category at each spatial resolution, the finally updated four The method adopts the semantic prototypes of all categories at four spatial resolutions and the feature maps of four different spatial resolutions of each source domain image to determine the probability map of the semantic prototype of each category at each spatial resolution after the multiple source domain images are finally updated; the size of the probability map of the semantic prototype of each category at each spatial resolution after the multiple source domain images are finally updated is up-sampled to make the size of the probability map of the semantic prototype of each category at each spatial resolution after the multiple source domain images are finally updated the same; the probability maps of the semantic prototype of each category at each spatial resolution with the same size after the final update are fused to obtain the target probability map; based on the probability map of the semantic prototype of each category at each spatial resolution after the multiple source domain images are finally updated and the category label of each image in each source domain image, the target segmentation loss function of the multi-scale prototype guidance mechanism is determined.

[0083] In this embodiment, feature extractors of four different spatial resolution scales are used to extract features from each source domain image respectively, and expressions of feature maps of four different spatial resolution scales of each source domain image are obtained as follows:

[0084]

[0085] Among them, f represents the feature extractor, F represents the output feature maps of different scales, and F i The feature map of the i-th scale, C i is the channel of the feature map of the i-th scale, H i and W i The distribution is the width and height of the feature map of the i-th scale. The downsampling of the multi-scale feature map of size I relative to the input sample original image is 1 / 4, 1 / 8, 1 / 16 and 1 / 32 respectively.

[0086] Based on the feature maps of each source domain image at each different spatial resolution scale and the category label of each image in each source domain image, the expression of the semantic prototype of each category at each spatial resolution in multiple source domain images is determined as follows:

[0087]

[0088] Among them, p c,i is the semantic prototype of each category at the i-th spatial resolution in multiple source domain images, The feature map of the i-th spatial resolution scale in each source domain image, The class label of each image in each source image at the i-th spatial resolution.

[0089] Based on the semantic prototype of each category at each spatial resolution after the final update, the semantic prototypes of all categories at the four spatial resolutions after the final update, and the feature maps of four different spatial resolution scales of each source domain image, the expression of the probability map of the semantic prototype of each category of multiple source domain images at each spatial resolution after the final update is determined as follows:

[0090]

[0091] in, is the probability map of the semantic prototype of each category at the i-th spatial resolution after the final update of multiple source domain images, are feature maps of four different spatial resolution scales for each source domain image, p j is the semantic prototype of all categories at the four spatial resolutions after the final update, p j；i is the semantic prototype of each category at the i-th spatial resolution after the final update.

[0092] The probability maps of the semantic prototypes of each category at each spatial resolution with the same final updated size are fused to obtain the expression of the target probability map:

[0093]

[0094] in, is the target probability map, and a1, a2, a3, and a4 are hyperparameter weights.

[0095] Based on the probability map of the semantic prototype of each category at each spatial resolution of multiple source domain images after the final update and the category label of each image in each source domain image, the expression of the target segmentation loss function of the multi-scale prototype guidance mechanism is determined as:

[0096]

[0097] Among them, L mult It is the target segmentation loss function of the multi-scale prototype guidance mechanism.

[0098] As an optional implementation method, step S106, the prototype reverse guidance module, includes: obtaining multiple target domain images, wherein each target domain image is an unlabeled image; using a preset feature extractor to extract features of each unlabeled target domain image respectively, to obtain a feature map of each unlabeled target domain image; using the semantic prototype of each category in the first part of the target domain images to segment the feature map of each unlabeled target domain image, to obtain a probability map of the semantic prototype of each category of the multiple unlabeled target domain images; processing the probability map of the semantic prototype of each category of the multiple unlabeled target domain images by a first function, to obtain a pseudo label of each category in each of the multiple unlabeled target domain images; based on each of the multiple unlabeled target domain images The pseudo-label of each category in the unlabeled target domain image and the feature map of each unlabeled target domain image are used to obtain the semantic prototype of each category in multiple unlabeled target domain images; the semantic prototype of each category in multiple unlabeled target domain images is used to guide the segmentation of the first part of the target domain images and the second part of the target domain images to obtain the unlabeled prototype-guided probability map of the first part of the target domain images and the unlabeled prototype-guided probability map of the second part of the target domain images; according to the unlabeled prototype-guided probability map of the first part of the target domain images, the unlabeled prototype-guided probability map of the second part of the target domain images, the true label of each image in the first part of the target domain images and the true label of each image in the second part of the target domain images, the target segmentation loss function of the prototype reverse guidance module is determined.

[0099] In this embodiment, based on the pseudo label of each category in each of the multiple unlabeled target domain images and the feature map of each unlabeled target domain image, the expression of the semantic prototype of each category in the multiple unlabeled target domain images is obtained as follows:

[0100]

[0101] in, is the semantic prototype of each category in multiple unlabeled target domain images, p j is the pseudo label of each category in each unlabeled target domain image in multiple unlabeled target domain images, is the feature map of each unlabeled target domain image, where the pseudo-label is the marked pseudo-label.

[0102] The probability map of the semantic prototype of each category of multiple unlabeled target domain images is processed by the first function, and the expression of the pseudo label of each category in each unlabeled target domain image in multiple unlabeled target domain images is obtained as follows:

[0103]

[0104] in, The pseudo-label of each category in each unlabeled target domain image in multiple unlabeled target domain images, where the pseudo-label is the calculated pseudo-label.

[0105] Based on the pseudo-label of each category in each unlabeled target domain image in multiple unlabeled target domain images and the feature map of each unlabeled target domain image, the expression of the semantic prototype of each category in multiple unlabeled target domain images is obtained as follows:

[0106]

[0107] in, is the semantic prototype of each category in multiple unlabeled target domain images, F aux (x,y) is the feature map of each unlabeled target domain image.

[0108] The semantic prototypes of each category in multiple unlabeled target domain images are used to guide the segmentation of the first part of the target domain images and the second part of the target domain images. The expressions of the probability map guided by the unlabeled prototype of the first part of the target domain images and the probability map guided by the unlabeled prototype of the second part of the target domain images are:

[0109]

[0110] in, is the probability map guided by the unlabeled prototype of the first part of the target domain image and the probability map guided by the unlabeled prototype of the second part of the target domain image, F (x,y) is the first part of the target domain image and the second part of the target domain image.

[0111] According to the probability map of the unlabeled prototype guidance of the first part of the target domain image, the probability map of the unlabeled prototype guidance of the second part of the target domain image, the true label of each image in the first part of the target domain image and the true label of each image in the second part of the target domain image, the expression of the target segmentation loss function of the prototype reverse guidance module is determined as:

[0112]

[0113] Among them, L aux is the target segmentation loss function of the prototype backward guidance module.

[0114] Experimental part: The present invention is simulated by using Pytorch on a CPU of Intel(R) Xeon(R) Silver4110CPU@2.10GHz, memory of 40G, and Linux operating system. The data used in the simulation is a public data set.

[0115] 1. Simulation content

[0116] The data used in the simulation are Postdam and Vaihingen datasets, both of which are aerial photography datasets released by ISPRS and contain six categories. Among them, the Postdam dataset contains 38 aerial images, each of which contains four spectral bands: red, green, blue and infrared (RGBIR). The present invention uses the three RGB channels and cuts them into 1330 512×512 image blocks. The Vaihingen dataset has a total of 33 RGB images, which are cut into 426 512ⅹ512 image blocks. The present invention conducts cross-domain experiments, selects one of the datasets as the source domain to retain all labels for migration training, and the other dataset is used as the target domain to retain a small number of sample labels for few-sample training and testing. For the target domain, ten pictures and their corresponding multi-category labels are selected for each category for model training, and the remaining pictures are used for testing. To ensure the fairness of the experiment, the training samples are randomly selected five times, and the test index selects the average value of the five groups of experimental indicators.

[0117] In order to prove the effectiveness of the algorithm, this paper selected PANet, SSP, DCAMA and HRNet for comparison on two cross-domain datasets. Among them, PANet is a method proposed in the document "Kaixin Wang, Jun Hao Liew, Yingtian Zou, Daquan Zhou, and Jiashi Feng. Panet: Few-shot Image Semantic Segmentation with Prototype Alignment. In IEEE International Conference on Computer Vision, 2019, pp. 9197-9206.", which is a classic small-shot semantic segmentation algorithm; SSP is a method proposed in the document "Qi Fan, Wenjie Pei, Yu-Wing Tai, and Chi-Keung Tang. Self-support Few-shot Semantic Segmentation n.InEuropean Conference on Computer Vision, 2022, pp.701-719." DCAMA is a small sample image segmentation method proposed in the literature "Xinyu Shi, Dong Wei, Yu Zhang, Donghuan Lu, Munan Ning, Jiashun Chen, Kai Ma, and Yefeng Zheng. Dense Cross-query-and-support Attention Weighted Mask Aggregation for Few-shot Segmentation. In European Conference on Computer Vision, 2022, pp.151-168." HRNet was proposed in the paper "KeSun, BinXiao, DongLiu, and JingdongWang. DeepHigh-resolutionRepresentationLearningforHumanPoseEstimation.InIEEEConferenceonComputerVisionandPatternRecognition,2019,pp.5693-5703." It is a classic semantic segmentation algorithm. We use it to verify the effect of the fine-tuning method on the cross-domain few-sample segmentation task. CDPLNet is the method proposed in this paper. OA and mIoU are the evaluation indicators of the quality of few-sample semantic segmentation. The comparison results are shown in Table 1:

[0118] Table 1 Segmentation comparison chart

[0119]

[0120]

[0121] As can be seen from Table 1, in the experimental results of the two cross-domain datasets Vaihingen->Postdam and Postdam->Vaihingen, the present invention is superior to other algorithms in both OA and mIoU indicators.

[0122] Figure 2 It is the semantic segmentation result diagram generated by the present invention (CDPLNet) and the comparison algorithm. It can be seen that the prediction segmentation effect of the method of the present invention on remote sensing images is significantly better than that of the comparison method, which proves that the present invention effectively utilizes cross-domain common semantic information. Compared with the fine-tuning method, the present invention can better segment the boundaries of ground objects and eliminate particle holes, reflecting the regularization advantage of the reverse guidance module. Compared with the baseline method, the present invention can better distinguish small-scale objects such as cars, and is better for large-scale objects such as building edges, proving the effectiveness of the multi-scale module of the present invention.

[0123] In an embodiment of the present invention, multiple source domain images and the category label of each image in each source domain image are obtained; a preset feature extractor is used to perform feature extraction on each source domain image to obtain a feature map of each source domain image; based on the feature map of each source domain image and the category label of each image in each source domain image, a semantic prototype of each category in the multiple source domain images is determined; the semantic prototype of each category in the multiple source domain images is continuously updated through iterative rounds of the feature extractor extraction process to obtain the semantic prototype of each category after the final update; based on the semantic prototype of each category after the final update, the semantic prototypes of all categories after the final update and the feature map of each source domain image, a probability map of the semantic prototype of each category of the multiple source domain images after the final update is determined; based on the semantic prototype of each category of the multiple source domain images after the final update The probability map of the semantic prototype is used to obtain the predicted segmentation results corresponding to multiple source domain images; in the process of obtaining the predicted segmentation results of multiple source domain images, the parameters of the image segmentation network framework are tuned through the cross-domain migration correction module, the multi-scale prototype guidance mechanism and the prototype reverse guidance module to obtain the final image segmentation network framework; the target domain image to be detected is obtained, and the target domain image to be detected is input into the final image segmentation network framework to obtain the segmentation result of the target domain image to be detected, which solves the technical problems in the prior art that the semantic segmentation of cross-domain remote sensing images requires a large amount of label data, only uses features of the same size, the semantic distribution gap of samples is large, and the segmentation accuracy is low, and achieves the technical effect of requiring a small amount of label data, making full use of the multi-scale characteristics of remote sensing images, reducing the semantic distribution gap of samples, and improving classification accuracy.

[0124] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0125] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0126] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0127] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed over multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0128] In addition, each functional unit in each embodiment of the present invention may be integrated into a first processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0129] The above are only preferred embodiments of the present invention. It should be pointed out that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A cross-domain remote sensing image semantic segmentation method based on multi-scale distribution correction under few labeled samples, characterized in that: include: Obtain multiple source domain images and the category label of each image in each source domain image; Using a preset feature extractor to extract features from each source domain image, and obtaining a feature map of each source domain image; Determine the semantic prototype of each category in multiple source domain images based on the feature map of each source domain image and the category label of each image in each source domain image; The semantic prototype of each category in multiple source domain images is continuously updated through iterative rounds of feature extractor extraction process to obtain the final updated semantic prototype of each category; Determine a probability map of the semantic prototype of each category of the multiple source domain images after the final update based on the semantic prototype of each category after the final update, the semantic prototypes of all categories after the final update, and the feature map of each source domain image; Based on the probability map of the semantic prototype of each category of the multiple source domain images after the final update, the predicted segmentation results corresponding to the multiple source domain images are obtained; In the process of obtaining the predicted segmentation results of multiple source domain images, the parameters of the image segmentation network framework are tuned through the cross-domain migration correction module, the multi-scale prototype guidance mechanism and the prototype reverse guidance module to obtain the final image segmentation network framework; The target domain image to be detected is obtained, and the target domain image to be detected is input into the final image segmentation network framework to obtain the segmentation result of the target domain image to be detected.

2. The method according to claim 1, characterized in that The cross-domain migration correction module includes: After determining the semantic prototype of each category after the multiple source domain images are finally updated, multiple target domain images and the category label of each image in each target domain image are obtained, the multiple target domain images are equally divided to obtain a first part of target domain images and a second part of target domain images, and a feature extractor is used to extract features from each target domain image in the first part of target domain images and the second part of target domain images, respectively, to obtain a feature map of each target domain image in the first part of target domain images and a feature map of each target domain image in the second part of target domain images; Determine a semantic prototype of each category in the first part of target domain images based on a feature map of each target domain image in the first part of target domain images and a category label of each image in each target domain image; The metric function is used to calculate the initial prototype similarity between the semantic prototype of each category in the target domain image in the first part and the semantic prototype of each category after the final update of multiple source domain images; Based on the initial prototype similarity and preset hyperparameters, the prototype comparison loss function is determined.

3. The method according to claim 1, characterized in that The cross-domain migration correction module also includes: After determining the probability map of the semantic prototype of each category of the multiple source domain images after the final update, a first target segmentation loss function is determined based on the probability map of the semantic prototype of each category of the multiple source domain images after the final update and the category label of each image in each source domain image.

4. The method according to claim 1, characterized in that: The cross-domain migration correction module also includes: After determining the probability map of the semantic prototype of each category of the multiple source domain images after final update, segment the feature map of each target domain image of the second part of target domain images using the semantic prototype of each category of the multiple source domain images after final update to obtain the probability map of the semantic prototype of each category of the second part of target domain images; Based on the probability map of the semantic prototype of each category of the second part target domain image, the predicted segmentation result corresponding to the second part target domain image is obtained; Based on the probability map of the semantic prototype of each category of the second part target domain image and the category label of each image in each target domain image of the second part target domain image, a transfer alignment loss function is determined.

5. The method according to claim 1, characterized in that The cross-domain migration correction module also includes: After determining the probability map of the semantic prototype of each category of the multiple source domain images after final update, segment the feature map of each target domain image of the first part of target domain images using the semantic prototype of each category of the multiple source domain images after final update to obtain the probability map of the semantic prototype of each category in the first part of target domain images; Processing the probability map of the semantic prototype of each category in the first part of the target domain images through the first function to obtain a prediction result of each category in each target domain image of the first part of the target domain images; Performing feature extraction on each target domain image of the first part of target domain images by a preset feature extractor to obtain a feature map of each target domain image of the first part of target domain images, inputting the feature map of each target domain image of the first part of target domain images into a decoder to obtain a prediction probability map corresponding to each category in the first part of target domain images, wherein each category corresponds to a prediction probability map; A second target segmentation loss function is determined based on a probability map of a semantic prototype of each category in the first part of target domain images and a prediction result of a category of each image in each target domain image of the first part of target domain images.

6. The method according to claim 1, characterized in that The cross-domain migration correction module also includes: Based on the prototype contrast loss function, the first target segmentation loss function, the migration alignment loss function and the second target segmentation loss function, the total target segmentation loss function of the cross-domain migration correction module is determined.

7. The method according to claim 1, characterized in that Multi-scale prototype guidance mechanism, including: Use feature extractors with four different spatial resolution scales to extract features from each source domain image, and obtain feature maps with four different spatial resolution scales for each source domain image; Based on the feature maps of each different spatial resolution scale of each source domain image and the category label of each image in each source domain image, determine the semantic prototype of each category at each spatial resolution in multiple source domain images; The semantic prototype of each category at each spatial resolution in multiple source domain images is continuously updated through iterative rounds of feature extractor extraction processes of four different spatial resolution scales, and the final updated semantic prototype of each category at each spatial resolution is obtained; Based on the semantic prototype of each category at each spatial resolution after the final update, the semantic prototypes of all categories at four spatial resolutions after the final update, and the feature maps of four different spatial resolution scales of each source domain image, determine the probability map of the semantic prototype of each category at each spatial resolution after the final update of multiple source domain images; The sizes of the probability maps of the semantic prototypes of each category at each spatial resolution after the final update of multiple source domain images are up-sampled so that the sizes of the probability maps of the semantic prototypes of each category at each spatial resolution after the final update are the same; The probability maps of the semantic prototypes of each category at each spatial resolution with the same final updated size are fused to obtain the target probability map; Based on the probability map of the semantic prototype of each category at each spatial resolution of multiple source domain images after the final update and the category label of each image in each source domain image, the target segmentation loss function of the multi-scale prototype guidance mechanism is determined.

8. The method according to claim 1, characterized in that Prototype reverse guidance module, including: Acquire multiple target domain images, where each target domain image is an unlabeled image; Using a preset feature extractor to extract features from each unlabeled target domain image, and obtaining a feature map of each unlabeled target domain image; The feature map of each unlabeled target domain image is segmented using the semantic prototype of each category in the target domain image of the first part, and the probability map of the semantic prototype of each category of multiple unlabeled target domain images is obtained; The probability map of the semantic prototype of each category of the multiple unlabeled target domain images is processed by the first function to obtain a pseudo label of each category in each of the multiple unlabeled target domain images; Based on the pseudo-label of each category in each of the multiple unlabeled target domain images and the feature map of each unlabeled target domain image, a semantic prototype of each category in the multiple unlabeled target domain images is obtained; Using the semantic prototype of each category in the multiple unlabeled target domain images to guide the segmentation of the first part of the target domain images and the second part of the target domain images, a probability map guided by the unlabeled prototype of the first part of the target domain images and a probability map guided by the unlabeled prototype of the second part of the target domain images are obtained; A target segmentation loss function of the prototype reverse guidance module is determined according to the unlabeled prototype-guided probability map of the first part target domain images, the unlabeled prototype-guided probability map of the second part target domain images, the true label of each image in the first part target domain images, and the true label of each image in the second part target domain images.

9. A processor, characterized in that: The processor is used to run a program, wherein the program executes the method according to any one of claims 1 to 8 when running.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 8.

Images