A cross-domain remote sensing image semantic segmentation method based on multi-scale distribution correction with few labeled samples

Abstract

The present invention discloses a cross-domain remote sensing image semantic segmentation method based on multi-scale distribution correction under sparsely labeled samples. The method comprises: obtaining multiple source domain images and the category label of each image in each source domain image; using a preset feature extractor to perform feature extraction on each source domain image respectively to obtain a feature map of each source domain image; based on the feature map of each source domain image and the category label of each image in each source domain image, determining the semantic prototype of each category in the multiple source domain images; continuously updating the semantic prototype of each category in the multiple source domain images through iterative rounds of the feature extractor extraction process to obtain the final updated semantic prototype of each category. The present invention solves the technical problems in the prior art that the semantic segmentation of cross-domain remote sensing images requires a large amount of label data, only uses features of the same size, has a large gap in the semantic distribution of samples, and has low segmentation accuracy.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a cross-domain remote sensing image semantic segmentation method based on multi-scale distribution correction with few labeled samples. Background Art

[0002] Few-shot semantic segmentation uses a small number of labeled samples to train the model and achieve pixel-level classification of images. Existing few-shot image segmentation methods generally use base class samples with the same distribution as the new class for meta-training, but in real scenarios it is often difficult to find training samples with the same distribution as the new class. Based on this, the few-shot image segmentation involved in the present invention aims to use samples with different distributions for cross-domain meta-training, and promote meta-knowledge from fields with sufficient training labels to low-resource fields. Existing few-shot image segmentation methods ignore the multi-scale characteristics of remote sensing images and have the problem of overfitting biased distribution. Based on this, the few-shot image segmentation involved in the present invention integrates semantic information of different scales and designs a regularization mechanism to alleviate the overfitting problem. The above technology plays a key role in actual high-complexity and highly dynamic scene applications such as urban planning, precision agriculture, and forest inspections.

[0003] With the development of deep learning, combined with the transfer effect of meta-learning and the few-shot adaptability of metric learning, semantic segmentation technology under few labeled samples has also made progress. For example, Z. Wang et al. designed a few-shot image segmentation algorithm based on prototype queue learning in the document "Zichao Wang, Zhiyu Jiang, and Yuan Yuan. Prototype Queue Learning for Multi-Class Few-Shot Semantic Segmentation. In IEEE International Conference on Image Processing, 2022, pp. 1721-1725". However, this technology often uses a large number of training samples of the same distribution but different categories for transfer training, and these samples are not easy to obtain in real scenarios. In recent years, there have also been some attempts to solve the problem of cross-domain few-shot segmentation. S. Lei et al. designed a few-shot image segmentation algorithm based on prototype queue learning in the document "Shuo Lei, Xuchao Zhang, Jianfeng He, Fanglan Chen, Bowen Du

[0004] , and Chang-Tien Lu. Cross-domain Few-shot Semantic Segmentation. In European Conference on Computer Vision, 2022, pp. 73–90.” The difficulty of the cross-domain task was evaluated and a pyramid-based module was proposed to convert domain-specific features into domain-agnostic features. Y. Wang et al. in the paper “Yixin Wang, Zhe Xu, Jiang Tian, ​​Jie Luo, Zhongchao Shi, Yang Zhang, Jianping Fan, and Zhiqiang He. Cross-domain Few-shot Learning for Rare-disease Skin Lesion Segmentation. In IEEE International Conference on Acoustics, Speech and Signal Processing, 2022, pp. 1086–1090.” The limited sample size problem of rare cases explores few-shot image segmentation algorithms from natural domains to medical images. Two decoders are designed, serving as feature knowledge and general knowledge learners, which are updated alternately to mine common information that is transferable across domains. H. Chen et al. ("Hao Chen, Yonghan Dong, Zheming Lu, Yunlong Yu, and Jungong Han. Pixel Matching Network for Cross-domain Few-shot Segmentation. In IEEE Winter Conference on Applications of Computer Vision, 2024, pp. 978–987") use bidirectional 3D convolutions to capture the pixel-to-pixel and pixel-to-image patch relationships between the support and query sets, and use a fixed-parameter feature extractor to achieve domain-independent pixel-level matching. These methods explore cross-domain applications of few-shot image segmentation, but primarily target natural and medical images and are limited to foreground and background segmentation, without further exploration of multi-class segmentation.

[0005] In addition, existing few-shot image segmentation methods lack the ability to utilize the multi-scale characteristics of remote sensing images, and their prediction effect on objects with large scale differences is weak. The model designed by X. Zhang et al. in the paper "Xiaolin Zhang, Yunchao Wei, Yi Yang, and Thomas S. Huang. SG-One: Similarity Guidance Network for One-Shot Semantic Segmentation. IEEE Transactions on Cybernetics, 50(9):3855-3865, 2020" only extracts semantic prototypes at a single scale to complete prediction, ignoring the differences in category semantics at different scales. C. Zhang et al. used the ASPP module in the document "Chi Zhang, Guosheng Lin, Fayao Liu, Rui Yao, and Chunhua Shen. Canet: Class-agnostic Segmentation Networks with Iterative Refinement and Attentive Few-shot Learning. In IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 5217–5226." to extract the multi-scale information of the test (query) image in the decoding stage, but lacked the utilization of the multi-scale information of the training (support) images. Summary of the Invention

[0006] An embodiment of the present invention provides a cross-domain remote sensing image semantic segmentation method based on multi-scale distribution correction under few labeled samples, so as to at least solve the technical problems in the prior art that the semantic segmentation model requires a large amount of labeled data, has no constraints on intra-class differences, leads to semantic aliasing, and has low multi-classification segmentation accuracy.

[0007] According to one aspect of an embodiment of the present invention, a cross-domain remote sensing image semantic segmentation method based on multi-scale distribution correction under few labeled samples is provided. The method may include: obtaining multiple source domain images and the category label of each image in each source domain image; using a preset feature extractor to perform feature extraction on each source domain image respectively to obtain a feature map of each source domain image; based on the feature map of each source domain image and the category label of each image in each source domain image, determining the semantic prototype of each category in the multiple source domain images; continuously updating the semantic prototype of each category in the multiple source domain images through iterative rounds of the feature extractor extraction process to obtain the final updated semantic prototype of each category; based on the final updated semantic prototype of each category, the final updated semantic prototypes of all categories and each source domain image, The method adopts a method for obtaining the predicted segmentation results of the multiple source domain images based on the feature map of the domain image, determining the probability map of the semantic prototype of each category of the multiple source domain images after the final update; obtaining the predicted segmentation results of the multiple source domain images based on the probability map of the semantic prototype of each category of the multiple source domain images after the final update; in the process of obtaining the predicted segmentation results of the multiple source domain images, tuning the parameters of the image segmentation network framework through the cross-domain migration correction module, the multi-scale prototype guidance mechanism and the prototype reverse guidance module to obtain the final image segmentation network framework; obtaining the target domain image to be detected, inputting the target domain image to be detected into the final image segmentation network framework, and obtaining the segmentation result of the target domain image to be detected.

[0008] Optionally, after determining the semantic prototype of each category after the multiple source domain images are finally updated, obtain multiple target domain images and the category label of each image in each target domain image, divide the multiple target domain images into equal parts to obtain a first part of target domain images and a second part of target domain images, use a feature extractor to perform feature extraction on each target domain image in the first part of target domain images and the second part of target domain images, respectively, to obtain a feature map of each target domain image in the first part of target domain images and a feature map of each target domain image in the second part of target domain images; based on the feature map of each target domain image in the first part of target domain images and the category label of each image in each target domain image, determine the semantic prototype of each category in the first part of target domain images; use a metric function to calculate the initial prototype similarity between the semantic prototype of each category in the first part of target domain images and the semantic prototype of each category after the multiple source domain images are finally updated; based on the initial prototype similarity and preset hyperparameters, determine the prototype comparison loss function.

[0009] Optionally, the cross-domain migration correction module also includes: after determining the probability map of the semantic prototype of each category of the multiple source domain images after the final update, determining the first target segmentation loss function based on the probability map of the semantic prototype of each category of the multiple source domain images after the final update and the category label of each image in each source domain image.

[0010] Optionally, the cross-domain migration correction module also includes: after determining the probability map of the semantic prototype of each category of the multiple source domain images after the final update, using the semantic prototype of each category of the multiple source domain images after the final update to segment the feature map of each target domain image of the second part of the target domain image, to obtain the probability map of the semantic prototype of each category of the second part of the target domain image; based on the probability map of the semantic prototype of each category of the second part of the target domain image, obtaining the predicted segmentation result corresponding to the second part of the target domain image; based on the probability map of the semantic prototype of each category of the second part of the target domain image and the category label of each image in each target domain image of the second part of the target domain image, determining the migration alignment loss function.

[0011] Optionally, the cross-domain migration correction module also includes: after determining the probability map of the semantic prototype of each category of the multiple source domain images after the final update, using the semantic prototype of each category of the multiple source domain images after the final update to segment the feature map of each target domain image of the first part of the target domain image, to obtain the probability map of the semantic prototype of each category in the first part of the target domain image; processing the probability map of the semantic prototype of each category in the first part of the target domain image through the first function to obtain the prediction result of each category in each target domain image of the first part of the target domain image; performing feature extraction on each target domain image of the first part of the target domain image through a preset feature extractor to obtain the feature map of each target domain image of the first part of the target domain image, inputting the feature map of each target domain image of the first part of the target domain image into the decoder to obtain the prediction probability map corresponding to each category in the first part of the target domain image, wherein each category corresponds to a prediction probability map; determining the second target segmentation loss function based on the probability map of the semantic prototype of each category in the first part of the target domain image and the prediction result of the category of each image in each target domain image of the first part of the target domain image.

[0012] Optionally, the cross-domain migration correction module further includes: determining a total target segmentation loss function of the cross-domain migration correction module based on the prototype contrast loss function, the first target segmentation loss function, the migration alignment loss function and the second target segmentation loss function.

[0013] Optionally, a multi-scale prototype guidance mechanism includes: using feature extractors of four different spatial resolution scales to extract features from each source domain image respectively, to obtain feature maps of four different spatial resolution scales for each source domain image; based on the feature maps of each different spatial resolution scale of each source domain image and the category label of each image in each source domain image, determining the semantic prototype of each category at each spatial resolution in multiple source domain images; continuously updating the semantic prototype of each category at each spatial resolution in multiple source domain images through iterative rounds of extraction processes using feature extractors of four different spatial resolution scales, to obtain the final updated semantic prototype of each category at each spatial resolution; based on the final updated semantic prototype of each category at each spatial resolution and the final updated semantic prototype of all four spatial resolutions The multi-scale prototype-guided mechanism is based on the semantic prototypes of all categories and the feature maps of four different spatial resolution scales of each source domain image to determine the probability map of the semantic prototype of each category at each spatial resolution after the final update of multiple source domain images; the size of the probability map of the semantic prototype of each category at each spatial resolution after the final update of multiple source domain images is up-sampled to make the size of the probability map of the semantic prototype of each category at each spatial resolution after the final update the same; the probability maps of the semantic prototype of each category at each spatial resolution with the same size after the final update are fused to obtain the target probability map; based on the probability map of the semantic prototype of each category at each spatial resolution after the final update of multiple source domain images and the category label of each image in each source domain image, the target segmentation loss function of the multi-scale prototype-guided mechanism is determined.

[0014] Optionally, the prototype reverse guidance module includes: obtaining multiple target domain images, wherein each target domain image is an unlabeled image; using a preset feature extractor to extract features from each unlabeled target domain image respectively to obtain a feature map of each unlabeled target domain image; using the semantic prototype of each category in the first part of the target domain image to segment the feature map of each unlabeled target domain image to obtain a probability map of the semantic prototype of each category of the multiple unlabeled target domain images; processing the probability map of the semantic prototype of each category of the multiple unlabeled target domain images by a first function to obtain a pseudo label of each category in each unlabeled target domain image in the multiple unlabeled target domain images; based on each unlabeled target domain image in the multiple unlabeled target domain images The pseudo label of each category in the unlabeled target domain images and the feature map of each unlabeled target domain image are used to obtain the semantic prototype of each category in multiple unlabeled target domain images; the semantic prototype of each category in multiple unlabeled target domain images is used to guide the segmentation of the first part of the target domain images and the second part of the target domain images to obtain the unlabeled prototype-guided probability map of the first part of the target domain images and the unlabeled prototype-guided probability map of the second part of the target domain images; according to the unlabeled prototype-guided probability map of the first part of the target domain images, the unlabeled prototype-guided probability map of the second part of the target domain images, the true label of each image in the first part of the target domain images and the true label of each image in the second part of the target domain images, the target segmentation loss function of the prototype reverse guidance module is determined.

[0015] Beneficial effects of the present invention:

[0016] The present invention proposes a cross-domain remote sensing image semantic segmentation method based on multi-scale distribution correction under few labeled samples. The present invention replaces the samples with the same distribution used for meta-training in the previous method with samples with different distributions, and corrects the category semantic distribution respectively at the feature level and the prototype level; performs feature extraction and semantic guidance on the image at different scales; uses the original sample semantic prototype to predict other samples to obtain labels, and then extracts other sample prototypes to reversely predict the original sample; the multi-scale prototype extraction module and the prototype reverse guidance module further optimize the segmentation results, which solves the technical problems in the prior art that the cross-domain remote sensing image semantic segmentation requires a large amount of label data, only uses features of the same size, has a large gap in sample semantic distribution, and has low segmentation accuracy. It achieves the technical effect of requiring a small amount of label data, making full use of the multi-scale characteristics of remote sensing images, reducing the gap in sample semantic distribution, and improving classification accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0018] Figure 1 This is a flowchart of a cross-domain remote sensing image semantic segmentation method based on multi-scale distribution correction with few labeled samples according to an embodiment of the present invention;

[0019] Figure 2 Schematic diagram of semantic segmentation results generated by the present invention (CDPLNet) and the comparison algorithm. DETAILED DESCRIPTION

[0020] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0021] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and to describe a specific order or precedence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or inherent to these processes, methods, products or devices.

[0022] Example 1

[0023] According to an embodiment of the present invention, a cross-domain remote sensing image semantic segmentation method based on multi-scale distribution correction with few labeled samples is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system comprising at least one set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in an order different from that shown here.

[0024] Figure 1 is a flowchart of a cross-domain remote sensing image semantic segmentation method based on multi-scale distribution correction under few labeled samples according to an embodiment of the present invention. Figure 1 As shown, the method may include the following steps:

[0025] Step S101: Acquire multiple source domain images and the category label of each image in each source domain image.

[0026] In the technical solution provided in step S101 of the present invention, multiple source domain images and the category label of each image in each source domain image are obtained. For example, if a source domain image contains a person, a car, and a building, the label of the person is 1, the label of the car is 2, and the label of the building is 3.

[0027] In step S102 , a preset feature extractor is used to extract features from each source domain image to obtain a feature map of each source domain image.

[0028] In the technical solution provided in step S102 of the present invention, a feature extractor (a feature extractor with a resolution) is used to extract features from each source domain image to obtain a feature map of each source domain image. .

[0029] Step S103 : determining a semantic prototype of each category in the plurality of source domain images based on the feature map of each source domain image and the category label of each image in each source domain image.

[0030] In the technical solution provided in step S103 of the present invention, the feature map of each source domain image and the category label of each image in each source domain image are calculated to obtain the expression of the semantic prototype of each category in multiple source domain images:

[0031]

[0032] in, is the semantic prototype of each category in multiple source domain images, is the category label of each image in each source domain image, x and y are the indexes of the spatial position, 1[.] represents the indicator function, which outputs a value of 1 when the internal judgment statement is correct, otherwise it outputs 0. is the feature map of each source domain image.

[0033] In step S104 , the semantic prototype of each category in the plurality of source domain images is continuously updated through iterative rounds of the feature extractor extraction process to obtain a final updated semantic prototype of each category.

[0034] In the technical solution provided in step S104 of the present invention, the semantic prototype of each category in the multiple source domain images is continuously updated through iterative rounds of the feature extractor extraction process to obtain the final updated semantic prototype of each category. For example, there are 30 rounds of epoachs, each epoachs includes 64 batch-sizes, that is, the final updated semantic prototype of each category is the semantic prototype of each category of the last batch-size of the 30th epoachs, wherein the expression of the final updated semantic prototype of each category is:

[0035]

[0036] in, is the semantic prototype of each category after the final update, is the semantic prototype before the final updated semantic prototype of each category, Iteration round is the weight factor, set to 0.8.

[0037] Step S105 : Based on the finally updated semantic prototype of each category, the finally updated semantic prototypes of all categories, and the feature map of each source domain image, a probability map of the semantic prototype of each category of the multiple source domain images after the final update is determined.

[0038] In the technical solution provided in step S105 of the present invention, the semantic prototype of each category after the final update, the semantic prototypes of all categories after the final update, and the feature map of each source domain image are calculated to obtain the expression of the probability map of the semantic prototype of each category of the multiple source domain images after the final update:

[0039]

[0040] in, is the probability map of the semantic prototype of each category of multiple source domain images after the final update, is the feature map of each source domain image, is the semantic prototype of all categories after the final update, is the semantic prototype of each category after final update.

[0041] Step S106 : obtaining predicted segmentation results corresponding to the multiple source domain images based on the probability maps of the semantic prototypes of each category after the multiple source domain images are finally updated.

[0042] In the technical solution provided in the above step S106 of the present invention, the predicted segmentation results corresponding to the multiple source domain images are obtained according to the probability map of the semantic prototype of each category after the multiple source domain images are finally updated.

[0043] In step S107, in the process of obtaining the predicted segmentation results of multiple source domain images, the parameters of the image segmentation network framework are tuned through the cross-domain migration correction module, the multi-scale prototype guidance mechanism, and the prototype reverse guidance module to obtain the final image segmentation network framework.

[0044] In the technical solution provided in the above step S107 of the present invention, in the process of obtaining the predicted segmentation results of multiple source domain images, the parameters of the image segmentation network framework are tuned through the cross-domain migration correction module, the multi-scale prototype guidance mechanism and the prototype reverse guidance module to obtain the final image segmentation network framework.

[0045] Step S108: obtaining the target domain image to be detected, inputting the target domain image to be detected into the final image segmentation network framework, and obtaining the segmentation result of the target domain image to be detected.

[0046] In the technical solution provided in the above step S108 of the present invention, the target domain image to be detected is obtained, and the target domain image to be detected is input into the final image segmentation network framework to obtain the segmentation result of the target domain image to be detected.

[0047] The above method of this embodiment is further introduced below.

[0048] As an optional embodiment, step S106, the cross-domain migration correction module includes: after determining the semantic prototype of each category after the multiple source domain images are finally updated, obtaining multiple target domain images and the category label of each image in each target domain image, dividing the multiple target domain images into equal parts to obtain a first part of target domain images and a second part of target domain images, using a feature extractor to extract features of each target domain image in the first part of target domain images and the second part of target domain images, respectively, to obtain a feature map of each target domain image in the first part of target domain images and a feature map of each target domain image in the second part of target domain images; based on the feature map of each target domain image in the first part of target domain images and the category label of each image in each target domain image, determining the semantic prototype of each category in the first part of target domain images; using a metric function to calculate the initial prototype similarity between the semantic prototype of each category in the first part of target domain images and the semantic prototype of each category after the multiple source domain images are finally updated; and determining the prototype contrast loss function based on the initial prototype similarity and preset hyperparameters.

[0049] In this embodiment, a feature extractor is used to extract features from each target domain image in the first part of the target domain image and the second part of the target domain image, and a feature map of each target domain image in the first part of the target domain image is obtained ( ) and the feature map of each target domain image in the second part of the target domain image ( ).

[0050] Based on the feature map of each target domain image in the first part of the target domain images and the category label of each image in each target domain image, the expression for determining the semantic prototype of each category in the first part of the target domain images is:

[0051]

[0052] in, is the semantic prototype of each category in the target domain image in the first part, is the feature map of each target domain image in the first part of the target domain image, is the category label of each image in each target domain image in the first part.

[0053] The expression for calculating the initial prototype similarity between the semantic prototype of each category in the first part of the target domain image and the semantic prototype of each category after the final update of multiple source domain images using the metric function is:

[0054]

[0055] in, is the initial prototype similarity between the semantic prototype of each category in the first part of the target domain image and the semantic prototype of each category after the final update of multiple source domain images, is the total number of the same categories of the source domain and the first part of the target domain images, is the metric function, Used to ensure that the similarity is a positive number, it is set to 1 in this section.

[0056] Based on the initial prototype similarity and preset hyperparameters, the expression of the prototype comparison loss function is determined as:

[0057]

[0058] in, is the prototype contrast loss function, is a hyperparameter, set to .

[0059] As an optional embodiment, step S106, the cross-domain migration correction module further includes: after determining the probability map of the semantic prototype of each category of the multiple source domain images after the final update, determining the first target segmentation loss function based on the probability map of the semantic prototype of each category of the multiple source domain images after the final update and the category label of each image in each source domain image.

[0060] In this embodiment, based on the probability map of the semantic prototype of each category of the multiple source domain images after the final update and the category label of each image in each source domain image, the expression of the first target segmentation loss function is determined as follows:

[0061]

[0062] in, is the first target segmentation loss function.

[0063] As an optional embodiment, step S106, the cross-domain migration correction module also includes: after determining the probability map of the semantic prototype of each category of the multiple source domain images after the final update, using the semantic prototype of each category of the multiple source domain images after the final update to segment the feature map of each target domain image of the second part of the target domain image, to obtain the probability map of the semantic prototype of each category of the second part of the target domain image; based on the probability map of the semantic prototype of each category of the second part of the target domain image, obtaining the predicted segmentation result corresponding to the second part of the target domain image; based on the probability map of the semantic prototype of each category of the second part of the target domain image and the category label of each image in each target domain image of the second part of the target domain image, determining the migration alignment loss function.

[0064] In this embodiment, the feature map of each target domain image in the second part of the target domain images is segmented using the semantic prototype of each category after the final update of the multiple source domain images, and the expression of the probability map of the semantic prototype of each category of the second part of the target domain images is obtained as follows:

[0065]

[0066] in, For the second part, the probability map of the semantic prototype of the target domain image in each category, is the semantic prototype of each category after the final update of multiple source domain images, It is the feature map of each target domain image in the second part of the target domain image.

[0067] Based on the probability map of the semantic prototype of each category of the second part of the target domain image and the category label of each image in each target domain image of the second part of the target domain image, the expression of the transfer alignment loss function is determined as follows:

[0068]

[0069] in, is the transfer alignment loss function, is the category label of each image in each target domain image of the second part, The second part is the probability map of the semantic prototype of each category of the target domain image.

[0070] As an optional embodiment, step S106, the cross-domain migration correction module further includes: after determining the probability map of the semantic prototype of each category of the multiple source domain images after the final update, using the semantic prototype of each category of the multiple source domain images after the final update to segment the feature map of each target domain image of the first part of the target domain image to obtain the probability map of the semantic prototype of each category in the first part of the target domain image; processing the probability map of the semantic prototype of each category in the first part of the target domain image through the first function to obtain the prediction of each category in each target domain image of the first part of the target domain image. Results: Feature extraction is performed on each target domain image of the first part of the target domain image through a preset feature extractor to obtain a feature map of each target domain image of the first part of the target domain image, and the feature map of each target domain image of the first part of the target domain image is input into the decoder to obtain a prediction probability map corresponding to each category in the first part of the target domain image, wherein each category corresponds to a prediction probability map; based on the probability map of the semantic prototype of each category in the first part of the target domain image and the prediction result of the category of each image in each target domain image of the first part of the target domain image, the second target segmentation loss function is determined.

[0071] In this embodiment, the probability map of the semantic prototype of each category in the first part of the target domain images is processed by the first function, and the expression of the prediction result of each category in each target domain image in the first part of the target domain images is obtained as follows:

[0072]

[0073] in, is the prediction result of each category in each target domain image of the first part, The second part is the probability map of the semantic prototype of the target domain image in each category, and the first function is the argmax function.

[0074] The feature map of each target domain image in the first part of the target domain image Input into the decoder to obtain the predicted probability map corresponding to each category in the first part of the target domain image .

[0075] Based on the probability map of the semantic prototype of each category in the first part of the target domain image and the prediction result of the category of each image in each target domain image in the first part of the target domain image, the expression of the second target segmentation loss function is determined as follows:

[0076]

[0077] in, is the second target segmentation loss function, It is the prediction result of the category of each image in each target domain image of the first part.

[0078] As an optional embodiment, in step S106, the cross-domain migration correction module also includes: determining the total target segmentation loss function of the cross-domain migration correction module based on the prototype contrast loss function, the first target segmentation loss function, the migration alignment loss function and the second target segmentation loss function.

[0079] In this embodiment, based on the prototype contrast loss function, the first target segmentation loss function, the migration alignment loss function and the second target segmentation loss function, the expression of the total target segmentation loss function of the cross-domain migration correction module is determined as follows:

[0080]

[0081] in, is the total target segmentation loss function of the cross-domain migration correction module, are hyperparameters, which are set to 0.5, 0.4, and 0.2 in this section.

[0082] As an optional embodiment, step S106, a multi-scale prototype guidance mechanism, includes: using feature extractors of four different spatial resolution scales to extract features from each source domain image respectively, to obtain feature maps of four different spatial resolution scales for each source domain image; based on the feature maps of each different spatial resolution scale of each source domain image and the category label of each image in each source domain image, determining the semantic prototype of each category at each spatial resolution in multiple source domain images; continuously updating the semantic prototype of each category at each spatial resolution in multiple source domain images through iterative rounds of extraction processes using feature extractors of four different spatial resolution scales, to obtain the final updated semantic prototype of each category at each spatial resolution; based on the final updated semantic prototype of each category at each spatial resolution, the final updated four The semantic prototypes of all categories at four spatial resolutions and the feature maps of four different spatial resolution scales of each source domain image are used to determine the probability map of the semantic prototype of each category at each spatial resolution after the final update of multiple source domain images; the size of the probability map of the semantic prototype of each category at each spatial resolution after the final update of multiple source domain images is upsampled to make the size of the probability map of the semantic prototype of each category at each spatial resolution after the final update the same; the probability maps of the semantic prototype of each category at each spatial resolution with the same size after the final update are fused to obtain the target probability map; based on the probability map of the semantic prototype of each category at each spatial resolution after the final update of multiple source domain images and the category label of each image in each source domain image, the target segmentation loss function of the multi-scale prototype guidance mechanism is determined.

[0083] In this embodiment, feature extractors of four different spatial resolution scales are used to extract features from each source domain image, and the expressions of the feature maps of four different spatial resolution scales of each source domain image are obtained as follows:

[0084]

[0085] in, represents the feature extractor, Represents the output feature maps of different scales, No. Scale feature map, It is The channels of the feature map of the scale, and The distribution is The width and height of the scaled feature map. Relative to the original input sample image The downsampling of the multi-scale feature maps is 1 / 4, 1 / 8, 1 / 16 and 1 / 32 respectively.

[0086] Based on the feature maps of each source image at different spatial resolution scales and the category label of each image in each source image, the expression for determining the semantic prototype of each category at each spatial resolution in multiple source images is:

[0087]

[0088] in, is the semantic prototype of each category at the i-th spatial resolution in multiple source domain images, The feature map of the i-th spatial resolution scale in each source domain image, The category label of each image in each source domain image at the i-th spatial resolution.

[0089] Based on the semantic prototype of each category at each spatial resolution after the final update, the semantic prototypes of all categories at the four spatial resolutions after the final update, and the feature maps of the four different spatial resolution scales of each source domain image, the expression of the probability map of the semantic prototype of each category at each spatial resolution after the final update of multiple source domain images is determined as follows:

[0090]

[0091] in, is the probability map of the semantic prototype of each category at the i-th spatial resolution after the final update of multiple source domain images, are feature maps of four different spatial resolution scales for each source domain image, is the semantic prototype of all categories at the four spatial resolutions after the final update, is the semantic prototype of each category at the i-th spatial resolution after the final update.

[0092] The probability maps of the semantic prototypes of each category at each spatial resolution with the same final updated size are fused to obtain the expression of the target probability map:

[0093]

[0094] in, is the target probability map, 、 、 、 is the hyperparameter weight.

[0095] Based on the probability map of the semantic prototype of each category at each spatial resolution after the final update of multiple source domain images and the category label of each image in each source domain image, the expression of the target segmentation loss function of the multi-scale prototype guidance mechanism is determined as follows:

[0096]

[0097] in, It is the target segmentation loss function of the multi-scale prototype guidance mechanism.

[0098] As an optional embodiment, step S106, the prototype reverse guidance module includes: obtaining multiple target domain images, wherein each target domain image is an unlabeled image; using a preset feature extractor to extract features of each unlabeled target domain image respectively to obtain a feature map of each unlabeled target domain image; using the semantic prototype of each category in the first part of the target domain image to segment the feature map of each unlabeled target domain image to obtain a probability map of the semantic prototype of each category of the multiple unlabeled target domain images; processing the probability map of the semantic prototype of each category of the multiple unlabeled target domain images by a first function to obtain a pseudo label of each category in each of the multiple unlabeled target domain images; based on each of the multiple unlabeled target domain images The pseudo label of each category in the unlabeled target domain image and the feature map of each unlabeled target domain image are used to obtain the semantic prototype of each category in multiple unlabeled target domain images; the semantic prototype of each category in multiple unlabeled target domain images is used to guide the segmentation of the first part of the target domain image and the second part of the target domain image to obtain the unlabeled prototype-guided probability map of the first part of the target domain image and the unlabeled prototype-guided probability map of the second part of the target domain image; according to the unlabeled prototype-guided probability map of the first part of the target domain image, the unlabeled prototype-guided probability map of the second part of the target domain image, the true label of each image in the first part of the target domain image and the true label of each image in the second part of the target domain image, the target segmentation loss function of the prototype reverse guidance module is determined.

[0099] In this embodiment, based on the pseudo label of each category in each of the multiple unlabeled target domain images and the feature map of each unlabeled target domain image, the expression of the semantic prototype of each category in the multiple unlabeled target domain images is obtained as follows:

[0100]

[0101] in, is the semantic prototype of each category in multiple unlabeled target domain images, is the pseudo label of each category in each unlabeled target domain image in multiple unlabeled target domain images, is the feature map of each unlabeled target domain image, where the pseudo label is the marked pseudo label.

[0102] The probability map of the semantic prototype of each category of multiple unlabeled target domain images is processed by the first function, and the expression of the pseudo label of each category in each unlabeled target domain image in multiple unlabeled target domain images is obtained as follows:

[0103]

[0104] in, The pseudo labels for each category in each of the multiple unlabeled target domain images are calculated.

[0105] Based on the pseudo label of each category in each unlabeled target domain image in multiple unlabeled target domain images and the feature map of each unlabeled target domain image, the expression of the semantic prototype of each category in multiple unlabeled target domain images is obtained as follows:

[0106]

[0107] in, is the semantic prototype of each category in multiple unlabeled target domain images, is the feature map of each unlabeled target domain image.

[0108] The semantic prototypes of each category in multiple unlabeled target domain images are used to guide the segmentation of the first part of the target domain image and the second part of the target domain image. The expressions of the probability map guided by the unlabeled prototype of the first part of the target domain image and the probability map guided by the unlabeled prototype of the second part of the target domain image are:

[0109]

[0110] in, is the probability map guided by the unlabeled prototype of the first part of the target domain image and the probability map guided by the unlabeled prototype of the second part of the target domain image, are the first part of the target domain image and the second part of the target domain image.

[0111] According to the probability map of the unlabeled prototype guidance of the first part of the target domain image, the probability map of the unlabeled prototype guidance of the second part of the target domain image, the true label of each image in the first part of the target domain image and the true label of each image in the second part of the target domain image, the expression of the target segmentation loss function of the prototype reverse guidance module is determined as:

[0112]

[0113] in, is the target segmentation loss function of the prototype backward guidance module.

[0114] Experimental: This paper simulates the problem using PyTorch on an Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz, 40GB of RAM, and a Linux operating system. The data used in the simulation is a publicly available dataset.

[0115] The data used in the simulation are the Postdam and Vaihingen datasets, both of which are aerial photography datasets released by ISPRS and contain six categories. The Postdam dataset contains 38 aerial images, each of which contains four spectral bands: red, green, blue, and infrared (RGBIR). The present invention uses the three RGB channels and crops them into 1,330 512×512 image blocks. The Vaihingen dataset has a total of 33 RGB images, which are cropped into 426 512×512 image blocks. The present invention conducts cross-domain experiments, selecting one of the datasets as the source domain and retaining all labels for transfer training, and the other dataset as the target domain and retaining a small number of sample labels for few-sample training and testing. For the target domain, ten images and their corresponding multi-category labels are selected from each category for model training, and the remaining images are used for testing. To ensure the fairness of the experiment, the training samples are randomly selected five times, and the test indicators are the average values ​​of the five groups of experimental indicators.

[0116] In order to prove the effectiveness of the algorithm, this paper selected PANet, SSP, DCAMA and HRNet for comparison on two cross-domain datasets. Among them, PANet is the method proposed in the document "Kaixin Wang, Jun Hao Liew, Yingtian Zou, Daquan Zhou, and Jiashi Feng. Panet: Few-shot Image Semantic Segmentation with Prototype Alignment. In IEEE International Conference on Computer Vision, 2019, pp. 9197–9206.", which is a classic small-shot semantic segmentation algorithm; SSP is the method proposed in the document "Qi Fan, Wenjie Pei, Yu-Wing Tai, and Chi-Keung Tang. Self-support Few-shot Semantic Segmentation n.InEuropeanConferenceonComputerVision,2022,pp.701–719.” The small sample image segmentation method proposed; DCAMA is the small sample image segmentation method proposed in the literature “XinyuShi,DongWei,YuZhang,DonghuanLu,MunanNing,JiashunChen,KaiMa,andYefengZheng.DenseCross-query-and-supportAttentionWeightedMaskAggregationforFew-shotSegmentation.InEuropeanConferenceonComputerVision,2022,pp.151–168.” HRNet, proposed in the paper "KeSun, BinXiao, DongLiu, and JingdongWang. Deep High-resolution Representation Learning for Human Pose Estimation. In IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 5693–5703," is a classic semantic segmentation algorithm. We used it to verify the effectiveness of the fine-tuning method on the cross-domain few-shot segmentation task. CDPLNet is the method proposed in this paper. OA and mIoU are evaluation metrics for the quality of few-shot semantic segmentation. The comparison results are shown in Table 1:

[0117] Table 1 Segmentation comparison diagram

[0118]

[0119] As can be seen from Table 1, in the experimental results of the two cross-domain datasets Vaihingen->Postdam and Postdam->Vaihingen, the proposed method outperforms other algorithms in both OA and mIoU indicators.

[0120] Figure 2 The following figure shows the semantic segmentation results generated by the present invention (CDPLNet) and the comparison algorithm. As can be seen, the present invention significantly outperforms the comparison method in predicting and segmenting remote sensing images, demonstrating the effective utilization of cross-domain common semantic information. Compared with the fine-tuning method, the present invention better segments the boundaries of objects and eliminates particle holes, demonstrating the regularization advantage of the reverse guidance module. Compared with the baseline method, the present invention better distinguishes small-scale objects such as cars and larger-scale objects such as building edges, demonstrating the effectiveness of the present invention's multi-scale module.

[0121] In an embodiment of the present invention, a plurality of source domain images and a category label of each image in each source domain image are obtained; a preset feature extractor is used to perform feature extraction on each source domain image to obtain a feature map of each source domain image; based on the feature map of each source domain image and the category label of each image in each source domain image, a semantic prototype of each category in the plurality of source domain images is determined; the semantic prototype of each category in the plurality of source domain images is continuously updated through iterative rounds of the feature extractor extraction process to obtain a final updated semantic prototype of each category; based on the final updated semantic prototype of each category, the final updated semantic prototypes of all categories and the feature map of each source domain image, a probability map of the semantic prototype of each category in the plurality of source domain images after the final update is determined; based on the final updated semantic prototype of each category in the plurality of source domain images after the final update, a probability map of the semantic prototype of each category in the plurality of source domain images after the final update is obtained. The probability map of the semantic prototype is used to obtain the predicted segmentation results corresponding to multiple source domain images; in the process of obtaining the predicted segmentation results of multiple source domain images, the parameters of the image segmentation network framework are tuned through the cross-domain migration correction module, the multi-scale prototype guidance mechanism and the prototype reverse guidance module to obtain the final image segmentation network framework; the target domain image to be detected is obtained, and the target domain image to be detected is input into the final image segmentation network framework to obtain the segmentation result of the target domain image to be detected, which solves the technical problems in the existing technology that the cross-domain remote sensing image semantic segmentation requires a large amount of label data, only uses the same size features, the sample semantic distribution gap is large, and the segmentation accuracy is low, and achieves the technical effect of requiring a small amount of label data, making full use of the multi-scale characteristics of remote sensing images, reducing the sample semantic distribution gap, and improving classification accuracy.

[0122] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0123] In the above embodiments of the present invention, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0124] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0125] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected to achieve the purpose of the present embodiment according to actual needs.

[0126] In addition, the functional units in various embodiments of the present invention may be integrated into a first processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0127] The above are only preferred embodiments of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A cross-domain remote sensing image semantic segmentation method based on multi-scale distribution correction under few labeled samples, characterized by: include: Obtain multiple source domain images and the category label of each image in each source domain image; Use the preset feature extractor to extract features from each source domain image and obtain a feature map of each source domain image; Determine the semantic prototype of each category in multiple source domain images based on the feature map of each source domain image and the category label of each image in each source domain image; The semantic prototype of each category in multiple source domain images is continuously updated through iterative rounds of feature extractor extraction process to obtain the final updated semantic prototype of each category; Determine a probability map of the semantic prototype of each category of the multiple source domain images after the final update based on the semantic prototype of each category after the final update, the semantic prototypes of all categories after the final update, and the feature map of each source domain image; Based on the probability map of the semantic prototype of each category of multiple source domain images after the final update, the predicted segmentation results corresponding to the multiple source domain images are obtained; In the process of obtaining the predicted segmentation results of multiple source domain images, the parameters of the image segmentation network framework are tuned through the cross-domain migration correction module, the multi-scale prototype guidance mechanism and the prototype reverse guidance module to obtain the final image segmentation network framework; Obtain the target domain image to be detected, input the target domain image to be detected into the final image segmentation network framework, and obtain the segmentation result of the target domain image to be detected.

2. The method according to claim 1, characterized in that The cross-domain migration correction module includes: After determining the semantic prototype of each category after the multiple source domain images are finally updated, multiple target domain images and the category label of each image in each target domain image are obtained, the multiple target domain images are equally divided to obtain a first part of target domain images and a second part of target domain images, and a feature extractor is used to extract features from each target domain image in the first part of target domain images and the second part of target domain images, respectively, to obtain a feature map of each target domain image in the first part of target domain images and a feature map of each target domain image in the second part of target domain images; determining a semantic prototype of each category in the first part of target domain images based on a feature map of each target domain image and a category label of each image in each target domain image; Use the metric function to calculate the initial prototype similarity between the semantic prototype of each category in the first part of the target domain image and the semantic prototype of each category after the final update of multiple source domain images; Based on the initial prototype similarity and preset hyperparameters, the prototype comparison loss function is determined.

3. The method according to claim 1, characterized in that The cross-domain migration correction module also includes: After determining the probability map of the semantic prototype of each category of the multiple source domain images after final update, a first target segmentation loss function is determined based on the probability map of the semantic prototype of each category of the multiple source domain images after final update and the category label of each image in each source domain image.

4. The method according to claim 1, wherein The cross-domain migration correction module also includes: After determining the probability map of the semantic prototype of each category of the multiple source domain images after final update, segmenting the feature map of each target domain image in the second part of the target domain images using the semantic prototype of each category of the multiple source domain images after final update, to obtain the probability map of the semantic prototype of each category of the second part of the target domain images; Based on the probability map of the semantic prototype of each category of the second part target domain image, the predicted segmentation result corresponding to the second part target domain image is obtained; The transfer alignment loss function is determined based on the probability map of the semantic prototype of each category of the second part target domain image and the category label of each image in each target domain image of the second part target domain image.

5. The method according to claim 1, wherein The cross-domain migration correction module also includes: After determining the probability map of the semantic prototype of each category of the multiple source domain images after final update, segmenting the feature map of each target domain image in the first part of the target domain images using the semantic prototype of each category after final update of the multiple source domain images to obtain the probability map of the semantic prototype of each category in the first part of the target domain images; Processing the probability map of the semantic prototype of each category in the first part of the target domain images through the first function to obtain a prediction result for each category in each target domain image in the first part of the target domain images; Performing feature extraction on each target domain image in the first part of the target domain images using a preset feature extractor to obtain a feature map of each target domain image in the first part of the target domain images, inputting the feature map of each target domain image in the first part of the target domain images into a decoder to obtain a prediction probability map corresponding to each category in the first part of the target domain images, wherein each category corresponds to a prediction probability map; A second target segmentation loss function is determined based on a probability map of a semantic prototype of each category in the first part of the target domain images and a prediction result of a category of each image in each target domain image in the first part of the target domain images.

6. The method according to claim 1, wherein The cross-domain migration correction module also includes: Based on the prototype contrast loss function, the first target segmentation loss function, the transfer alignment loss function and the second target segmentation loss function, the total target segmentation loss function of the cross-domain transfer correction module is determined.

7. The method according to claim 1, characterized in that Multi-scale prototype guidance mechanism, including: Four feature extractors with different spatial resolution scales are used to extract features from each source domain image, and feature maps of four different spatial resolution scales are obtained for each source domain image. Based on the feature maps of each source image at different spatial resolution scales and the category label of each image in each source image, the semantic prototype of each category at each spatial resolution in multiple source images is determined; The semantic prototypes of each category at each spatial resolution in multiple source domain images are continuously updated through iterative rounds of feature extractors with four different spatial resolution scales, and the final updated semantic prototypes of each category at each spatial resolution are obtained; Based on the semantic prototype of each category at each spatial resolution after the final update, the semantic prototypes of all categories at the four spatial resolutions after the final update, and the feature maps of the four different spatial resolution scales of each source domain image, determine the probability map of the semantic prototype of each category at each spatial resolution after the final update of multiple source domain images; Upsampling the sizes of the probability maps of the semantic prototypes of each category at each spatial resolution after the final update of multiple source domain images so that the sizes of the probability maps of the semantic prototypes of each category at each spatial resolution after the final update are the same; The probability maps of the semantic prototypes of each category at each spatial resolution with the same final updated size are fused to obtain the target probability map; Based on the probability map of the semantic prototype of each category at each spatial resolution of multiple source domain images after the final update and the category label of each image in each source domain image, the target segmentation loss function of the multi-scale prototype guidance mechanism is determined.

8. The method according to claim 1, characterized in that Prototype reverse guidance module, including: Acquire multiple target domain images, where each target domain image is an unlabeled image; Use the preset feature extractor to extract features from each unlabeled target domain image to obtain a feature map of each unlabeled target domain image; The feature map of each unlabeled target domain image is segmented using the semantic prototype of each category in the first part of the target domain image, and the probability map of the semantic prototype of each category of multiple unlabeled target domain images is obtained; The probability maps of the semantic prototypes of each category of the multiple unlabeled target domain images are processed by the first function to obtain the pseudo label of each category in each of the multiple unlabeled target domain images; Based on the pseudo label of each category in each unlabeled target domain image and the feature map of each unlabeled target domain image in multiple unlabeled target domain images, a semantic prototype of each category in the multiple unlabeled target domain images is obtained; Using the semantic prototype of each category in multiple unlabeled target domain images to guide segmentation of the first part of the target domain images and the second part of the target domain images, a probability map guided by the unlabeled prototype of the first part of the target domain images and a probability map guided by the unlabeled prototype of the second part of the target domain images are obtained; The target segmentation loss function of the prototype reverse guidance module is determined according to the unlabeled prototype-guided probability map of the first part of the target domain images, the unlabeled prototype-guided probability map of the second part of the target domain images, the true label of each image in the first part of the target domain images, and the true label of each image in the second part of the target domain images.

9. A processor, characterized in that: The processor is configured to run a program, wherein the program executes the method according to any one of claims 1 to 8 when running.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 8.

Images