Multi-source domain adaptive semantic segmentation method and device based on multi-level domain correlation

Abstract

The present invention discloses a multi-source domain adaptive semantic segmentation method based on multi-level domain relevance: (1) pre-training a cross-domain semantic segmentation model F on all source domains and target domains; calculating the multi-level domain relevance based on the pre-trained F, including the domain-level source-target relevance d i And pixel-level source-target correlation is based on constructing source-target domain mixed images and corresponding pseudo labels based on random sampling method. i , and , train F on all source and target domains; then use the trained F to perform multi-source domain adaptive semantic segmentation on the image to be detected, extracting the target object in the image. This method increases the weights of source domains and source domain pixels with high correlation with the target domain, while reducing the weights of source domains and source domain pixels with low correlation with the target domain, thereby preventing interference from irrelevant information in multiple source domains on training.

Description

Technical Field

[0001] The present invention belongs to the technical field of remote sensing image semantic segmentation, and in particular relates to a multi-source domain adaptive semantic segmentation method based on multi-level domain correlation. Background Art

[0002] Semantic segmentation is a pixel-level image interpretation task that aims to assign a semantic category to each pixel, thereby extracting the target object in the image (such as mountains and water in landscape photos). In recent years, with the rapid development of deep neural networks, semantic segmentation has attracted widespread attention in computer vision and achieved remarkable progress. However, achieving satisfactory performance often requires a large amount of real-world data and expensive fine-grained semantic label annotation. To overcome this bottleneck, a natural solution is to construct synthetic datasets to train semantic segmentation models and then use them in real-world settings. However, due to the severe domain shift between real and synthetic data, directly applying models trained on synthetic images to segment real images leads to significant performance degradation. To address this issue, unsupervised domain adaptation (UDA) methods have been proposed to narrow the distribution gap between training and test data. In practice, UDA methods have received considerable attention because they do not require any annotation of the target domain, ultimately minimizing labeling costs. However, most existing UDA methods focus on single-source domain adaptation (SSDA). Only a few works have considered more practical scenarios, namely, multiple labeled source datasets with various distributions, such as the SYNTHIA dataset and the GTA5 dataset. Training on multi-source data with different distributions can encourage the model to learn more complementary knowledge to achieve cross-domain semantic segmentation. A straightforward approach is to simply mix all source domains into one domain and then train the UDA model on the mixed source domain, just like the common SSDA method. While this simple approach often improves model performance, it fails to fully utilize the rich and valuable information across multiple source domains, which is conducive to learning more powerful cross-domain segmentation models.

[0003] To better utilize multi-source domain information, multi-source domain adaptation (MSDA) methods have been proposed to transfer models from multiple source domains (multiple semantically labeled training datasets) to a single target domain (a single unlabeled test dataset). These MSDA methods can better utilize multi-source domains and achieve better adaptation performance than SSDA methods. However, in addition to MADAN [Sicheng Zhao, Bo Li, Pengfei Xu, Xiangyu Yue, Guiguang Ding, and Kurt Keutzer. 2021. MADAN: multi-source adversarial domain aggregation network for domain adaptation [C]. International Journal of ComputerVision 129, 8 (2021), 2399–2424.], MDACL [Jianzhong He, Xu Jia,Shuaijun Chen,andJianzhuang Liu.2021.Multi-source domain adaptation with collaborativelearning for semantic segmentation[C].In Proceedings of the IEEE Conference onComputer Vision and Pattern Recognition.11008–11017.]and MDAPLR[So Jeong Park,Hae Ju Park,Eun Su Kang,Ba HungNgo,Ho Sub Lee,and Sung In Cho.2022.PseudoLabel Rectification via Co-Teaching andDecoupling for Multisource DomainAdaptation in Semantic Segmentation[J].IEEE Access 10(2022),91137–91149.] In addition to working on pixel-level semantic segmentation tasks, most algorithms focus on image-level classification tasks.Specifically, Zhao et al. [Sicheng Zhao, Bo Li, Pengfei Xu, Xiangyu Yue, Guiguang Ding, and Kurt Keutzer. 2021. MADAN: multi-source adversarial domain aggregation network for domain adaptation [C]. International Journal of Computer Vision 129, 8 (2021), 2399–2424.] align multiple source and target domains in the image space and eliminate the distribution offset between multiple source images. He et al. [Jianzhong He, Xu Jia, Shuaijun Chen, and Jianzhuang Liu. 2021. Multi-source domain adaptation with collaborative learning for semantic segmentation [C]. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 11008–11017] first stylize the source image to the target style, and then train multiple segmentation models to learn common semantic knowledge across multiple stylized source and target images through collaborative learning. These methods have indeed improved the performance of MSDA in semantic segmentation, but they ignore the impact of different source domains on domain adaptation. Since MSDA involves multiple source domains with different data distributions, some source domains with data distributions close to (similar to) the target domain contribute to domain adaptation, while some dissimilar source domains are detrimental to domain adaptation. Simply treating all source domains equally is not the optimal solution for the MSDA problem. To address this issue, Zuo et al. [Yukun Zuo, Hantao Yao, and Changsheng Xu. 2021. Attention-based multi-source domain adaptation [J]. IEEE Transactions on Image Processing 30 (2021), 3793–3803.] proposed estimating the similarity between the target domain and each source domain. They then align multiple source domains with the target domain by weighting the different domains, thereby reducing the negative impact of dissimilar source domains. However, these MSDA methods are only applicable to image-level classification tasks. Since they do not transfer pixel-level knowledge, they cannot be directly applied to semantic segmentation tasks.Furthermore, for pixel-level segmentation tasks, existing MSDA methods ignore the fact that even in similar source domains, some irrelevant pixels can affect adaptive performance. These irrelevant source pixels can lead to performance degradation, while relevant source pixels can significantly improve adaptive performance. Therefore, to improve the performance of multi-source domain adaptation, it is necessary to minimize the influence of irrelevant source domains and the pixels within them.

[0004] Existing multi-source domain-adaptive semantic segmentation methods ignore the importance of domain-level source-target correlation (DSC) and pixel-level source-target correlation (PSC) between domains. Similar existing multi-source domain-adaptive image classification methods, while considering the similarity between the target domain and each source domain, do not account for irrelevant pixels in similar source domains, which can also degrade model performance. Furthermore, multi-source domain-adaptive image classification methods do not transfer pixel-level knowledge and are therefore not suitable for multi-source domain-adaptive semantic segmentation methods. Summary of the Invention

[0005] The purpose of the present invention is to provide a multi-source domain adaptive semantic segmentation method and device based on multi-level domain relevance, which increases the weight of source domains and source domain pixels with high correlation with the target domain, and reduces the weight of source domains and source domain pixels with low correlation with the target domain, so as to avoid the interference of irrelevant information in multiple source domains on training.

[0006] The present invention provides the following technical solutions:

[0007] A multi-source domain adaptive semantic segmentation method based on multi-level domain relevance, the method comprising:

[0008] (1) In all source domains and target domain Pre-trained cross-domain semantic segmentation model F;

[0009] (2) Calculate multi-level domain relevance based on the pre-trained cross-domain semantic segmentation model F, including the domain-level source-target relevance d i and pixel-level source-target correlation (h, w) indicates the position coordinates of the pixel in the image;

[0010] (3) Constructing source-target domain hybrid images based on pixel-level source-target correlation and the corresponding pseudo labels

[0011] (4) Constructing source-target domain mixed images based on random sampling method and the corresponding pseudo labels

[0012] (5) Based on the domain-level source-target correlation d i , pixel-level source-target correlation Source-target domain mixed image and the corresponding pseudo labels and source-target domain mixed images and the corresponding pseudo labels In all source domains and target domain Train the cross-domain semantic segmentation model F;

[0013] (6) The trained cross-domain semantic segmentation model F performs multi-source domain adaptive semantic segmentation on the image to be detected to extract the target object in the image.

[0014] In this invention, given N source domains with semantic labels and a target domain without semantic labels Where i∈N, each source domain All contain images and the corresponding semantic labels Assume that the semantic label contains C types of objects, the target domain Contains image x t , H is the image length, and W is the image width.

[0015] In step (1), the method of pre-training the cross-domain semantic segmentation model F is:

[0016]

[0017] in: For all source domains Semantic segmentation training loss on For all source domains and target domain Multi-source domain adaptation loss between all source domains Include images and the corresponding semantic labels Target domain Contains image x t .

[0018] In step (2), the domain-level source-target correlation Calculated source domain and target domain The correlation between is a domain-level source-target correlation calculation function; the pixel-level source-target correlation Calculated source domain Any pixel in and target domain The correlation between them is δ, and δ is the pixel-level source-target correlation calculation function.

[0019] In step (3), based on the pixel-level source-target correlation Select and crop the source domain image Pixels in and Then paste it into the target domain image x t and pseudo labels On top, construct the source-target domain mixed image and pseudo labels

[0020] Specifically, based on the pixel-level source-target correlation Select and crop the source domain image Pixels with high correlation with the target domain and It can be determined according to actual needs.

[0021] Furthermore, the source-target domain mixed image and the corresponding pseudo labels is calculated as follows:

[0022]

[0023]

[0024] in, To select the indicator matrix, the size is H×W, Indicates that the pasted pixels come from the source domain image Indicates that the pasted pixel comes from the target domain image x t :

[0025] Specifically, The calculation method is:

[0026]

[0027] in, is the correlation threshold, and is a hyperparameter. Preferably, ).

[0028] In step (4), given a source domain image and the target domain image x t , select and crop source domain images based on random sampling method Pixels in and Then paste it into the target domain image xt and pseudo labels On top, construct the source-target domain mixed image and pseudo labels

[0029] Preferably, the source domain image is selected and cropped Pixels from a random half of the categories in .

[0030] In step (5), the method for training the cross-domain semantic segmentation model F is:

[0031]

[0032] Among them, · represents the weighted operation, For all source domains Semantic segmentation training loss, For all source-target domain mixed images and the corresponding pseudo labels Semantic segmentation training loss on For all source-target domain mixed images and the corresponding pseudo labels Semantic segmentation training loss on For all source domains and target domain Multi-source domain adaptation loss between ; Include images and the corresponding semantic labels i∈N, target Contains image x t .

[0033] The present invention also provides a multi-source domain adaptive semantic segmentation device based on multi-level domain relevance, comprising a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the above-mentioned multi-source domain adaptive semantic segmentation method based on multi-level domain relevance.

[0034] The present invention also provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, is used to implement the above-mentioned multi-source domain adaptive semantic segmentation method based on multi-level domain relevance.

[0035] In the present invention, the multi-source domain adaptive semantic segmentation method and device are used to extract target objects in images, such as mountains and water in landscape photos, people in portrait photos, cats and dogs in pet photos, lesion sites in medical images, buildings and other landforms in remote sensing images, pedestrians and cars in autonomous driving images, etc.

[0036] The present invention calculates and uses domain-level source-target correlation and pixel-level source-target correlation when performing multi-source domain adaptation and source-target mixed sampling (data enhancement), thereby improving the training performance of the model and making it more suitable for multi-source domain adaptive semantic segmentation.

[0037] Based on the existing multi-source domain adaptive training, the present invention increases the weights (influence on training) of source domains and source domain pixels with high correlation with the target domain, while reducing the weights (influence) of source domains and source domain pixels with low correlation with the target domain, so as to avoid the interference of irrelevant information in multi-source domains on training. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A flowchart of a multi-source domain adaptive semantic segmentation method based on multi-level domain relevance provided by the present invention;

[0039] Figure 2 This is an example diagram of constructing a source-target mixed image based on pixel-level source-target correlation in an embodiment. DETAILED DESCRIPTION

[0040] In order to better understand the technical solution of this application, the following examples are used to describe the implementation of the invention. It should be clear that the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this application.

[0041] The terms used in the embodiments of the present application are for the purpose of describing specific embodiments only and are not intended to limit the present application. The singular forms "a", "an", "the" and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise.

[0042] In the embodiment provided in this application: given N source domains with semantic labels and a target domain without semantic labels Each source domain All contain images and the corresponding semantic labels (assuming the semantic label contains C types of objects), the target domain Contains only image x t Multi-source domain adaptation aims to adapt the semantic segmentation model F = E·G (E is the feature extractor in the semantic segmentation model, G is the classifier in the semantic segmentation model) trained on multiple source domains to the target domain.

[0043] like Figure 1As shown, the multi-source domain adaptive semantic segmentation method based on multi-level domain relevance provided by this embodiment includes the following steps:

[0044] (1) Cross-domain semantic segmentation model pre-training: in all source domains (Including images and the corresponding semantic labels ) and target domain (contains image x t ) is used to pre-train a cross-domain semantic segmentation model F.

[0045] The pre-training formula is:

[0046]

[0047] in: For all source domains (including images and the corresponding semantic labels ) is a commonly used (general purpose) semantic segmentation training loss on . For all source domains (including images and the corresponding semantic labels ) and target domain (contains image x t ) is a commonly used (general) multi-source domain adaptation loss between .

[0048] In the present invention, the semantic segmentation model F can adopt a general semantic segmentation model structure without special restrictions.

[0049] In this example, the semantic segmentation training loss The cross-entropy loss function can be used:

[0050]

[0051] In this embodiment, the multi-source domain adaptive loss The entropy minimization loss function can be used:

[0052]

[0053] (2) Calculation of multi-level domain relevance (domain-level source-target relevance, pixel-level source-target relevance): Multi-level domain relevance is calculated based on the pre-trained cross-domain semantic segmentation model F. The domain-level source-target relevance d i and pixel-level source-target correlation (h∈H, w∈W, (h, w) indicates the position coordinates of the pixel point in the image).

[0054] Among them, the domain-level source-target correlation Calculated source domain and target domain The correlation between is the domain-level source-target correlation calculation function), pixel-level source-target correlation Calculated source domain Any pixel in and target domain The correlation between them (δ is the pixel-level source-target correlation calculation function).

[0055] In this embodiment, the domain-level source-target correlation d i The calculation method can be (exp is the exponential function, c∈C is the category, is the source domain (or target domain ) of the prototype of category c in ):

[0056]

[0057] Specifically, the domain (source domain or target domain ) is calculated as follows (* is matrix dot multiplication, is the indicator function (when F(x) (h,w) = = c, Otherwise, it is 0. The indicator function reference: Zhou Zhihua. Machine Learning [M]. Beijing: Tsinghua University Press, 2016. Main symbol table):

[0058]

[0059] In this embodiment, the pixel-level source-target correlation The calculation method can be (exp is the exponential function, α, β are hyperparameters (for example, α = 2, β = 2), Source domain Any pixel in To the nearest target domain prototype distance, For all pixels in the image Corresponding Average value of

[0060]

[0061] Specifically, The calculation method is (the min function takes the minimum value of all distance values):

[0062]

[0063] (3) Constructing source-target domain hybrid images based on pixel-level source-target correlation and the corresponding pseudo labels

[0064] Given a source domain image and the target domain image x t , based on pixel-level source-target correlation Select and crop the source domain image Pixels with high correlation with the target domain (set) Then paste it into the target domain image x t On top, construct the source-target domain mixed image

[0065] At the same time, the target domain image x t Input the cross-domain semantic segmentation model F to get F(x t ), and F(x t ) as the pseudo label of the target domain image

[0066] Similarly, select and cut the source domain image Corresponding tags Pixels with high correlation with the target domain (set) Then paste it into the target domain image x t The corresponding pseudo labels On the source-target domain, construct the pseudo label of the mixed image

[0067] Specifically, the source-target domain mixed image and the corresponding pseudo labels The calculation method is as follows ( To select the indicator matrix, the size is H×W, Indicates that the pasted pixels come from the source domain image Indicates that the pasted pixel comes from the target domain image x t ):

[0068]

[0069]

[0070] Specifically, The calculation method is ( is the correlation threshold, a hyperparameter, e.g. ):

[0071]

[0072] like Figure 2As shown in, it is an example of constructing a source-target mixed image based on pixel-level source-target correlation; Figure 2 In the figure: (a) is the source domain image (Source Image), (b) is the target domain image (Target Image), (c) is the pixel points in the source domain image that are highly correlated with the target domain, and (d) is the constructed source-target mixed image (pasting (c) onto (b)).

[0073] (4) Constructing source-target mixed images based on random sampling method The corresponding pseudo labels

[0074] Given a source domain image and the target domain image x t , select and crop the source domain image Randomly select half of the categories (including pixels) in the image and paste them onto the target domain image x t , construct source-target domain mixed image At the same time, build a hybrid image The corresponding pseudo labels

[0075] Source-target domain mixed images and the corresponding pseudo labels The construction method is similar to (3), the only difference is that Replace with Randomly select source domain images Half of the categories (the pixels they contain).

[0076] (5) Cross-domain semantic segmentation model training based on multi-level domain relevance: Based on the domain-level source-target relevance d i and pixel-level source-target correlation In all source domains (including images and the corresponding semantic labels ) and target domain (contains image x t ) to train a cross-domain semantic segmentation model F.

[0077] The training formula is:

[0078]

[0079] Where: · represents a weighted operation. For all source domains (including images and the corresponding semantic labels ) commonly used (general purpose) semantic segmentation training loss on ; For all source-target domain mixed images and the corresponding pseudo labels Common (general) semantic segmentation training loss on ; For all source-target domain mixed images and the corresponding pseudo labels Common (general) semantic segmentation training loss on ; For all source domains (including images and the corresponding semantic labels ) and target domain (contains image x t ) is a commonly used (general) multi-source domain adaptation loss between .

[0080] In this example, the semantic segmentation training loss The cross-entropy loss function can be used:

[0081]

[0082] In this example, the semantic segmentation training loss The cross-entropy loss function can be used:

[0083]

[0084] In this embodiment, the semantic segmentation training loss based on multi-level domain relevance is The cross-entropy loss function can be used:

[0085]

[0086] In this embodiment, the multi-source domain adaptive loss based on multi-level domain correlation The entropy minimization loss function can be used:

[0087]

[0088] As shown in Tables 1-3, the results of the intersection-over-union comparison between the proposed method and the most advanced SSDA method and MSDA method are compared (from the multi-source autonomous driving synthesis (SYNTHIA) [German Ros, Laura Sellart, Joanna Materzynska, David Vazquez, and Antonio M Lopez. 2016. The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes [C]. In Proceedings of the IEEE conference on computer vision and pattern recognition. 3234–3243.] dataset and the autonomous driving game simulation (GTA5) [Stephan RRichter, Vibhav Vineet, Stefan Roth, and Vladlen Koltun. 2016. Playing for data: Ground truth from computer games [C]. In Proceedings of the European conference on computer vision. 102–118.] dataset to the autonomous driving real city landscape (cityscape) [Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. 2016. The cityscapes dataset for semantic urban scene understanding [C]. In Proceedings of the IEEEconference on computer vision and pattern recognition. 3213–3223.] dataset).

[0089] Here, S, G, and A represent the use of different datasets (SYNTHIA, GTA5, and All) to train the model. mIoU represents the mean intersection-over-union ratio.

[0090] Tables 1-3 show that different source domains have different effects on domain adaptation. Baseline source-only methods (without domain adaptation), such as ProDA and CPSL, trained on GTA5 achieve better adaptation performance than similar models trained on SYNTHIA. This is primarily due to the higher correlation between the GTA5 and Cityscapes datasets than between SYNTHIA and Cityscapes. Furthermore, using multiple source domains significantly improves the generalization performance of baseline source-only models (without domain adaptation) on Cityscapes. For example, the source-only model improves from 32.3% (SYNTHIA → Cityscapes) and 36.4% (GTA5 → Cityscapes) to 41.1% (SYNTHIA + GTA5 → Cityscapes) on Cityscapes. However, training SSDA methods directly on both SYNTHIA and GTA5 does not yield significant improvements; for example, BiSMAP achieves only a 0.4% improvement. This is because SSDA methods do not consider the complex domain shifts between multiple source and target domains, especially the negative impact of low-correlation source domains and pixels. An interesting phenomenon is that due to the rapid development of adaptive techniques in SSDA methods, the performance of some SSDA methods has surpassed many MSDA methods. By fully utilizing highly correlated source domains and reducing the negative impact of noisy pixels, the method provided by this invention achieved 63.8% mIoU on 16 categories, an improvement of more than 3.3% compared to all SSDA and MSDA methods.

[0091] Table 1 Performance comparison of the present invention and different types of multi-source domain adaptive semantic segmentation

[0092]

[0093]

[0094] Table 2 Performance comparison of the present invention and different types of multi-source domain adaptive semantic segmentation

[0095]

[0096]

[0097] Table 3 Performance comparison of the present invention and different types of multi-source domain adaptive semantic segmentation

[0098]

[0099]

[0100] When performing domain adaptation, pixels in the source domain image that are similar to and highly correlated with the target domain are selected for adaptive training, which helps the segmentation model better learn the potential features of the target domain. However, source domains and source domain pixels that are different from the target domain and have low correlation are often not conducive to domain adaptation. Therefore, when adapting the model from multiple source domains to the target domain, it is necessary to reduce the influence of low-correlation source domains and source domain pixels to enhance the adaptive learning of the target model: Existing multi-source domain adaptation algorithms usually only use step (1), ignoring the domain-level source-target correlation (DSC) and pixel-level source-target correlation (PSC) between domains, resulting in irrelevant domains and pixels in the source domain weakening the model performance; existing source-target mixed sampling (data enhancement) techniques, such as step (4), ignore the pixel-level source-target correlation.

[0101] In contrast, the method provided by the present invention calculates and uses domain-level source-target correlation and pixel-level source-target correlation when performing multi-source domain adaptation and source-target mixed sampling (data enhancement), thereby improving the training performance of the model.

[0102] The embodiment of the present invention also provides a multi-source domain adaptive semantic segmentation device based on multi-level domain relevance, including one or more processors, and an executable code stored in the memory. When the processor executes the executable code, it is used to implement the multi-source domain adaptive semantic segmentation method based on multi-level domain relevance in the above embodiment. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capability in which it is located to read the corresponding computer program instructions in the non-volatile memory into the memory and run it. From the hardware level, in addition to the processor, memory, network interface, and non-volatile memory, the device with data processing capability in which the device in the embodiment is located can also include other hardware according to the actual function of the device with data processing capability, which will not be described in detail.

[0103] An embodiment of the present invention also provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, it implements the multi-source domain adaptive semantic segmentation method based on multi-level domain relevance described in the above embodiment: the computer-readable storage medium can be an internal storage unit of any device with data processing capabilities described in any of the above embodiments, such as a hard disk or memory. The computer-readable storage medium can also be any device with data processing capabilities, such as a plug-in hard disk, smart media card (SMC), SD card, flash memory card (Flash8 Card), etc. equipped on the device. Furthermore, the computer-readable storage medium can also include both an internal storage unit of any device with data processing capabilities and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and can also be used to temporarily store data that has been output or is to be output.

[0104] The foregoing is merely an embodiment of the present invention and is not intended to limit the present invention. It will be apparent to those skilled in the art that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention are intended to be included within the scope of the claims of the present invention.

Claims

1. A multi-source domain adaptive semantic segmentation method based on multi-level domain relevance, characterized by: The method comprises: (1) In all source domains and target domain Pre-trained cross-domain semantic segmentation model F; (2) Calculate multi-level domain relevance based on the pre-trained cross-domain semantic segmentation model F, including the domain-level source-target relevance d i and pixel-level source-target correlation h∈H, w∈W, (h, w) indicates the position coordinates of the pixel in the image; (3) Constructing source-target domain hybrid images based on pixel-level source-target correlation and the corresponding pseudo labels (4) Constructing source-target domain mixed images based on random sampling method and the corresponding pseudo labels (5) Based on the domain-level source-target correlation d i , pixel-level source-target correlation Source-target domain mixed image and the corresponding pseudo labels and source-target domain mixed images and the corresponding pseudo labels In all source domains and target domain Train the cross-domain semantic segmentation model F; (6) The trained cross-domain semantic segmentation model F performs multi-source domain adaptive semantic segmentation on the image to be detected to extract the target object in the image; In step (1), the method of pre-training the cross-domain semantic segmentation model F is: in: For all source domains Semantic segmentation training loss on For all source domains and target domain Multi-source domain adaptation loss between all source domains Include images and the corresponding semantic labels i∈N, target domain Contains image x t ; In step (3), based on the pixel-level source-target correlation Select and crop the source domain image Pixels in and Then paste it into the target domain image x t and pseudo labels On top, construct the source-target domain mixed image and pseudo labels Source-target domain mixed image and the corresponding pseudo labels is calculated as follows: in, To select the indicator matrix, the size is H×W, Indicates that the pasted pixels come from the source image Indicates that the pasted pixel comes from the target domain image x t : Specifically, The calculation method is: in, is the correlation threshold, which is a hyperparameter; In step (5), the method for training the cross-domain semantic segmentation model F is: Among them, · represents the weighted operation, For all source domains Semantic segmentation training loss, For all source-target domain mixed images and the corresponding pseudo labels Semantic segmentation training loss on For all source-target domain mixed images and the corresponding pseudo labels Semantic segmentation training loss on For all source domains and target domain Multi-source domain adaptation loss between ; Include images and the corresponding semantic labels i∈N, target Contains image x t .

2. The multi-source domain adaptive semantic segmentation method based on multi-level domain relevance according to claim 1 is characterized in that: Given N source domains with semantic labels and a target domain without semantic labels Where i∈N, each source domain All contain images and the corresponding semantic labels Assume that the semantic label contains C types of objects, the target domain Contains image x t , H is the image length, and W is the image width.

3. The multi-source domain adaptive semantic segmentation method based on multi-level domain relevance according to claim 1 is characterized in that: In step (2), the domain-level source-target correlation Calculated source domain and target domain The correlation between is a domain-level source-target correlation calculation function; the pixel-level source-target correlation Calculated source domain Any pixel in and target domain The correlation between them is δ, and δ is the pixel-level source-target correlation calculation function.

4. The multi-source domain adaptive semantic segmentation method based on multi-level domain relevance according to claim 1 is characterized in that: In step (4), given a source domain image and the target domain image x t , select and crop source domain images based on random sampling method Pixels in and Then paste it into the target domain image x t and pseudo labels On top, construct the source-target domain mixed image and pseudo labels 5. A multi-source domain adaptive semantic segmentation device based on multi-level domain relevance, comprising a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the multi-source domain adaptive semantic segmentation method based on multi-level domain relevance described in any one of claims 1-4.

6. A computer-readable storage medium having a program stored thereon, which, when executed by a processor, is used to implement the multi-source domain adaptive semantic segmentation method based on multi-level domain relevance described in any one of claims 1-4.

Images