Image semantic segmentation method and device
Through the method based on semantic prototype comparison learning, the semantic segmentation model is trained using cross entropy loss function and loss function, and the semantic prototype center is updated, which solves the problem of class pixel dispersion in the existing technology, and improves the robustness and self-function of the semantic segmentation model. Adaptability.
Patent Information
- Application Number
- CN202111016029.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-08-31
- Publication Date
- 2024-12-31
- Estimated Expiration
- 2041-08-31
AI Technical Summary
In the existing domain adaptive semantic segmentation technology, pixels of the same category are dispersed in the feature space, and the boundaries of pixels of different categories are not obvious, resulting in poor robustness of the semantic segmentation model.
The semantic prototype comparison learning method is adopted to train the semantic segmentation model through cross-entropy loss function and loss function, and use the contrast learning model to update the semantic prototype center, improving the aggregation of similar pixels and the separation ability of different pixels.
It improves the accuracy and stability of the semantic segmentation model, enhances the adaptive ability of domain migration, so that similar pixels gather in the feature space and different pixels are separated.
Smart Images
Figure CN113902913B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing technology, and in particular to a domain adaptive image semantic segmentation method and device based on semantic prototype contrast learning. Background Art
[0002] Domain adaptation, as a type of transfer learning, is an important and challenging task in the field of machine learning. It has wide applications in image recognition, target detection, and image semantic segmentation.
[0003] In recent years, with the development of unsupervised adaptive semantic segmentation technology, various novel domain alignment methods, such as style transfer at the image level and alignment at the feature or segmentation output level using generative adversarial networks, have achieved certain results. However, the boundaries between different categories of pixels generated by these methods are not obvious, and pixels of the same category are scattered in the feature space, resulting in poor semantic segmentation accuracy, which affects the robustness of the semantic segmentation model. Summary of the invention
[0004] The present invention provides a domain adaptive image semantic segmentation method and device based on semantic prototype contrast learning, so as to solve the defects in the prior art that pixels of the same category are scattered in the feature space and the boundaries of pixels of different categories are not obvious in the feature space, resulting in poor robustness of the semantic segmentation model, so as to improve the adaptive ability of semantic segmentation domain migration and strengthen the positive concentration and negative separation characteristics of the network for cross-domain pixel representation.
[0005] The present invention provides a domain adaptive image semantic segmentation based on semantic prototype contrast learning, comprising: obtaining a source image and its corresponding real label from a source domain, and obtaining a target image from a target domain; inputting the source image and the target image into a semantic segmentation model respectively to obtain a prediction result; the semantic segmentation model is obtained by training using a cross entropy loss function and a loss function; wherein the cross entropy loss function is obtained based on the prediction result corresponding to the source image and its corresponding real label; and the loss function is obtained based on inputting the real label, the prediction result and the intermediate output of the semantic segmentation model into a contrast learning model.
[0006] According to a domain adaptive image semantic segmentation based on semantic prototype contrast learning provided by the present invention, the semantic segmentation model includes: a feature extraction layer, respectively extracting semantic features of the input source image and the target image to obtain corresponding feature maps; wherein the feature maps include source feature maps and target feature maps; a semantic segmentation layer, respectively performing semantic segmentation based on the source feature maps and the target feature maps to obtain corresponding segmentation results; wherein the segmentation results include a source segmentation result corresponding to the source feature map and a target segmentation result corresponding to the target feature map; a result output layer, adding corresponding labels to the source segmentation result and the target segmentation result to obtain corresponding prediction results; wherein the target prediction results include a source prediction result corresponding to the source segmentation result and a target prediction result corresponding to the target segmentation result.
[0007] According to a domain adaptive image semantic segmentation based on semantic prototype contrast learning provided by the present invention, the loss function includes a source contrast loss function, the intermediate output includes a true label, and the contrast learning model includes: a first mask layer, which obtains a source mask of a source image based on an input true label; a first center acquisition layer, which updates a first category center of a source prediction result of the same semantic prototype based on the source mask; and a first function acquisition layer, which obtains a source contrast loss function based on the source mask and the first category center.
[0008] According to a domain adaptive image semantic segmentation based on semantic prototype contrast learning provided by the present invention, the loss function includes a target contrast loss function, the intermediate output includes a target segmentation result, and the contrast learning model includes: a first confidence layer, which generates a first confidence map based on the input target segmentation result; a second mask layer, which obtains a target mask according to the confidence map; a second center acquisition layer, which updates the second category center of the target prediction result of the same semantic prototype based on the target mask; and a second function acquisition layer, which obtains a target contrast loss function based on the target mask and the second category center.
[0009] According to a domain adaptive image semantic segmentation based on semantic prototype contrast learning provided by the present invention, the loss function includes a self-supervised loss function, the intermediate output includes a target prediction result, and the contrast learning model includes: a second confidence layer, which generates a second confidence map based on the input target prediction result; a pseudo-label prediction layer, which generates a pseudo-label based on the second confidence; and a third function acquisition layer, which uses the pseudo-label and the target prediction result to obtain a self-supervised loss function.
[0010] According to a domain adaptive image semantic segmentation based on semantic prototype contrast learning provided by the present invention, the loss function includes a source contrast loss function, a target contrast loss function and a self-supervised loss function;
[0011] The source contrast loss function is expressed as:
[0012]
[0013] in, represents the source contrast loss function; Ⅱ represents the indicator function, if If the condition is met, it returns 1, otherwise it returns 0; h', w' respectively represent the spatial dimensions of the corresponding image; c represents the number of source masks; M si represents the i-th source mask; F si represents the i-th source feature map; represents the first category center of the cth semantic prototype corresponding to the source image, and denote the first category center of positive and negative semantic prototypes respectively; l(·) denotes the loss function of the source feature map;
[0014] The target contrast loss function is expressed as:
[0015]
[0016] in, represents the target contrast loss function; Ⅱ represents the indicator function, if If the condition is met, it returns 1, otherwise it returns 0; h', w' respectively represent the spatial dimensions of the corresponding image; c represents the number of target masks; M tj represents the jth target mask; F tj Represents the jth target feature map; represents the second category center of the cth semantic prototype corresponding to the target image, and denote the second category center of positive and negative semantic prototypes respectively; l(·) denotes the loss function of the target feature map;
[0017] The self-supervised loss function is expressed as:
[0018]
[0019] Among them, L ssl represents the self-supervised loss function; Ⅱ represents the indicator function, if If the condition is met, it returns 1, otherwise it returns 0; h' and w' represent the spatial dimensions of the corresponding image; c represents the number of pseudo labels; represents the jth pseudo label; P j Represents the j-th target prediction result.
[0020] The present invention also provides a domain adaptive image semantic segmentation device based on semantic prototype contrast learning, comprising: a data acquisition module, which acquires a source image and its corresponding true label from a source domain, and acquires a target image from a target domain; a semantic segmentation module, which inputs the source image and the target image into a semantic segmentation model respectively to obtain a prediction result; the semantic segmentation model is trained using a cross entropy loss function and a loss function; wherein the cross entropy loss function is obtained based on the prediction result corresponding to the source image and its corresponding true label; the loss function is obtained based on inputting the true label, the prediction result, and the intermediate output feature map and segmentation result of the semantic segmentation model into a contrast learning model.
[0021] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the domain adaptive image semantic segmentation method based on semantic prototype contrast learning as described above are implemented.
[0022] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the domain adaptive image semantic segmentation methods based on semantic prototype contrast learning as described above.
[0023] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of any of the above-mentioned domain adaptive image semantic segmentation methods based on semantic prototype contrast learning.
[0024] The domain adaptive image semantic segmentation method and device based on semantic prototype contrast learning provided by the present invention perform semantic segmentation on the source image and the target image using the cross entropy loss function and the semantic segmentation model trained by the loss function, so that the semantic segmentation model aggregates the same type of pixels of the input image data and separates the pixels of different types, so as to improve the feature representation of the same type of pixels and the pixels of different types in the feature space, improve the accuracy and stability of the semantic segmentation model, and improve the adaptive ability of the semantic segmentation domain migration. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0026] Figure 1It is one of the flow charts of the domain adaptive image semantic segmentation method based on semantic prototype contrast learning provided by the present invention;
[0027] Figure 2 This is the second flow chart of the domain adaptive image semantic segmentation method based on semantic prototype contrast learning provided by the present invention;
[0028] Figure 3 This is one of the training effect comparison diagrams provided by the present invention;
[0029] Figure 4 This is the second schematic diagram of the training effect comparison provided by the present invention;
[0030] Figure 5 This is the third schematic diagram of training effect comparison provided by the present invention;
[0031] Figure 6 It is a schematic diagram of the results of the global alignment, coarse-grained class alignment and contrastive learning model and the real label map provided by the present invention;
[0032] Figure 7 It is a schematic diagram of the comparison results provided by the present invention with the CAG-UDA segmentation results and t-SNE visualization;
[0033] Figure 8 It is a structural schematic diagram of a domain adaptive image semantic segmentation device based on semantic prototype contrast learning provided by the present invention;
[0034] Fig. 9 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION
[0035] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0036] Figure 1 A flowchart of a domain adaptive image semantic segmentation method based on semantic prototype contrast learning is shown, and the method includes:
[0037] S01, obtain source images and their corresponding true labels from the source domain, and obtain target images from the target domain;
[0038] S02, input the source image and the target image into the semantic segmentation model respectively to obtain the prediction result; the semantic segmentation model is trained using the cross entropy loss function and the loss function; wherein the cross entropy loss function is obtained based on the prediction result corresponding to the source image and its corresponding true label; the loss function is obtained based on inputting the true label, the prediction result and the intermediate output of the semantic segmentation model into the contrastive learning model.
[0039] It should be noted that the S0N in this specification does not represent the order of the domain adaptive image semantic segmentation method based on semantic prototype contrast learning. Figure 2-7 The present invention describes a domain adaptive image semantic segmentation method based on semantic prototype contrast learning.
[0040] S01, obtain source images and their corresponding true labels from the source domain, and obtain target images from the target domain.
[0041] In this embodiment, the acquired source image set is expressed as:
[0042]
[0043] Among them, D s represents the source image set; X si represents the i-th source image X s ; Y si Represents the true label Y corresponding to the i-th source image s .
[0044] Similarly, the target image set obtained is expressed as:
[0045]
[0046] Among them, D t represents the source image set; X tj represents the jth source image X t ; Y tj Represents the true label Y corresponding to the jth source image t .
[0047] It should be noted that the target image is an unlabeled image. s , as well as Where H and W represent the spatial dimensions of the image, and C is the number of semantic class labels.
[0048] S02, input the source image and the target image into the semantic segmentation model respectively to obtain the prediction result; the semantic segmentation model is trained using the cross entropy loss function and the loss function; wherein the cross entropy loss function is obtained based on the prediction result corresponding to the source image and its corresponding true label; the loss function is obtained based on inputting the true label, the prediction result and the intermediate output of the semantic segmentation model into the contrastive learning model.
[0049] In this embodiment, the semantic segmentation model includes: a feature extraction layer, which extracts semantic features of the input source image and the target image respectively to obtain corresponding feature maps; wherein the feature maps include source feature maps and target feature maps; a semantic segmentation layer, which performs semantic segmentation based on the source feature maps and the target feature maps respectively to obtain corresponding segmentation results; wherein the segmentation results include source segmentation results corresponding to the source feature maps and target segmentation results corresponding to the target feature maps; a result output layer, which adds corresponding labels to the source segmentation results and the target segmentation results to obtain corresponding prediction results; wherein the target prediction results include source prediction results corresponding to the source segmentation results and target prediction results corresponding to the target segmentation results.
[0050] It should be noted that the source image and the target image are respectively input into the semantic segmentation model to obtain the prediction result, including: s and the target image X t Input the feature extraction layer to obtain the source feature map F corresponding to the output of the feature extraction layer s and the target feature map F t , where F s , The source feature map F s and the target feature map F t Input the semantic segmentation layer and obtain the source segmentation result O corresponding to the output of the semantic segmentation layer s And the target segmentation result O t , where O s , The source segmentation result O s And the target segmentation result O t Input the result output layer and get the source prediction result P corresponding to the output of the result output layer s And the target prediction result P t , where P s ,
[0051] In addition, the pixel representations obtained in the source and target domains are separated using the masks corresponding to the source and target images and transferred to the contrastive learning model to cluster the pixel clusters belonging to the same semantic prototype in the feature space and separate them from other semantic prototypes to improve the compactness and separability at the pixel level. The semantic prototype is the semantic information represented by the pixel value, and the pixels with the same semantic information are classified into the same semantic prototype.
[0052] In an optional embodiment, the loss function includes a source contrast loss function, the intermediate output includes a true label, and the contrast learning model includes: a first mask layer, based on the input true label, obtains a source mask of the source image; a first center acquisition layer, based on the source mask, updates the first category center of the source prediction result of the same semantic prototype; a first function acquisition layer, based on the source mask and the first category center, obtains a source contrast loss function. Specifically:
[0053] First, based on the true label Y of the input s , get the source image X s The source mask M s . The source mask is expressed as:
[0054]
[0055] It should be noted that h' and w' represent the spatial dimensions of the corresponding images; M si represents the i-th source mask; Y s represents the true label.
[0056] Secondly, based on the source mask M s , update the first category center of the source prediction result of the same semantic prototype It should be noted that when updating the first category center μ of the source prediction results of the same semantic prototype c Before that, you need to initialize the contrastive learning model and calculate the category center Category Center It is expressed as:
[0057]
[0058] in, represents the category center of the cth semantic prototype corresponding to the source image; s represents the source domain, n s Represents the number of pixels corresponding to the source domain; Ⅱ represents the indicator function, if If the condition is met, it returns 1, otherwise it returns 0; Indicates that it contains the source feature map F si is the pixel set of all pixel features of the cth semantic prototype in; |·| is the number of pixels in the pixel set; F si represents the ith source feature map; h' and w' respectively represent the spatial dimensions of the corresponding image. It should be noted that the contrastive learning model is only calculated once when the model is initialized, and the contrastive learning model is continuously updated as the model learning proceeds.
[0059] It should be noted that each time the contrastive learning model is updated, since the pixel features in the encoding of each source image are different, the category center needs to be involved in the update process to represent the latest semantic knowledge. Use the feature map F s The average update of the pixel features in the source encoding of the cth class of the mapping is expressed as:
[0060]
[0061] Among them, α∈[0.1] is the coefficient of updating the contrastive learning model, and α is set to 0.1; II represents the indicator function. If If the condition holds, it returns 1, otherwise it returns 0. It should be noted that α = 1.0 means that the contrastive learning model is fixed during the training process during testing. Using the updated contrastive learning model, the cross-domain pixel representations can be dynamically guided to cluster around the corresponding homogeneous category centers and reveal the intrinsic distinction between the source and target domains.
[0062] Finally, based on the source mask and the first category center, the source contrast loss function is obtained.
[0063] The source contrast loss function is expressed as:
[0064]
[0065] in, represents the source contrast loss function; Ⅱ represents the indicator function, if If the condition is met, it returns 1, otherwise it returns 0; h', w' respectively represent the spatial dimensions of the corresponding image; c represents the number of source masks; M si represents the i-th source mask; F si represents the i-th source feature map; represents the first category center of the cth semantic prototype corresponding to the source image, and denote the first category centers of positive and negative semantic prototypes respectively; l(·) denotes the loss function of the source feature map.
[0066] The loss function of the source feature map is expressed as:
[0067]
[0068] It should be noted that by calculating the source feature map F s The loss function reflects the data incompatibility, and Represent the positive and negative first category centers respectively; T represents temperature; the loss function of the source feature map is to use the softmax formula of temperature T to aggregate the positive first category centers and push the negative first category centers away, that is, to aggregate the same type of semantic prototypes and push the different types of semantic prototypes away, so as to learn multiple useful semantic knowledge through multiple unlabeled data.
[0069] In an optional embodiment, the loss function includes a target contrast loss function, the intermediate output includes a target segmentation result, and the contrast learning model includes: a first confidence layer, which generates a first confidence map based on the input target segmentation result; a second mask layer, which obtains a target mask according to the confidence map; a second center acquisition layer, which updates the second category center of the target prediction result of the same semantic prototype based on the target mask; and a second function acquisition layer, which obtains the target contrast loss function based on the target mask and the second category center.
[0070] First, based on the input target segmentation result O t Generate the first confidence map It should be noted that the first confidence map Each confidence value in is the maximum item of the softmax output of each classification channel output by the semantic segmentation layer, so that the generated pseudo-label is associated with the confidence value, thereby avoiding training errors when the semantic segmentation model generates target prediction results. In addition, when selecting the confidence value to form the first confidence map, if the maximum item of the softmax output of a classification channel is higher than 0.9, its corresponding confidence is set to 0.9; otherwise, the maximum item of the softmax output of the classification channel is directly used as the corresponding confidence value.
[0071] Secondly, according to the confidence map Get the target mask M t , target mask M t It is expressed as:
[0072]
[0073] It should be noted that Ⅱ represents the indicator function. If If the condition is met, it returns 1, otherwise it returns 0; t Represents the target segmentation result; h' and w' respectively represent the spatial dimensions of the corresponding image.
[0074] Then, based on the target mask M t , update the second category center of the target prediction result of the same semantic prototype It should be noted that when updating the second category center of the target prediction result of the same semantic prototype Before that, you need to initialize the contrastive learning model and calculate the second category center Category II Center It is expressed as:
[0075]
[0076] in, represents the category center of the cth semantic prototype corresponding to the source image; t represents the target domain, n t Represents the number of pixels corresponding to the target domain; Ⅱ represents the indicator function. If If the condition is met, it returns 1, otherwise it returns 0; Indicates that it contains the target feature map F tj The pixel set of all pixel features of the cth semantic class in ; |·| is the number of pixels in the set; F tj Denotes the jth target feature map. It should be noted that the contrastive learning model is only calculated once when the model is initialized, and the contrastive learning model is continuously updated as the model learning proceeds.
[0077] It should be noted that each time the contrastive learning model is updated, since the pixel features in the encoding of each target image are different, the second category center needs to participate in the update process to represent the latest semantic knowledge. Use the feature map F t The average value of the pixel features in the encoding of the cth class of the mapping is updated as:
[0078]
[0079] Where α∈[0.1] is the coefficient for updating the contrastive learning model, and α is set to 0.1. It should be noted that α=1.0 means that the contrastive learning model is fixed during the training process during testing. Using the updated contrastive learning model, the cross-domain pixel representations can be dynamically guided to cluster around the corresponding centers of the same semantic prototype category and reveal the intrinsic distinction between the source and target domains.
[0080] Finally, based on the target mask M t and second category centers The target contrast loss function is obtained. The target contrast loss function is expressed as:
[0081]
[0082] in, represents the target contrast loss function; Ⅱ represents the indicator function, if If the condition is met, it returns 1, otherwise it returns 0; h', w' respectively represent the spatial dimensions of the corresponding image; c represents the number of target masks; M tj represents the jth target mask; F tj represents the jth target feature map; represents the second category center of the cth semantic prototype corresponding to the target image, and denote the second category centers of positive and negative semantic prototypes respectively; l(·) denotes the loss function of the target feature map.
[0083] The loss function of the target feature map is expressed as:
[0084]
[0085] It should be noted that by calculating the target feature map F t The loss function reflects the data incompatibility, and They represent the second category centers of positive and negative semantic prototypes respectively; T represents temperature; the loss function of the target feature map is to use the softmax formula of temperature T to aggregate the positive second category centers and push the negative second category centers apart, that is, to aggregate the same type of semantic prototypes and push the different types of semantic prototypes apart, so as to learn multiple useful semantic knowledge through multiple unlabeled data.
[0086] In an optional embodiment, the cross entropy loss function L seg It is expressed as:
[0087]
[0088] Among them, Ys i represents the true label corresponding to the i-th source image; h and w represent the spatial dimensions of the corresponding image; c represents the number of true labels; Ps i Represents the source prediction result corresponding to the i-th source image.
[0089] In an optional embodiment, the loss function includes a source contrast loss function and a target loss function, and the contrast learning model includes: a confidence layer, which generates a first confidence map based on the input target segmentation result; a mask layer, which obtains a source mask of the source image and a target mask of the corresponding target image based on the input true label and the confidence map; a category center acquisition layer, which updates the first category center and the second category center of the source prediction result of the same semantic prototype based on the source mask and the target mask, respectively; a function acquisition layer, which obtains a source contrast loss function based on the source mask and the first category center, and obtains a target contrast loss function based on the target mask and the second category center. It should be noted that in order to facilitate the training of the semantic segmentation model, the cross entropy loss function, the source contrast loss function, the target loss function and the weight λ are combined to narrow the domain gap between the source image data and the target image data and perform the segmentation task, which is expressed as:
[0090]
[0091] By combining the cross entropy loss function, the source contrast loss function and the target loss function, the semantic segmentation model is adjusted to aggregate similar pixel clusters in the feature space and separate different pixel clusters in the feature space.
[0092] In an optional embodiment, the loss function includes a self-supervised loss function, the intermediate output includes a target prediction result, and the comparative learning model includes: a second confidence layer, which generates a second confidence map based on the input target prediction result; a pseudo-label prediction layer, which generates a pseudo-label based on the second confidence layer; and a third function acquisition layer, which uses the pseudo-label and the target prediction result to obtain a self-supervised loss function.
[0093] First, based on the input target prediction result P t Generate the second confidence map It should be noted that the second confidence map The generation method of can refer to the first confidence map, which will not be further described here.
[0094] Secondly, based on the second confidence level, a pseudo label is generated. It is expressed as:
[0095]
[0096] in, represents the pseudo label; Ⅱ represents the indicator function, if If the condition is met, it returns 1, otherwise it returns 0; t Represents the target prediction result.
[0097] Finally, the third function acquisition layer uses the pseudo-label and target prediction results to obtain the self-supervised loss function. The self-supervised loss function L ssl It is expressed as:
[0098]
[0099] Among them, L ssl represents the self-supervised loss function; Ⅱ represents the indicator function, if If the condition is met, it returns 1, otherwise it returns 0; h' and w' represent the spatial dimensions of the corresponding image; c represents the number of pseudo labels; represents the jth pseudo label; P j Represents the j-th target prediction result.
[0100] It should be noted that after adjusting the semantic segmentation model by combining the cross entropy loss function, the source contrast loss function and the target loss function, the self-supervised loss function is used to fine-tune the semantic segmentation model so that the semantic segmentation model is more adapted to the target image data for the target domain.
[0101] In an alternative embodiment, reference Figure 2 , the method further comprises:
[0102] S11, obtain source images and their corresponding true labels from the source domain, and obtain target images from the target domain;
[0103] S12, the source image X s and the target image X t Input the semantic segmentation model respectively to obtain the prediction results; the semantic segmentation model includes an encoder E and a decoder D, which transforms the source image X s and the target image X t Input encoder E respectively to get the corresponding feature map F s 、F t ; The feature map F s 、F t Input to the decoder and get the segmentation prediction result P output by the decoder s , P t ;
[0104] S13, obtain the source mask Ms of the source image based on the true label; based on the source mask Ms, calculate the center of the cth class on the entire source domain; based on the different source masks input each time, iteratively update the corresponding first category center to dynamically guide the cross-domain pixels to gather around the corresponding first category center; according to the mask Ms and the feature map Fs, calculate the source contrast loss function on the source domain;
[0105] Based on the encoder's segmentation prediction Ot, generate a confidence map, obtain the mask Mt of the target image, and calculate the target contrast loss function of all target pixels;
[0106] Calculate the cross entropy loss function based on the source image and the corresponding true label;
[0107] Use source contrast loss, target contrast loss, and cross entropy loss to train a semantic segmentation model.
[0108] S14, based on the segmentation prediction result Pt, select a confidence threshold to generate a pseudo label and fine-tune the semantic segmentation model using a self-supervised loss function.
[0109] It should be noted that the pixel representations obtained in the source and target domains are separated based on the masks corresponding to the source and target images and transferred to the contrastive learning model to cluster the pixel clusters belonging to the same semantic prototype in the feature space and separate them from other semantic prototypes to improve the compactness and separability at the pixel level.
[0110] In an optional embodiment, four popular benchmark datasets are used to evaluate the domain adaptive image semantic segmentation method based on semantic prototype contrastive learning in a "Sim-to-Real" scenario, that is, transferring from synthetic images (GTA5, Synscapes, and SYNTHIA) to real images (Cityscapes). Cityscapes contains 5,000 city scene images with a resolution of 2048×1024. It is divided into training, validation, and test sets, containing 2,975, 500, and 1,525 images, respectively. Similar to AdaptSegNet and CRST, the semantic segmentation model is evaluated on the validation set. GTA5 contains 24,966 images with a resolution of 1,914×1,052. Synscapes contains 25,000 images with a resolution of 1,440×720. SYNTHIA provides 9,400 images with a resolution of 1,280×760. PSACAL VOC Intersection-over-Union (IoU) is used as the evaluation metric. For the GTA5 Cityscapes and SYNTHIA Cityscapes tasks, we obtain results for the common 19 classes and the tail classes. For the SYNTHIA Cityscapes task, we obtain results for 16 and 13 semantically prototypical classes.
[0111] refer to Figure 3-Figure 5 ,All models use the DeepLab-v2 framework, Figure 3 A quantitative comparison of the Cityscapes validation set with a model trained on GTA5 is shown. mIoU tail represents the mean IoU of the tail class in blue. Figure 4 A quantitative comparison of the Cityscapes validation set with a model trained on Synscapes is shown. Figure 5 : Quantitative comparison of the Cityscapes validation set with the model trained on SYNTHIA, mIoU* represents the mean IoU of the 13 classes, excluding classes with *.
[0112] Specifically, for the GTA5 Cityscapes task, the method of this embodiment surpasses the method based on global alignment, and for VGG-16 and ResNet-101, the adaptive network increases by +9.8% and +9.7%, respectively; for synscape, the method of this embodiment obtains a +4.4% improvement, based on the ResNet-101 urban landscape task; for the SYNTHIA urban landscape task, the method of this embodiment is more favorable to others.
[0113] In addition, the overall increase of the method of this embodiment is more than 1.0% compared with other class alignment methods (CLAN, SSF-DAN, CAG-UDA, SIM, FADA, etc.). In addition, the performance of the method of this embodiment is also comparable to or even better than the strong baselines (i.e., self-supervised BDL, PyCDA, IntraDA, AdvEnt, CRST, and pixel-associated PLCA methods). The above results show that the way of learning per-pixel distinguishable representation is beneficial to adaptive ability.
[0114] refer to Figure 6 , the qualitative results of GTA5's urban landscape task verify the significant visual improvement brought by the method of this embodiment.
[0115] refer to Figure 7 ,From the perspective of tSNE visualization, t-SNE was used to measure the high-dimensional pixel representation of CAG-UDA and this method in 2D space, and the results of the two were compared, further proving that the method of this embodiment is more discriminative and consistent at the fine-grained level.
[0116] To summarize, the embodiments of the present invention utilize the cross entropy loss function and the semantic segmentation model obtained by training the loss function to perform semantic segmentation on the source image and the target image, so that the semantic segmentation model can aggregate similar pixels and separate different types of pixels of the input image data, so as to improve the feature representation of similar semantic prototype pixels and different types of semantic prototype pixels in the feature space, improve the accuracy and stability of the semantic segmentation model, and improve the adaptability of the semantic segmentation domain migration.
[0117] The domain adaptive image semantic segmentation device based on semantic prototype contrast learning provided by the present invention is described below. The domain adaptive image semantic segmentation device based on semantic prototype contrast learning described below and the domain adaptive image semantic segmentation method based on semantic prototype contrast learning described above can be referenced to each other.
[0118] refer to Figure 8 , Figure 8 A schematic diagram of the structure of a domain adaptive image semantic segmentation device based on semantic prototype contrast learning is shown, the device comprising:
[0119] A data acquisition module 81 acquires source images and their corresponding true labels from a source domain, and acquires target images from a target domain;
[0120] The semantic segmentation module 82 inputs the source image and the target image into the semantic segmentation model respectively to obtain the prediction result; the semantic segmentation model is trained using the cross entropy loss function and the loss function; wherein the cross entropy loss function is obtained based on the prediction result corresponding to the source image and its corresponding true label; the loss function is obtained based on inputting the true label, the prediction result and the intermediate output of the semantic segmentation model into the contrastive learning model.
[0121] In this embodiment, the semantic segmentation module 82 includes: a semantic segmentation submodule, which obtains a prediction result based on the input source image and the target image; a contrastive learning submodule, a cross entropy loss function, and a loss function to train the semantic segmentation submodule; wherein the cross entropy loss function is obtained based on the prediction result corresponding to the source image and its corresponding true label; the loss function is obtained based on inputting the true label, the prediction result, and the intermediate output of the semantic segmentation model into the contrastive learning model.
[0122] Specifically, the semantic segmentation submodule includes: a feature extraction unit, which extracts the semantic features of the input source image and the target image respectively to obtain the corresponding feature map; wherein the feature map includes the source feature map and the target feature map; a semantic segmentation unit, which performs semantic segmentation based on the source feature map and the target feature map respectively to obtain the corresponding segmentation result; wherein the segmentation result includes the source segmentation result corresponding to the source feature map and the target segmentation result corresponding to the target feature map; a result output unit, which adds corresponding labels to the source segmentation result and the target segmentation result to obtain the corresponding prediction result; wherein the target prediction result includes the source prediction result corresponding to the source segmentation result and the target prediction result corresponding to the target segmentation result. The pixel representations obtained in the source domain and the target domain are separated by using the masks corresponding to the source image and the target image, and are transmitted to the contrastive learning model to cluster the pixel clusters belonging to the same semantic prototype in the feature space and separate them from other semantic prototypes to improve the compactness and separability at the pixel level. The semantic prototype is the semantic information represented by the pixel value, and the pixels with the same semantic information are classified as the same semantic prototype.
[0123] In an optional embodiment, the loss function includes a source contrast loss function, the intermediate output includes a true label, and the contrast learning submodule includes: a first mask unit, which obtains a source mask of a source image based on an input true label; a first category center acquisition unit, which updates a first category center of a source prediction result of the same semantic prototype based on the source mask; and a first function acquisition unit, which obtains a source contrast loss function based on the source mask and the first category center.
[0124] In an optional embodiment, the loss function includes a target contrast loss function, the intermediate output includes a target segmentation result, and the contrast learning submodule includes: a first confidence unit, which generates a first confidence map based on the input target segmentation result; a second mask unit, which obtains a target mask according to the confidence map; a second category center acquisition unit, which updates the second category center of the target prediction result of the same semantic prototype based on the target mask; and a second function acquisition unit, which obtains a target contrast loss function based on the target mask and the second category center.
[0125] In an optional embodiment, the loss function includes a source contrast loss function and a target loss function, and the contrast learning submodule includes: a first confidence unit, which generates a first confidence map based on the input target segmentation result; a mask unit, which obtains a source mask of the source image and a target mask of the corresponding target image based on the input true label and the confidence map; a category center acquisition unit, which updates the first category center and the second category center of the source prediction result of the same semantic prototype based on the source mask and the target mask, respectively; a function acquisition unit, which obtains a source contrast loss function based on the source mask and the first category center, and obtains a target contrast loss function based on the target mask and the second category center. It should be noted that in order to facilitate the training of the semantic segmentation model, the cross entropy loss function, the source contrast loss function, the target loss function and the weight λ are combined to narrow the domain gap between the source image data and the target image data and perform the segmentation task.
[0126] In an optional embodiment, the loss function includes a self-supervised loss function, the intermediate output includes a target prediction result, and the contrastive learning submodule includes: a second confidence unit, which generates a second confidence map based on the input target prediction result; a pseudo-label prediction unit, which generates a pseudo-label based on the second confidence layer; and a third function acquisition unit, which uses the pseudo-label and the target prediction result to obtain a self-supervised loss function.
[0127] Fig. 9 An example of a physical structure diagram of an electronic device is shown in FIG. Fig. 9As shown, the electronic device may include: a processor 91, a communication interface 92, a memory 93 and a communication bus 94, wherein the processor 91, the communication interface 92 and the memory 930 communicate with each other through the communication bus 94. The processor 91 may call the logic instructions in the memory 93 to execute a domain adaptive image semantic segmentation method based on semantic prototype contrast learning, the method comprising: obtaining a source image and its corresponding true label from a source domain, and obtaining a target image from a target domain; inputting the source image and the target image into a semantic segmentation model respectively to obtain a prediction result; the semantic segmentation model is obtained by training using a cross entropy loss function and a loss function; wherein the cross entropy loss function is obtained based on the prediction result corresponding to the source image and its corresponding true label; the loss function is obtained based on inputting the true label, the prediction result and the intermediate output of the semantic segmentation model into the contrast learning model.
[0128] In addition, the logic instructions in the above-mentioned memory 93 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.
[0129] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the domain adaptive image semantic segmentation method based on semantic prototype contrast learning provided by the above methods, and the method includes: obtaining the prediction result corresponding to the source image and its corresponding true label from the source domain, and obtaining the target image from the target domain; inputting the source image and the target image into the semantic segmentation model respectively to obtain the prediction result; the semantic segmentation model is trained using the cross entropy loss function and the loss function; wherein the cross entropy loss function is obtained based on the source image and its corresponding true label; the loss function is obtained based on inputting the true label, the prediction result and the intermediate output of the semantic segmentation model into the contrast learning model.
[0130] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the domain adaptive image semantic segmentation method based on semantic prototype contrast learning provided by the above methods, the method comprising: obtaining a prediction result corresponding to a source image and its corresponding true label from a source domain, and obtaining a target image from a target domain; inputting the source image and the target image into a semantic segmentation model respectively to obtain a prediction result; the semantic segmentation model is trained using a cross entropy loss function and a loss function; wherein the cross entropy loss function is obtained based on the source image and its corresponding true label; the loss function is obtained based on inputting the true label, the prediction result and the intermediate output of the semantic segmentation model into the contrast learning model.
[0131] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0132] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0133] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A domain adaptive image semantic segmentation method based on semantic prototype contrast learning, characterized in that: include: Obtain source images and their corresponding true labels from the source domain, and obtain target images from the target domain; Inputting the source image and the target image into a semantic segmentation model respectively to obtain a prediction result; The semantic segmentation model is trained using a cross entropy loss function and a loss function; The cross entropy loss function is obtained based on the prediction result corresponding to the source image and its corresponding true label; The loss function is obtained by inputting the true label, the prediction result and the intermediate output of the semantic segmentation model into the contrastive learning model; The loss function includes a source contrast loss function, the intermediate output includes a true label, and the contrast learning model includes: The first mask layer obtains the source mask of the source image based on the input true label; A first center acquisition layer, which updates the first category center of the source prediction result of the same semantic prototype based on the source mask; A first function acquisition layer obtains a source contrast loss function based on the source mask and the first category center; Before the first category center of the source prediction result of the same semantic prototype is updated, it includes: Initialize the contrastive learning model and calculate the category center; The calculation category center is expressed as: in, represents the category center of the cth semantic prototype corresponding to the source image; s represents the source domain, n s Represents the number of pixels corresponding to the source domain; Ⅱ represents the indicator function, if If the condition is met, it returns 1, otherwise it returns 0; Indicates that it contains the source feature map F si The pixel set of all pixel features of the cth semantic prototype in ; |·| is the number of pixels in the pixel set; F si represents the i-th source feature map; h' and w' respectively represent the spatial dimensions of the corresponding image; The category center of the cth semantic prototype Use the feature map F s The average update of the pixel features in the source encoding of the cth class of the mapping is expressed as: Among them, α∈[0.1] is the coefficient of updating the contrastive learning model, and α is set to 0.1; II represents the indicator function. If If the condition is met, it returns 1, otherwise it returns 0; The loss function includes a source contrast loss function, which is expressed as: in, represents the source contrast loss function; Ⅱ represents the indicator function, if If the condition is met, it returns 1, otherwise it returns 0; h', w' respectively represent the spatial dimensions of the corresponding image; c represents the number of source masks; M si represents the i-th source mask; F si represents the i-th source feature map; represents the first category center of the cth semantic prototype corresponding to the source image, and denote the first category centers of positive and negative semantic prototypes respectively; l(·) denotes the loss function of the source feature map.
2. The domain adaptive image semantic segmentation method based on semantic prototype contrast learning according to claim 1, characterized in that: The semantic segmentation model includes: A feature extraction layer, which extracts semantic features of the input source image and the target image respectively to obtain corresponding feature maps; wherein the feature maps include a source feature map and a target feature map; A semantic segmentation layer, performing semantic segmentation based on the source feature map and the target feature map, respectively, to obtain corresponding segmentation results; wherein the segmentation results include a source segmentation result corresponding to the source feature map and a target segmentation result corresponding to the target feature map; The result output layer adds corresponding labels to the source segmentation result and the target segmentation result to obtain corresponding prediction results; wherein the corresponding prediction results include source prediction results corresponding to the source segmentation results and target prediction results corresponding to the target segmentation results.
3. The domain adaptive image semantic segmentation method based on semantic prototype contrast learning according to claim 2, characterized in that: The loss function includes a target contrast loss function, the intermediate output includes a target segmentation result, and the contrast learning model includes: A first confidence layer, generating a first confidence map based on an input target segmentation result; A second mask layer, obtaining a target mask according to the confidence map; A second center acquisition layer, based on the target mask, updates the second category center of the target prediction result of the same semantic prototype; The second function acquisition layer obtains a target contrast loss function based on the target mask and the second category center.
4. The domain adaptive image semantic segmentation method based on semantic prototype contrast learning according to claim 2, characterized in that: The loss function includes a self-supervised loss function, the intermediate output includes a target prediction result, and the contrastive learning model includes: A second confidence layer generates a second confidence map based on the input target prediction result; A pseudo label prediction layer generates a pseudo label based on the second confidence level; The third function acquisition layer uses the pseudo label and the target prediction result to obtain a self-supervised loss function.
5. The domain adaptive image semantic segmentation method based on semantic prototype contrast learning according to claim 1, characterized in that: The loss function includes a target contrast loss function and a self-supervision loss function; The target contrast loss function is expressed as: in, represents the target contrast loss function; Ⅱ represents the indicator function, if If the condition is met, it returns 1, otherwise it returns 0; h', w' respectively represent the spatial dimensions of the corresponding image; c represents the number of target masks; M tj represents the jth target mask; F tj Represents the jth target feature map; represents the second category center of the cth semantic prototype corresponding to the target image, and denote the second category center of positive and negative semantic prototypes respectively; l(·) denotes the loss function of the target feature map; The self-supervised loss function is expressed as: Among them, L ssl represents the self-supervised loss function; Ⅱ represents the indicator function, if If the condition is met, it returns 1, otherwise it returns 0; h' and w' represent the spatial dimensions of the corresponding image; c represents the number of pseudo labels; represents the jth pseudo label; P j Represents the j-th target prediction result.
6. A domain adaptive image semantic segmentation device based on semantic prototype contrast learning, characterized in that: include: The data acquisition module acquires source images and their corresponding true labels from the source domain, and acquires target images from the target domain; A semantic segmentation module, inputting the source image and the target image into a semantic segmentation model respectively to obtain a prediction result; The semantic segmentation model is trained using a cross entropy loss function and a loss function; The cross entropy loss function is obtained based on the prediction result corresponding to the source image and its corresponding true label; The loss function is obtained by inputting the true label, the prediction result and the intermediate output of the semantic segmentation model into the contrastive learning model; The loss function includes a source contrast loss function, the intermediate output includes a true label, and the contrast learning model includes: The first mask layer obtains the source mask of the source image based on the input true label; A first center acquisition layer, which updates the first category center of the source prediction result of the same semantic prototype based on the source mask; A first function acquisition layer obtains a source contrast loss function based on the source mask and the first category center; The first central acquisition layer is further used for: Before the first category center of the source prediction result of the same semantic prototype is updated, the contrastive learning model is initialized and the category center is calculated; The calculation category center is expressed as: in, represents the category center of the cth semantic prototype corresponding to the source image; s represents the source domain, n s Represents the number of pixels corresponding to the source domain; Ⅱ represents the indicator function, if If the condition is met, it returns 1, otherwise it returns 0; Indicates that it contains the source feature map F si The pixel set of all pixel features of the cth semantic prototype in ; |·| is the number of pixels in the pixel set; F si represents the i-th source feature map; h' and w' respectively represent the spatial dimensions of the corresponding image; The category center of the cth semantic prototype Use the feature map F s The average update of the pixel features in the source encoding of the cth class of the mapping is expressed as: Among them, α∈[0.1] is the coefficient of updating the contrastive learning model, and α is set to 0.1; II represents the indicator function. If If the condition is met, it returns 1, otherwise it returns 0; The loss function includes a source contrast loss function, which is expressed as: in, represents the source contrast loss function; Ⅱ represents the indicator function, if If the condition is met, it returns 1, otherwise it returns 0; h', w' respectively represent the spatial dimensions of the corresponding image; c represents the number of source masks; M si represents the i-th source mask; F si represents the i-th source feature map; represents the first category center of the cth semantic prototype corresponding to the source image, and denote the first category centers of positive and negative semantic prototypes respectively; l(·) denotes the loss function of the source feature map.
7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the domain adaptive image semantic segmentation method based on semantic prototype contrast learning as described in any one of claims 1 to 5 are implemented.
8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the domain adaptive image semantic segmentation method based on semantic prototype contrast learning are implemented as described in any one of claims 1 to 5.
9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the domain adaptive image semantic segmentation method based on semantic prototype contrast learning are implemented as described in any one of claims 1 to 5.