Weakly supervised semantic segmentation method and system based on component-aware learning segmentation network

Abstract

The present invention discloses a weakly supervised semantic segmentation method and system based on a component-aware learning segmentation network. The method comprises the following steps: step 1: constructing a component-aware learning segmentation network; step 2: obtaining a SAM mask set of a training set image; step 3: obtaining a mask pseudo-label set of the training set image and a relationship matrix between target components and the whole in the mask; step 4: obtaining an initial component-aware learning segmentation network model; step 5: obtaining and optimizing a CAM set of target masks in the training set image to obtain a final pseudo-label set, and using a segmentation decoder to obtain a semantic segmentation result in the image to be segmented; removing redundant masks through a component-whole refinement module in the network and capturing the relationship between the target components and the whole in the mask; and encouraging the network to learn the relationship between each component and the whole of the target in the pseudo-label set through a component-aware learning module in the network, thereby achieving accurate and complete segmentation of the target under weak supervision.

Description

Technical Field

[0001] The present invention belongs to the field of semantic segmentation, and in particular relates to a weakly supervised semantic segmentation method and system based on a component-aware learning segmentation network. Background Art

[0002] Semantic segmentation is an important task in computer vision and has made remarkable progress in the past few decades. Typically, semantic segmentation methods adopt a fully supervised learning paradigm, using large-scale datasets containing pixel-level annotations to train segmentation models. However, obtaining accurate annotations is a time-consuming and costly process. In addition, labels are often incorrect due to misjudgment by the annotator. To address these challenges, weakly supervised semantic segmentation (WSSS) has become an important research hotspot. It only uses weakly and low-cost annotation forms such as image-level labels, points, lines, and bounding boxes as the main supervision information for training segmentation models.

[0003] In the pursuit of accurate weakly supervised semantic segmentation, previous studies typically generate Class Activation Maps (CAMs) (reference: B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba, "Learning deep features for discriminative localization," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2016, pp. 2921–2929.) or their variants through classification supervision and use them as pseudo-labels. However, CAM-based methods primarily rely on classification losses and often only activate the most discriminative regions of the object. As a result, the generated pseudo-labels are often incomplete, covering only a portion of the object.

[0004] To achieve accurate semantic segmentation, Wei et al. (Yunchao Wei, Jiashi Feng, Xiaodan Liang, Ming-Ming Cheng, Yao Zhao, and Shuicheng Yan, “Object regionmining with adversarial erasing: A simple classification to semantic segmentation approach,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2017, pp. 1568–1576) used an adversarial erasing strategy to further mine more target regions based on the original CAM. Wang et al. (Yude Wang, Jie Zhang, Meina Kan, Shiguang Shan, and Xilin Chen, “Self-supervised equivariant attention mechanism for weakly supervised semantic segmentation,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2020, pp. 12275–12284) introduced an auxiliary task with scale-invariance regularization to regulate the training objective. Yao et al. (Reference: YazhouYao, Tao Chen, Guo-Sen Xie, Chuanyi Zhang, Fumin Shen, Qi Wu, ZhenminTang, and Jian Zhang, “Self-supervised equivariant attention mechanism for weakly supervised semantic segmentation,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2021, pp. 2623–2632.) adopted additional saliency maps as supervisory information to suppress background regions and mine non-salient objects.Zhou et al. (Reference: Tianfei Zhou, Meijie Zhang, Fang Zhao, and Jianwu Li, “Regional semantic contrast and aggregation for weakly supervised semantic segmentation,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2022, pp. 4299–4309.) promote the complete activation of the target region by comparing pixel and prototype representations.

[0005] However, these methods based on Convolutional Neural Networks (CNNs) are mainly used to identify the main feature areas, resulting in pseudo-labels lacking detailed boundary information. Compared with CNNs, the Vision Transformer (ViT) (Reference: Alexey, Dosovitskiy. “An image is worth 16x16 words: Transformers for image recognition at scale,” arXiv preprint arXiv:2010.11929, 2020.) has achieved significant success in WSSS because its self-attention mechanism can capture global information. These ViT-based methods enhance long-range feature dependencies, thereby better learning semantic or positional affinity and differences, making CAM boundaries more accurate. However, pseudo-labels from ViT still have the problem of inaccurate boundaries.

[0006] Recently, the SegmentAnything Model (SAM) (Reference: A.Kirillov, E.Mintun, N.Ravi, H.Mao, C.Rolland, L.Gustafson, T.Xiao, S.Whitehead, A.C.Berg, W.-Y.Lo et al., “Segment anything,” arXiv preprint arXiv:2304.02643, 2023.) has performed well in unsupervised semantic segmentation. Therefore, the present invention aims to generate more accurate pseudo labels through SAM and reduce the dependence on CAM. However, SAM still has the following problems in mask prediction: (1) SAM methods usually generate a large number of low-quality masks that are difficult to distinguish between objects and background without supervision. This lack of discriminability leads to inaccurate segmentation results and difficulty in separating overlapping objects. (2) SAM often segments the entire object into multiple segments, resulting in complex and inaccurate pseudo labels because it tends to classify them based on pixel similarity. However, even when trained with CAM guidance, SAM fails to generate complete and accurate masks for complex, textured objects. This limitation suggests that SAM-based methods still face some challenges in providing complete and accurate pseudo-labels when dealing with challenging segmentation tasks. Summary of the Invention

[0007] To address these issues, the present invention proposes a weakly supervised semantic segmentation method and system based on a component-aware learning segmentation network. This method uses masks predicted by the SAM to guide the generation of more accurate CAMs, thereby producing more precise pseudo-labels with clearer boundaries. This technical solution overcomes the limitations of traditional CAM-based methods and solves the difficulties of using SAM in weakly supervised semantic segmentation, thereby improving the accuracy of weakly supervised semantic segmentation.

[0008] On the one hand, a weakly supervised semantic segmentation method based on a component-aware learning segmentation network includes:

[0009] Step 1: Build a component-aware learning segmentation network;

[0010] The component perception learning segmentation network includes a component-whole refinement module and a backbone network connected in series, as well as a classification branch network and a component branch network; the classification branch network and the component branch network are connected in parallel and then connected in series to the back end of the backbone network;

[0011] The component-whole refinement module sequentially performs small target mask removal, high similarity mask removal, and relationship matrix recording processing between the target component and the whole in the mask;

[0012] The component branch network includes a component perception learning module, a Transformer Encoder module, a fully connected layer, and a classifier connected in sequence; the component perception learning module sequentially performs embedding feature extraction, embedding feature scale reshaping, mask feature fusion, and image feature output processing;

[0013] Step 2: Input the training set image into the SAM segmentation model to obtain the SAM mask set of the training set image;

[0014] Step 3: Use the SAM mask set as input to the part-whole refinement module of the part-aware learning segmentation network to obtain the mask pseudo-label set of the training set image and the relationship matrix between the target part and the whole in the mask;

[0015] Step 4: Take the known training set images, image-level labels, the mask pseudo-label set of the training set images, and the relationship matrix between the target parts and the whole in the mask as the input of the component-aware learning segmentation network, train the component-aware learning segmentation network, and obtain the initial component-aware learning segmentation network model;

[0016] Step 5: Use the initial component-aware learning segmentation network model to obtain the CAM set of the target mask in the training set image, and obtain the final pseudo-label set through threshold segmentation; use the final pseudo-label set as supervision information, and simultaneously train the initial component-aware learning segmentation network model and the segmentation decoder connected in series to the back end of the component branch network to obtain a complete component-aware learning segmentation network model. Finally, input the image to be segmented into the complete component-aware learning segmentation network model to obtain the semantic segmentation result in the image to be segmented.

[0017] In weakly supervised semantic segmentation tasks, pseudo-labels are approximate pixel-level labels because they do not represent the true classification information for each pixel. Typically, the trained model is used to predict the target region in the image, and these predictions are used as pseudo-pixel-level labels for the target. Therefore, compared to image-level labels, pseudo-labels are more granular but slightly less accurate. They are typically generated using class activation maps (CAMs) or other feature-based strategies. This paper utilizes the unsupervised SAM segmentation model for prediction and preprocesses its output to generate pseudo-labels, which are used to guide the model in learning the regional boundaries of different targets in the image.

[0018] The process of training the segmentation decoder is as follows: input the training set image to the component-aware learning segmentation network, and use the output of the component branch network connected in series after the backbone network as input. The final pseudo-label is used as supervision information, and the conventional cross-entropy loss function is used. While training the initial component-aware learning segmentation network, the segmentation decoder is trained at the same time to obtain a complete component-aware learning segmentation network model.

[0019] Furthermore, the backbone network is composed of ViT (Vision Transformer Encoder); the classification branch network includes fully connected layers and classifiers connected in sequence.

[0020] Furthermore, the working process of the component-whole refining module is as follows:

[0021] Small object mask removal and high similarity mask removal are used to refine the SAM mask set, screening valid masks to improve data quality. The relationship matrix between the target parts and the whole in the recorded mask serves as the core of the part-whole refinement module. This matrix constructs and extracts the relationship matrix between the target parts and the whole. This matrix will play a key role in subsequent model training, helping the model better learn the association between the target parts and the whole.

[0022] Step A1: small object mask removal;

[0023] Remove small object masks that do not meet the requirements from the SAM mask set of the training set images;

[0024] Input is the SAM mask set of the training set image calculate The proportion of foreground pixels in each mask is U. If the proportion U is lower than the set threshold τ = 0.05, the mask is removed and the output mask set is obtained.

[0025]

[0026] Among them, M i represents the i-th mask in the SAM mask set of the training set image, where M represents the SAM mask set, i is the index of the mask in the set; K is the total number of masks; represents the i-th mask in the output mask set, where Represents the output mask set, i is the index of the mask in the set; is the number of masks in the output mask set;

[0027] Step A2: high similarity mask removal;

[0028] Remove masks with too high similarity from the output mask set to reduce redundancy;

[0029] Input is the output mask set The overlap between masks is calculated, and masks with an overlap exceeding the set threshold β are judged as high-similarity masks and removed to obtain the pseudo-label set of the training set images.

[0030] The pseudo-label set of the training set images consists of relatively independent valid masks, which are used for subsequent model training;

[0031]

[0032] in, and Represents masks respectively and mask The size of the intersection and union areas of the two masks; usually the number of pixels in the overlapping area of ​​the two masks and the total area covered are calculated separately;

[0033] Among them, i,j are Different mask indices in , representing the i-th mask and the j-th mask; represents the i-th mask in the pseudo-label set of the training set images, is the total number of masks in the pseudo-label set of the training set images; f(i,j) is used to determine whether to retain the mask; if there is no other mask, and If the overlap exceeds the threshold β, then f(i,j)=1, and the mask is retained Otherwise f(i,j)=1, and remove the mask The threshold β was set to 0.95;

[0034] Step A3: record the relationship matrix between the target part and the whole in the mask;

[0035] Take the pseudo-label set of training set images As input, the relationship matrix between the target parts and the whole in the mask of the pseudo label set is constructed;

[0036] By calculating the overlap between masks and setting a threshold, we can determine which masks represent the relationship between the parts and the whole. The mask in and The overlapping part occupy When the ratio is greater than or equal to the threshold λ, As the overall mask, yes Component mask; using the indicator function h(i,j), establish the relationship matrix between the target and its components

[0037]

[0038] Where λ is the threshold value of 0.9, if yes If the component mask is , then h(i,j)=1; otherwise h(i,j)=0; through h(i,j), we can get the relationship matrix between the target component and the whole in the mask

[0039]

[0040] Through these three closely coordinated components, the part-whole refinement module not only effectively optimizes the SAM mask set in the training set images, obtaining a higher-quality pseudo-label set for the training set images, but also accurately constructs the relationship between the target parts and the whole. This module provides more reliable additional information for model training, and by utilizing this information, it enhances the model's ability to associate the overall characteristics of the target with the characteristics of the parts.

[0041] Furthermore, the working process of the component perception learning module is as follows:

[0042] Step B1: Extract embedding features;

[0043] Embedding features of training set images extracted from the penultimate layer of the Transformer encoder By removing the embedded features Classify the label dimension and get the remaining embedding features

[0044] Among them, B, N, and E represent the size of the input batch, the number of image blocks into which the image is divided, and the dimension of the feature vector of each image block, respectively. represents the field of real numbers;

[0045] Step B2: Reshape the embedding feature scale;

[0046] Embed features The shape of is rearranged from B×N×E to B×C×H×W; then, the mask pseudo label set of the training set image output from the part-whole refinement module is combined with the embedded feature Point product to get the embedded feature set Then embed the feature set The shape of is rearranged from B×k×C×H×W to B×k×N×E; and Each feature in is reduced to 1 / k by a 1×1 filter layer, and the mask feature set is output.

[0047] Among them, H, W, and C represent the height, width, and feature dimension of the training set image respectively, and k represents the number of masks in the mask pseudo-label set;

[0048] Step B3: mask feature fusion;

[0049] Mask feature set Perform average pooling and normalization to obtain the weighted coefficient

[0050] Based on the relationship matrix The corresponding overall mask and component mask are used to obtain the corresponding overall mask feature and component mask features

[0051] Introducing the relationship matrix output by the component-whole refinement module When the indicator function h(i,j) in the relationship matrix is ​​1, it means the mask is the overall mask, mask Its component mask; where i, j is the pseudo label set of the training set image Different mask indices in , representing the i-th mask and the j-th mask; due to the pseudo label set of the training set image With mask feature set One-to-one correspondence, overall mask The mask feature Recorded as the overall mask feature, and the mask feature that satisfies the number h(i,j)=1 Recorded as component mask feature; the number of component masks associated with the overall mask feature is determined by the overall mask The corresponding entry h(i,j)=1 is determined;

[0052] By applying a shared 3×3 convolutional layer to each component mask feature, the component mask feature is updated, and all updated component mask features are added and fused with their corresponding overall mask features to obtain the fused overall mask feature. Another shared 3×3 convolutional layer is then used to update the fused overall mask feature to generate a new overall mask feature.

[0053] Integrate all updated overall and component mask features to obtain a new mask feature set

[0054] The updated mask feature set is updated by the weight coefficient W Then perform weighted splicing to generate weighted mask features

[0055]

[0056] Among them, cat is the feature splicing operation; W1, W2, W k represents the weighting coefficient, is the mask feature in the updated mask feature set;

[0057] Step B4: output image features;

[0058] Weighted mask features Use 1×1 convolution layer to get features Then output features It is concatenated with the removed category token to obtain the final output feature.

[0059] By utilizing the mask pseudo-label set of the training set images output by the part-whole refinement module and the relationship matrix between the target part and the whole in the mask, and through the synergistic effect of the various units of the above component perception learning module, the model can efficiently extract and fuse the mask features of the training set images. At the same time, the part-whole relationship matrix is ​​used to strengthen the model's learning of the relationship between the target part and the whole, further improving the model's performance.

[0060] Furthermore, the training set images are input into the SAM segmentation model to obtain the SAM mask set of the training set images as follows:

[0061] Taking the known training set images as input, the training images are first preprocessed, including image resizing, image grayscale and normalization, and color space conversion operations; then the preprocessed training set images are input into the pre-trained SAM segmentation model. The pre-trained SAM segmentation model automatically segments each training image, generates segmentation masks for multiple target areas, and obtains the SAM mask set of the training set images.

[0062] Furthermore, the process of training the component-aware learning segmentation network using the known training set images, image-level labels, the mask pseudo-label set of the training set images, and the relationship matrix between the target component and the whole in the mask as inputs to obtain the initial component-aware learning segmentation network model is as follows:

[0063] Taking the input training set images as input, image-level labels as supervision information, the mask pseudo-label set of the training set images and the relationship matrix between the target parts and the whole in the mask as auxiliary information, a joint loss function is used to train the component-aware learning segmentation network containing the component-aware learning module using a classifier to obtain a trained initial component-aware learning segmentation network model.

[0064] Furthermore, the joint loss function is composed of a classification loss function, an auxiliary classification loss function, and a contrast loss function, and the formula is:

[0065]

[0066] Among them, λ1, λ2, λ3 are used to balance the classification loss L cls 、 and contrast loss L reg Regularization parameter of ;

[0067] L cls Represents the cross entropy loss function:

[0068]

[0069] in, is the predicted probability of category c, y c is the true label; C represents the number of categories;

[0070] Represents the auxiliary classification loss function:

[0071]

[0072] in, Represents the predicted label obtained from the features of the component branch network, and softmax is the activation function;

[0073] L reg Represents the contrast loss function:

[0074]

[0075]

[0076] Among them, CosSim() represents the cosine similarity calculation, N + / N - Respectively represent the number of positive sample pairs (samples with component relationships) and negative sample pairs (samples other than positive samples), Represents the mask feature set elements.

[0077] Furthermore, the final pseudo-label set is used as supervision information to calculate the segmentation loss function. Based on the joint loss function plus the segmentation loss function, the initial component-aware learning segmentation network and the additional segmentation decoder connected in series to the back end of the component branch network are trained simultaneously to obtain a complete component-aware learning segmentation network model. Finally, the image to be segmented is input into the complete component-aware learning segmentation network model to infer and obtain the semantic segmentation result in the image to be segmented.

[0078] The second aspect is a weakly supervised semantic segmentation system based on a component-aware learning segmentation model, including:

[0079] Network building module: used to build component-aware learning segmentation network;

[0080] The component perception learning segmentation network includes a component-whole refinement module and a backbone network connected in series, as well as a classification branch network and a component branch network; the classification branch network and the component branch network are connected in parallel and then connected in series to the back end of the backbone network;

[0081] The component branch network includes a component perception learning module, a Transformer Encoder module, a fully connected layer, and a classifier connected in sequence; the component perception learning module includes an embedding feature extraction unit, an embedding feature rescaling unit, a mask feature fusion unit, and an image feature output unit connected in series in sequence;

[0082] SAM mask set acquisition module: input the training set image into the SAM segmentation model to obtain the SAM mask set of the training set image;

[0083] The part-whole refinement module includes a small object removal mask unit, a high similarity removal mask unit, and a relationship matrix unit that records the relationship between the target parts and the whole in the mask. It uses the SAM mask set as input to obtain the mask pseudo-label set of the training set images and the relationship matrix between the target parts and the whole in the mask.

[0084] Segmentation network training module: Using the known training set images, image-level labels, mask pseudo-label set, and the relationship matrix between the target parts and the whole in the mask, the component-aware learning segmentation network is trained to obtain the initial component-aware learning segmentation network model;

[0085] Segmentation result optimization module: Use the initial component-aware learning segmentation network model to obtain the CAM set of the target mask in the training set image, and obtain the final pseudo-label set through threshold segmentation; use the final pseudo-label set as supervision information, and simultaneously train the initial component-aware learning segmentation network and the segmentation decoder connected in series to the back end of the component branch network to obtain a complete component-aware learning segmentation network model to obtain the semantic segmentation results in the image to be segmented.

[0086] In a third aspect, a computer storage medium stores a computer program, wherein the computer program is called by a processor to implement:

[0087] The above steps are based on a weakly supervised semantic segmentation method of a component-aware learning segmentation model.

[0088] Beneficial effects

[0089] Compared with the existing method, the advantages of the present invention are:

[0090] 1. Unlike previous methods, the present invention utilizes the SAM mask set to guide the generation of more accurate CAMs, thereby producing final pseudo-labels with clearer boundaries. The present invention proposes a part-aware learning segmentation network to overcome the limitations of traditional CAM-based methods and address two challenges in SAM-based WSSS: (1) SAM generates a large number of low-quality masks and has difficulty distinguishing foreground and background under unsupervised conditions. This limitation leads to a decrease in segmentation accuracy and an inability to effectively separate overlapping targets. (2) Because SAM relies on pixel similarity for differentiation, it often segments a single target into multiple parts, thereby generating complex and inaccurate pseudo-labels. Compared with existing SAM-based WSSS methods, the present invention relies on the part-whole relationship between target masks to effectively utilize the SAM mask set to generate high-quality pseudo-labels.

[0091] 2. The proposed part-whole refinement module refines the SAM mask set and captures the part-whole relationship in the refined mask, obtaining a relationship matrix between the target part and the whole in the mask. This module can provide a high-quality mask pseudo-label set and provide more reliable information for training the segmentation network model.

[0092] 3. The part-aware learning module proposed in this invention can effectively capture the relationship between the target as a whole and its various parts, thereby avoiding the loss of target information caused by the model only focusing on the most salient areas, thereby improving the accuracy of weakly supervised semantic segmentation.

[0093] 4. The present invention preferably proposes a joint loss function, including classification loss, auxiliary classification loss and contrast loss, which can help the model better learn the relationship between the target as a whole and its components, while effectively utilizing pseudo labels to distinguish the target foreground and background. BRIEF DESCRIPTION OF THE DRAWINGS

[0094] Figure 1 Schematic diagram of a component-aware learning segmentation network according to an embodiment of the present invention;

[0095] Figure 2 Schematic diagram of the overall refining module of the components mentioned in the embodiment of the present invention;

[0096] Figure 3 Schematic diagram of the component perception learning module structure mentioned in an embodiment of the present invention;

[0097] Figure 4 This figure shows a comparison of the semantic segmentation results of the method described in the present invention on the VOC and COCO datasets, with the evaluation indicator being mIoU (mean intersection over union). PLS is the abbreviation for the weakly supervised semantic segmentation method based on the component-aware learning segmentation model.

[0098] Figure 5This is a schematic diagram of the visualization results of the method described in the present invention on the VOC dataset. DETAILED DESCRIPTION

[0099] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0100] Example 1

[0101] The embodiment of the present invention provides a weakly supervised semantic segmentation method based on component-aware learning segmentation network, such as Figure 1 The specific process is as follows:

[0102] Step 1: Build a component-aware learning segmentation network;

[0103] The component-aware learning segmentation network includes a component-whole refinement module, a backbone network, two parallel classification branch networks and a component branch network connected in series after the backbone network, such as Figure 1 shown; in Figure 1 In the example, ViT is the backbone network used to extract image features; SAM represents the SAM mask set of the training set images obtained using the SAM model, and N + represents the positive sample, including all mask features with whole-part relationship; N - Represents a negative sample, and "3×3" indicates a convolution layer with a convolution kernel of 3×3 size. All component mask features share one convolution layer, and all overall mask features share another convolution layer with a convolution kernel of 3×3 size.

[0104] The backbone network is composed of ViT (Vision Transformer Encoder);

[0105] The classification branch network includes a fully connected layer and a classifier connected in sequence; Figure 1 As shown, “FC” represents the fully connected layer; “Classifier” represents the classifier;

[0106] The component branch network includes a component perception learning module, a Transformer Encoder module, a fully connected layer and a classifier connected in sequence. Figure 1 As shown in the figure, "TEB" represents the Transformer Encoder module; "FC" represents the fully connected layer; "Classifier" represents the classifier;

[0107] Since the classifier is only related to the Class Token of the last Transformer Encoder module, the present invention adds an extra Transformer Encoder module after the component perception learning module, so that the output embedding of the component perception learning module affects the classifier.

[0108] The component-whole refinement module sequentially performs small target mask removal, high similarity mask removal, and relationship matrix recording processing between the target component and the whole in the mask, such as Figure 2 As shown;

[0109] Among them, removing small object masks and removing high similarity masks are used to refine the SAM mask set, and by screening effective masks, the data quality is improved. The relationship matrix between the target parts and the whole in the recording mask is the core of the part-whole refinement module. It constructs and extracts the relationship matrix between the target parts and the whole. This matrix will play a key role in subsequent model training, helping the model better learn the correlation between the target parts and the whole.

[0110] Step A1: remove small object mask;

[0111] The main task is to remove small target masks that do not meet the conditions from the SAM mask set of the training set image; the input is the SAM mask set of the training set image Among them, M i Represents the mask in the SAM mask set of the i-th training set image, K is the total number of masks; the specific operation is to calculate The proportion of foreground pixels in each mask is U. If the proportion is lower than the set threshold τ = 0.05, the mask is removed. After this processing, the output mask set of step A1 is obtained. in Represents the number of masks in the output mask set; the formula is as follows:

[0112]

[0113] Step A2: remove high similarity masks;

[0114] Remove masks with too high similarity from the output mask set filtered in step A1 to reduce redundancy; the input is the mask set in, represents the mask in the i-th output mask set, is the total number of masks; the specific operation is to calculate the overlap between masks (i.e., intersection-over-union ratio), and for masks whose overlap exceeds the set threshold β, it is determined to be a high-similarity mask and removed; the output mask set consists of relatively independent valid masks, which are used as the pseudo-label set of the training set image for subsequent model training, and are recorded as the pseudo-label set of the training set image The formula is as follows:

[0115]

[0116] Among them, i,j are Different mask indices in , representing the i-th mask and the j-th mask; represents the mask in the pseudo-label set of the i-th training set image, is the total number of the mask; and Represents masks respectively and mask The size of the intersection and union areas of the two masks; usually the number of pixels in the overlapping area of ​​the two masks and the total area covered are calculated separately;

[0117] f(i,j) is used to determine whether to keep the mask; if there is no other mask and If the overlap exceeds β, then f(i,j)=1, and the mask is retained Otherwise f(i,j)=1, and remove the mask The threshold β was set to 0.95;

[0118] Step A3: Record the relationship matrix between the target component and the whole in the mask;

[0119] The pseudo label set of the training set images As input, the relationship matrix between the target parts and the whole in the mask of the pseudo label set is constructed; the specific operation is to calculate the overlap between the masks and set the threshold to determine which masks represent the relationship between the parts and the whole; when the pseudo label set of the training set image The mask in and The overlapping part occupy When the ratio is greater than or equal to the threshold β, it is considered yes Component mask; using the indicator function h(i,j), the relationship matrix between the target and its components can be constructed It is defined as:

[0120]

[0121] Where λ is the threshold value of 0.9, if yes If the component mask is , then h(i,j)=1; otherwise h(i,j)=0; through h(i,j), we can get the relationship matrix between the target component and the whole in the mask

[0122]

[0123] Through the close collaboration of these three steps, the part-whole refinement module not only effectively optimizes the SAM mask set in the training set images to obtain a higher-quality pseudo-label set for the training set images, but also accurately constructs the relationship between the target parts and the whole. This module provides more reliable additional information for model training, and by utilizing this information, it enhances the model's ability to associate the overall characteristics of the target with the characteristics of the parts.

[0124] like Figure 3 As shown, the working process of the component perception learning module is as follows:

[0125] Step B1: Extract embedding features;

[0126] The input is the training set image embedding features extracted from the penultimate layer of the Transformer encoder Among them, B, N, and E represent the size of the input batch, the number of image blocks into which the image is divided, and the dimension of the feature vector of each image block, respectively. Represents the real number field; by removing the embedded features Classify the label dimension and output the remaining embedded features

[0127] Step B2: Reshape the embedding feature scale

[0128] Embed features The shape of is rearranged from B×N×E to B×C×H×W, where H, W, and C represent the height, width, and feature dimension of the training set image, respectively. Then, the mask pseudo-label set of the training set image output from the part-whole refinement module is combined with the embedded feature Point product to get the embedded feature set Wherein, k represents the number of masks (in the embodiment of the present invention, the value is 7); then the embedded feature set The shape of is rearranged from B×k×C×H×W to B×k×N×E; and Each feature in is reduced to 1 / k by a 1×1 filter layer, and the mask feature set is output.

[0129] Step B3: Mask feature fusion

[0130] right Perform average pooling and normalization to obtain the weighted coefficient Based on the relationship matrix The corresponding overall mask and component mask are used to obtain the corresponding overall mask feature and component mask features

[0131] According to the relationship matrix output from the component-whole refinement module When the indicator function h(i,j) in the relationship matrix is ​​1, it means the mask is the overall mask, mask Its component mask; where i, j is the pseudo label set of the training set image Different mask indices in , representing the i-th mask and the j-th mask; due to the pseudo label set of the training set image With mask feature set One-to-one correspondence, overall mask The mask feature Recorded as the overall mask feature, and the mask feature that satisfies the number h(i,j)=1 Recorded as component mask feature; the number of component masks associated with the overall mask feature is determined by the overall mask The corresponding entry h(i,j)=1 is determined;

[0132] By applying a shared 3×3 convolutional layer to each component mask feature, the component mask feature is updated, and all updated component mask features are added and fused with their corresponding overall mask features to obtain the fused overall mask feature. Another shared 3×3 convolutional layer is then used to update the fused overall mask feature to generate a new overall mask feature.

[0133] Integrate all updated overall and component mask features to obtain a new mask feature set

[0134] The updated mask feature set is updated by the weight coefficient W Then perform weighted splicing to generate weighted mask features

[0135]

[0136] Among them, cat is the feature splicing operation; W1, W2, W k represents the weighting coefficient, is the mask feature in the updated mask feature set;

[0137] Step B4: output image features;

[0138] Weighted mask features Use 1×1 convolution layer to get features Finally, the unit will output the features It is concatenated with the removed category token to obtain the final output feature.

[0139] By utilizing the mask pseudo-label set of the training set images output by the part-whole refinement module and the relationship matrix between the target part and the whole in the mask, and through the synergistic effect of the various steps of the above component perception learning module, the model can efficiently extract and fuse the mask features of the training set images. At the same time, the part-whole relationship matrix is ​​used to strengthen the model's learning of the relationship between the target part and the whole, further improving the model's performance.

[0140] Step 2: Input the training set image into the SAM segmentation model to obtain the SAM mask set of the training set image;

[0141] Taking the known training set images as input, the images are first preprocessed, including image resizing, image grayscale and normalization, and color space conversion operations; then the preprocessed training set images are input into the pre-trained SAM segmentation model. The model will automatically segment each training image, generate segmentation masks for multiple target areas, and obtain the SAM mask set of the training set images.

[0142] Step 3: Utilize the part-whole refinement module of the component-aware learning segmentation network and take the SAM mask set as input to obtain the mask pseudo-label set of the training set image and the relationship matrix between the target parts and the whole in the mask. This includes three steps: removing small target masks, removing high similarity masks, and recording the relationship matrix between the target parts and the whole in the mask.

[0143] In the weakly supervised semantic segmentation task, the pseudo-label is an approximate pixel-level label because it is not the true classification information of each pixel. Usually, the target area in the image is predicted by the currently trained model, and these prediction results are used as the pseudo pixel-level labels of the target. Therefore, compared with the image-level label, the pseudo-label has a higher degree of granularity but slightly lower accuracy. It is usually generated by the class activation map (CAM) or other feature-based strategies. In this paper, the unsupervised SAM segmentation model is used for prediction and its output is preprocessed to generate pseudo-labels, which are used to guide the model to learn the regional boundaries of different targets in the image.

[0144] Step 4: Using the component-aware learning segmentation network that includes the component-aware learning module, with the known training set images, image-level labels, the mask pseudo-label set of the training set images, and the relationship matrix between the target parts in the mask and the whole as input, the process of obtaining the initial component-aware learning segmentation network model is as follows:

[0145] Taking the input training set images as input, image-level labels as supervision information, the mask pseudo-label set of the training set images and the relationship matrix between the target parts and the whole in the mask as auxiliary information, a joint loss function is used to train the component-aware learning segmentation network containing the component-aware learning module using a classifier to obtain a trained initial component-aware learning segmentation network model;

[0146] The joint loss function is composed of the classification loss function, the auxiliary classification loss function and the contrast loss function, and the formula is:

[0147]

[0148] Among them, λ1, λ2, λ3 are used to balance the classification loss L cls 、 and contrast loss L reg Regularization parameter of ;

[0149] L cls Represents the cross entropy loss function:

[0150]

[0151] in, is the predicted probability of category c, y c is the true label; C represents the number of categories;

[0152] Represents the auxiliary classification loss function:

[0153]

[0154] in, Represents the predicted label obtained from the features of the component branch network, and softmax is the activation function;

[0155] L reg Represents the contrast loss function:

[0156]

[0157] Among them, CosSim() represents the cosine similarity calculation, N + / N - Represent the number of positive sample pairs and negative sample pairs respectively, Represents the mask feature set elements.

[0158] Step 5: Use the initial component-aware learning segmentation network model to obtain the CAM set of the target mask in the training set image, and obtain the final pseudo-label set through threshold segmentation; use the final pseudo-label set as supervision information to calculate the segmentation loss function, and based on the joint loss function plus the segmentation loss function, simultaneously train the initial component-aware learning segmentation network model and the segmentation decoder connected in series to the back end of the component branch network to obtain a complete component-aware learning segmentation network model, and finally input the image to be segmented into the complete component-aware learning segmentation network model to obtain the semantic segmentation result in the image to be segmented.

[0159] The process of obtaining the CAM of the target mask in the image to be segmented using the initial component-aware learning segmentation network model is as follows: the image to be segmented is input into the trained initial component-aware learning segmentation network model, the category weight of the image to be segmented is obtained by model prediction, and it is multiplied and added with the last layer feature to obtain the CAM of the target mask.

[0160] like Figure 4 As shown in FIG, the method described in the embodiment of the present invention achieves the best results in the semantic segmentation evaluation index mIoU (mean intersection over union) on the VOC and COCO datasets, and the segmentation accuracy exceeds that of the existing methods; in addition, Figure 5 As shown in the figure, the visualization results of the method described in the example of the present invention on the VOC dataset are better than those of the existing methods. These results verify the effectiveness of the technical solution of the present invention in the weakly supervised semantic segmentation task.

[0161] Example 2

[0162] A weakly supervised target localization system based on a multi-layer decoupled attention localization network, including:

[0163] Network building module: used to build component-aware learning segmentation network;

[0164] The component perception learning segmentation network includes a component-whole refinement module and a backbone network connected in series, as well as a classification branch network and a component branch network; the classification branch network and the component branch network are connected in parallel and then connected in series to the back end of the backbone network;

[0165] The component branch network includes a component perception learning module, a Transformer Encoder module, a fully connected layer, and a classifier connected in sequence; the component perception learning module includes an embedding feature extraction unit, an embedding feature rescaling unit, a mask feature fusion unit, and an image feature output unit connected in series in sequence;

[0166] SAM mask set acquisition module: input the training set image into the SAM segmentation model to obtain the SAM mask set of the training set image;

[0167] The part-whole refinement module includes a small object removal mask unit, a high similarity removal mask unit, and a relationship matrix unit that records the relationship between the target parts and the whole in the mask. It uses the SAM mask set as input to obtain the mask pseudo-label set of the training set images and the relationship matrix between the target parts and the whole in the mask.

[0168] Segmentation network training module: Using the known training set images, image-level labels, mask pseudo-label set, and the relationship matrix between the target parts and the whole in the mask, the component-aware learning segmentation network is trained to obtain the initial component-aware learning segmentation network model;

[0169] Segmentation result optimization module: Use the initial component-aware learning segmentation network model to obtain the CAM set of the target mask in the training set image, and obtain the final pseudo-label set through threshold segmentation; use the final pseudo-label set as supervision information, and simultaneously train the initial component-aware learning segmentation network and the segmentation decoder connected in series to the back end of the component branch network to obtain a complete component-aware learning segmentation network model to obtain the semantic segmentation results in the image to be segmented.

[0170] It should be understood that the implementation process of each module can refer to the description of the aforementioned method. The above division of functional modules is merely a division of logical functions. In actual implementation, other division methods may be used. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not implemented. Furthermore, the above-mentioned integrated units may be implemented in the form of hardware or software functional units.

[0171] Example 3

[0172] A computer-readable storage medium stores a computer program, wherein the computer program is called by a processor to implement:

[0173] The above steps are based on a weakly supervised target localization method of a multi-layer decoupled attention localization network.

[0174] For the specific implementation process of each step, please refer to the description of the above method.

[0175] It should be understood that in the embodiments of the present invention, the processor referred to may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The memory may include a read-only memory and a random access memory, and provides instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type.

[0176] The readable storage medium is a computer-readable storage medium, which can be an internal storage unit of the software and hardware device described in any of the aforementioned embodiments, such as a hard disk or memory of a controller. The readable storage medium can also be an external storage device of the controller, such as a plug-in hard disk equipped on the controller, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Furthermore, the readable storage medium can also include both an internal storage unit of the controller and an external storage device. The readable storage medium is used to store the computer program and other programs and data required by the controller. The readable storage medium can also be used to temporarily store data that has been output or is to be output.

[0177] Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned readable storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0178] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The present application is a flow chart according to the method, device (system), and computer program product of the embodiment of the present application and / or the instructions executed by the processor to generate a device for realizing the function specified in one flow chart or multiple flows and / or one box or multiple boxes of the block diagram. These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a product comprising an instruction device, which realizes the function specified in one flow chart or multiple flows and / or one box or multiple boxes of the block diagram. These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0179] It should be emphasized that the examples described in the present invention are illustrative rather than restrictive. Therefore, the present invention is not limited to the examples described in the specific embodiments. Any other embodiments derived by those skilled in the art based on the technical solution of the present invention that do not depart from the purpose and scope of the present invention, whether modified or replaced, also fall within the scope of protection of the present invention.

Claims

1. A weakly supervised semantic segmentation method based on component-aware learning segmentation network, characterized in that include: Step 1: Build a component-aware learning segmentation network; The component perception learning segmentation network includes a component-whole refinement module and a backbone network connected in series, as well as a classification branch network and a component branch network; the classification branch network and the component branch network are connected in parallel and then connected in series to the back end of the backbone network; The component-whole refinement module sequentially performs small target mask removal, high similarity mask removal, and relationship matrix recording processing between the target component and the whole in the mask; The component branch network includes a component perception learning module, a Transformer Encoder module, a fully connected layer, and a classifier connected in sequence; the component perception learning module sequentially performs embedding feature extraction, embedding feature scale reshaping, mask feature fusion, and image feature output processing; Step 2: Input the training set image into the SAM segmentation model to obtain the SAM mask set of the training set image; Step 3: Use the SAM mask set as input to the part-whole refinement module of the part-aware learning segmentation network to obtain the mask pseudo-label set of the training set image and the relationship matrix between the target part and the whole in the mask; Step 4: Take the known training set images, image-level labels, the mask pseudo-label set of the training set images, and the relationship matrix between the target parts and the whole in the mask as the input of the component-aware learning segmentation network, train the component-aware learning segmentation network, and obtain the initial component-aware learning segmentation network model; Step 5: Use the initial component-aware learning segmentation network model to obtain the CAM set of the target mask in the training set image, and obtain the final pseudo-label set through threshold segmentation; use the final pseudo-label set as supervision information, and simultaneously train the initial component-aware learning segmentation network model and the segmentation decoder connected in series to the back end of the component branch network to obtain a complete component-aware learning segmentation network model. Finally, input the image to be segmented into the complete component-aware learning segmentation network model to obtain the semantic segmentation result in the image to be segmented.

2. The method according to claim 1, characterized in that The backbone network is composed of ViT; the classification branch network includes fully connected layers and classifiers connected in sequence.

3. The method according to claim 1, characterized in that The working process of the component-whole refining module is as follows: Step A1: small object mask removal; Input is the SAM mask set of the training set image calculate The proportion of foreground pixels in each mask is U. If the proportion U is lower than the set threshold τ, the mask is removed and the output mask set is obtained. Among them, M i represents the i-th mask in the SAM mask set of the training set image, where M represents the SAM mask set, i is the index of the mask in the set; K is the total number of masks; represents the i-th mask in the output mask set, where Represents the output mask set, i is the index of the mask in the set; is the number of masks in the output mask set; Step A2: high similarity mask removal; Input is the output mask set The overlap between masks is calculated, and masks with an overlap exceeding the set threshold β are judged as high-similarity masks and removed to obtain the pseudo-label set of the training set images. Among them, i,j are Different mask indices in , representing the i-th mask and the j-th mask; represents the i-th mask in the pseudo-label set of the training set images, is the total number of masks in the pseudo-label set of the training set images; and Represents masks respectively and mask The size of the intersection and union of the two masks; the number of pixels in the overlapping area and the total area covered by the two masks are usually calculated respectively; f(i,j) is used to determine whether to retain the mask; if there is no other mask, and If the overlap exceeds the threshold β, then f(i,j)=1, and the mask is retained Otherwise f(i,j)=1, and remove the mask Step A3: record the relationship matrix between the target part and the whole in the mask; Take the pseudo-label set of training set images As input, the relationship matrix between the target parts and the whole in the mask of the pseudo label set is constructed; By calculating the overlap between masks and setting a threshold, we can determine which masks represent the relationship between the parts and the whole. The mask in and The overlapping part occupy When the ratio is greater than or equal to the threshold λ, As the overall mask, identify yes Component mask; using the indicator function h(i,j), establish the relationship matrix between the target and its components Where λ is the threshold, if yes If the component mask is , then h(i,j)=1; otherwise h(i,j)=0; through h(i,j), we can get the relationship matrix between the target component and the whole in the mask 4. The method according to claim 3, characterized in that The working process of the component perception learning module is as follows: Step B1: Extract embedding features; Embedding features of training set images extracted from the penultimate layer of the Transformer encoder By removing the embedded features Classify the label dimension and get the remaining embedding features Among them, B, N, and E represent the size of the input batch, the number of image blocks into which the image is divided, and the dimension of the feature vector of each image block, respectively. represents the field of real numbers; Step B2: Reshape the embedding feature scale; Embed features The shape of is rearranged from B×N×E to B×C×H×W; then, the mask pseudo label set of the training set image output from the part-whole refinement module is combined with the embedded feature Point product to get the embedded feature set Then embed the feature set The shape of is rearranged from B×k×C×H×W to B×k×N×E; and Each feature in is reduced to 1 / k by a 1×1 filter layer, and the mask feature set is output. Among them, H, W, and C represent the height, width, and feature dimension of the training set image respectively, and k represents the number of masks in the mask pseudo-label set; Step B3: mask feature fusion; Mask feature set Perform average pooling and normalization to obtain the weighted coefficient Based on the relationship matrix The corresponding overall mask and component mask are used to obtain the corresponding overall mask feature and component mask features By applying a shared 3×3 convolutional layer to each component mask feature, the component mask feature is updated, and all updated component mask features are added and fused with their corresponding overall mask features to obtain the fused overall mask feature. Another shared 3×3 convolutional layer is then used to update the fused overall mask feature to generate a new overall mask feature. Integrate all updated overall and component mask features to obtain a new mask feature set The updated mask feature set is updated by the weight coefficient W Then perform weighted splicing to generate weighted mask features Among them, cat is the feature splicing operation; W1, W2, W k represents the weighting coefficient, is the mask feature in the updated mask feature set; Step B4: output image features; Weighted mask features Use 1×1 convolution layer to get features Then output features It is concatenated with the removed category token to obtain the final output feature.

5. The method according to claim 1, wherein The training set images are input into the SAM segmentation model to obtain the SAM mask set of the training set images as follows: Taking the known training set images as input, the training images are first preprocessed, including image resizing, image grayscale and normalization, and color space conversion operations; then the preprocessed training set images are input into the pre-trained SAM segmentation model. The pre-trained SAM segmentation model automatically segments each training image, generates segmentation masks for multiple target areas, and obtains the SAM mask set of the training set images.

6. The method according to claim 1, characterized in that The process of using the known training set images, image-level labels, the mask pseudo-label set of the training set images, and the relationship matrix between the target component and the whole in the mask as inputs to the component-aware learning segmentation network, training the component-aware learning segmentation network, and obtaining the initial component-aware learning segmentation network model is as follows: Taking the input training set images as input, image-level labels as supervision information, the mask pseudo-label set of the training set images and the relationship matrix between the target parts and the whole in the mask as auxiliary information, a joint loss function is used to train the component-aware learning segmentation network containing the component-aware learning module using a classifier to obtain a trained initial component-aware learning segmentation network model.

7. The method according to claim 6, characterized in that The joint loss function is composed of the classification cross entropy loss function, the auxiliary classification loss function and the contrast loss function, and the formula is: Among them, L all Represents the joint loss function, λ1, λ2, λ3 are used to balance L cls 、 and L reg Regularization parameter of ; L cls Represents the classification cross entropy loss function: in, is the predicted probability of category c, y c is the true label; C represents the number of categories; Represents the auxiliary classification loss function: in, Represents the predicted label obtained from the features of the component branch network, and softmax is the activation function; L reg Represents the contrast loss function: Among them, CosSim() represents the cosine similarity calculation, N + and N - They represent the number of positive sample pairs, i.e., samples with component relationships, and negative sample pairs, i.e., the number of samples other than positive samples. Represents the mask feature set elements.

8. The method according to claim 7, characterized in that The final pseudo-label set is used as supervision information to calculate the segmentation loss function. Based on the joint loss function plus the segmentation loss function, the initial component-aware learning segmentation network and the additional segmentation decoder connected in series to the back end of the component branch network are trained simultaneously to obtain a complete component-aware learning segmentation network model. Finally, the image to be segmented is input into the complete component-aware learning segmentation network model to infer and obtain the semantic segmentation result in the image to be segmented.

9. A weakly supervised semantic segmentation system based on a component-aware learning segmentation model, characterized in that include: Network building module: used to build component-aware learning segmentation network; The component perception learning segmentation network includes a component-whole refinement module and a backbone network connected in series, as well as a classification branch network and a component branch network; the classification branch network and the component branch network are connected in parallel and then connected in series to the back end of the backbone network; The component branch network includes a component perception learning module, a Transformer Encoder module, a fully connected layer, and a classifier connected in sequence; the component perception learning module includes an embedding feature extraction unit, an embedding feature rescaling unit, a mask feature fusion unit, and an image feature output unit connected in series in sequence; SAM mask set acquisition module: input the training set image into the SAM segmentation model to obtain the SAM mask set of the training set image; The part-whole refinement module includes a small object removal mask unit, a high similarity removal mask unit, and a relationship matrix unit that records the relationship between the target parts and the whole in the mask. It uses the SAM mask set as input to obtain the mask pseudo-label set of the training set images and the relationship matrix between the target parts and the whole in the mask. Segmentation network training module: Using the known training set images, image-level labels, mask pseudo-label set, and the relationship matrix between the target parts and the whole in the mask, the component-aware learning segmentation network is trained to obtain the initial component-aware learning segmentation network model; Segmentation result optimization module: Use the initial component-aware learning segmentation network model to obtain the CAM set of the target mask in the training set image, and obtain the final pseudo-label set through threshold segmentation; use the final pseudo-label set as supervision information, and simultaneously train the initial component-aware learning segmentation network and the segmentation decoder connected in series to the back end of the component branch network to obtain a complete component-aware learning segmentation network model. Input the image to be segmented into the complete component-aware learning segmentation network model to obtain the semantic segmentation result in the image to be segmented.

10. A computer storage medium, characterized in that: A computer program is stored, which is called by a processor to implement: The steps of the method according to any one of claims 1 to 8.

Images