Weak supervision semantic segmentation method and system based on component perception learning segmentation network

Abstract

The invention discloses a weak supervision semantic segmentation method and system based on a component perception learning segmentation network. The method comprises the following steps: step 1, constructing the component perception learning segmentation network; step 2, acquiring an SAM mask set of the training set image; step 3, obtaining a mask pseudo label set of the training set image and a relation matrix between a target component in a mask and the whole; 4, obtaining an initial part perception learning segmentation network model; step 5, obtaining and optimizing a CAM set of a target mask in the training set image to obtain a final pseudo label set, and obtaining a semantic segmentation result in the to-be-segmented image by using a segmentation decoder; removing redundant masks through a part-whole refining module in the network, and capturing a relationship between a target part and a whole in the masks; through a component perception learning module in the network, the network is encouraged to learn the relationship between each component of the target in the pseudo tag set and the whole, and accurate and complete segmentation of the target under a weak supervision condition is realized.

Description

Technical Field

[0001] The present invention belongs to the field of semantic segmentation, and in particular relates to a weakly supervised semantic segmentation method and system based on a component-aware learning segmentation network. Background Art

[0002] Semantic segmentation is an important task in computer vision and has made remarkable progress in the past few decades. Typically, semantic segmentation methods adopt a fully supervised learning paradigm and use large-scale datasets with pixel-level annotations to train segmentation models. However, obtaining accurate annotations is a time-consuming and costly process. In addition, labels are often incorrect due to misjudgment by the annotator. To address these challenges, weakly supervised semantic segmentation (WSSS) has become an important research hotspot. It only uses weakly and low-cost annotation forms, such as image-level labels, points, lines, and bounding boxes, as the main supervision information for training segmentation models.

[0003] In the pursuit of accurate weakly supervised semantic segmentation, previous studies usually generate class activation maps (CAMs) (reference: B.Zhou, A.Khosla, A.Lapedriza, A.Oliva, and A.Torralba, "Learning deep features for discriminative localization," in Proc.IEEE Conf.Comput.Vis.Pattern Recognit., 2016, pp.2921–2929.) or their variants through classification supervision and use them as pseudo labels. However, CAM-based methods mainly rely on classification losses and often only activate the most discriminative regions of the target. Therefore, the generated pseudo labels are usually incomplete and only cover part of the target.

[0004] In pursuit of accurate semantic segmentation, Wei et al. (reference: Yunchao Wei, Jiashi Feng, Xiaodan Liang, Ming-Ming Cheng, Yao Zhao, and Shuicheng Yan, "Object regionmining with adversarial erasing: A simple classification to semantic segmentation approach," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2017, pp. 1568–1576.) further mined more target regions based on the original CAM through an adversarial erasing strategy. Wang et al. (reference: Yude Wang, Jie Zhang, Meina Kan, Shiguang Shan, and Xilin Chen, "Self-supervised equivariant attention mechanism for weakly supervised semantic segmentation," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2020, pp. 12275–12284.) regularized the training objective by introducing an auxiliary task of scale-invariant regularization. Yao et al. (reference: YazhouYao, Tao Chen, Guo-Sen Xie, Chuanyi Zhang, Fumin Shen, Qi Wu, ZhenminTang, and Jian Zhang, “Self-supervised equivariant attention mechanism for weakly supervised semantic segmentation,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2021, pp. 2623–2632.) adopted additional saliency maps as supervisory information to suppress background regions and mine non-salient objects.Zhou et al. (Reference: Tianfei Zhou, Meijie Zhang, Fang Zhao, and Jianwu Li, “Regional semantic contrast and aggregation for weakly supervised semantic segmentation,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2022, pp. 4299–4309.) promote the complete activation of the target region by comparing pixel and prototype representations.

[0005] However, these Convolutional Neural Network (CNN)-based methods are mainly used to identify the main feature areas, resulting in pseudo-labels lacking detailed boundary information. Compared with CNN, Vision Transformer (ViT) (reference: Alexey, Dosovitskiy. "An image is worth 16x16words: Transformers for image recognition at scale," arXiv preprint arXiv:2010.11929, 2020.) has achieved remarkable success in WSSS because its self-attention mechanism can capture global information. These ViT-based methods enhance long-range feature dependencies to better learn semantic or positional affinity and differences, making CAM boundaries more accurate. However, pseudo-labels from ViT still have the problem of inaccurate boundaries.

[0006] Recently, SegmentAnything Model (SAM) (reference: A.Kirillov, E.Mintun, N.Ravi, H.Mao, C.Rolland, L.Gustafson, T.Xiao, S.Whitehead, A.C.Berg, W.-Y.Lo et al., “Segment anything,” arXiv preprint arXiv:2304.02643, 2023.) has performed well in unsupervised semantic segmentation. Therefore, the present invention aims to generate more accurate pseudo-labels through SAM and reduce the reliance on CAM. However, SAM still has the following problems in mask prediction: (1) The SAM method usually generates a large number of low-quality masks that are difficult to distinguish between objects and backgrounds without supervision. This lack of distinguishability leads to inaccurate segmentation results and difficulty in separating overlapping objects. (2) SAM often segments the entire object into multiple segments, resulting in complex and inaccurate pseudo-labels because it tends to classify them based on pixel similarity. However, even when trained with CAM guidance, SAM fails to generate complete and accurate masks for complex, textured objects. This limitation suggests that SAM-based methods still face some challenges in providing complete and accurate pseudo-labels when dealing with challenging segmentation tasks. Summary of the invention

[0007] In view of the above problems, the technical solution of the present invention proposes a weakly supervised semantic segmentation method and system based on a component-aware learning segmentation network, which uses the mask predicted by SAM to guide the generation of more accurate CAM, thereby generating more precise pseudo labels with clearer boundaries. This technical solution overcomes the limitations of traditional CAM-based methods, solves the difficulty of using SAM in weakly supervised semantic segmentation, and improves the accuracy of weakly supervised semantic segmentation.

[0008] On the one hand, a weakly supervised semantic segmentation method based on a component-aware learning segmentation network comprises:

[0009] Step 1: Build a component-aware learning segmentation network;

[0010] The component-aware learning segmentation network includes a component-whole refining module and a backbone network connected in series, as well as a classification branch network and a component branch network; the classification branch network and the component branch network are connected in parallel and then connected in series to the back end of the backbone network;

[0011] The component-whole refining module sequentially performs small target mask removal, high similarity mask removal, and relationship matrix recording processing between the target component and the whole in the mask;

[0012] The component branch network includes a component perception learning module, a Transformer Encoder module, a fully connected layer and a classifier connected in sequence; the component perception learning module sequentially performs embedding feature extraction, embedding feature scale reshaping, mask feature fusion and image feature output processing;

[0013] Step 2: Input the training set images into the SAM segmentation model to obtain the SAM mask set of the training set images;

[0014] Step 3: Use the SAM mask set as input to the component-whole refinement module of the component-aware learning segmentation network to obtain the mask pseudo-label set of the training set image and the relationship matrix between the target component and the whole in the mask;

[0015] Step 4: Take the known training set images, image-level labels, the mask pseudo-label set of the training set images, and the relationship matrix between the target component and the whole in the mask as the input of the component-aware learning segmentation network, train the component-aware learning segmentation network, and obtain the initial component-aware learning segmentation network model;

[0016] Step 5: Use the initial component-aware learning segmentation network model to obtain the CAM set of the target mask in the training set image, and obtain the final pseudo-label set through threshold segmentation; use the final pseudo-label set as supervision information, and simultaneously train the initial component-aware learning segmentation network model and the segmentation decoder connected in series to the back end of the component branch network to obtain a complete component-aware learning segmentation network model, and finally input the image to be segmented into the complete component-aware learning segmentation network model to obtain the semantic segmentation result in the image to be segmented.

[0017] In the weakly supervised semantic segmentation task, a pseudo-label is an approximate pixel-level label because it is not the true classification information of each pixel. Usually, the target area in the image is predicted by the currently trained model, and these prediction results are used as the pseudo pixel-level labels of the target. Therefore, compared with the image-level labels, pseudo-labels have a higher degree of granularity, but slightly lower accuracy, and are usually generated by class activation maps (CAM) or other feature-based strategies. In this paper, the unsupervised SAM segmentation model is used for prediction and its output is preprocessed to generate pseudo-labels to guide the model to learn the regional boundaries of different targets in the image.

[0018] The process of training the segmentation decoder is as follows: input the training set image into the component-aware learning segmentation network, and use the output of the component branch network connected in series after the backbone network as input. The final pseudo-label is used as supervision information, and the conventional cross-entropy loss function is used. While training the initial component-aware learning segmentation network, the segmentation decoder is trained at the same time to obtain a complete component-aware learning segmentation network model.

[0019] Furthermore, the backbone network is composed of ViT (Vision Transformer Encoder); the classification branch network includes fully connected layers and classifiers connected in sequence.

[0020] Furthermore, the working process of the component-whole refining module is as follows:

[0021] Small object mask removal and high similarity mask removal are used to refine the SAM mask set, and improve data quality by screening effective masks; while the relationship matrix between the target parts and the whole in the record mask is the core of the part-whole refinement module, which constructs and extracts the relationship matrix between the target parts and the whole. This matrix will play a key role in subsequent model training, helping the model to better learn the correlation between the target parts and the whole;

[0022] Step A1: small object mask removal;

[0023] Remove small object masks that do not meet the requirements from the SAM mask set of the training set images;

[0024] Input is the SAM mask set of the training set image calculate The proportion of foreground pixels in each mask is U. If the proportion U is lower than the set threshold τ = 0.05, the mask is removed to obtain the output mask set

[0025]

[0026] Among them, M i represents the i-th mask in the SAM mask set of the training set image, where M represents the SAM mask set, i is the index of the mask in the set; K is the total number of masks; represents the i-th mask in the output mask set, where represents the output mask set, i is the index of the mask in the set; is the number of masks in the output mask set;

[0027] Step A2: high similarity mask removal;

[0028] Remove masks with too high similarity from the output mask set to reduce redundancy;

[0029] Input is the output mask set The overlap between masks is calculated, and masks with an overlap exceeding the set threshold β are judged as high-similarity masks and removed to obtain the pseudo-label set of the training set images.

[0030] The pseudo-label set of the training set images consists of relatively independent valid masks, which are used for subsequent model training;

[0031]

[0032] in, and Respectively represent masks and mask The size of the intersection and union area of ​​the two masks; usually the number of pixels in the overlapping area of ​​the two masks and the total area covered are calculated respectively;

[0033] Among them, i,j are Different mask indices in , representing the i-th mask and the j-th mask; represents the i-th mask in the pseudo-label set of the training set images, is the total number of masks in the pseudo-label set of the training set images; f(i,j) is used to determine whether to retain the mask; if there is no other mask, and If the overlap exceeds the threshold β, then f(i,j) = 1, and the mask is retained. Otherwise f(i,j)=1 and remove the mask The threshold β was set to 0.95;

[0034] Step A3: record the relationship matrix between the target part and the whole in the mask;

[0035] Take the pseudo-label set of training set images As input, construct the relationship matrix between the target parts and the whole in the mask of the pseudo label set;

[0036] By calculating the overlap between masks and setting a threshold, we can determine which masks represent the relationship between the parts and the whole. The mask in and The overlapping part occupy When the ratio is greater than or equal to the threshold λ, As the overall mask, identify yes Component mask; using the indicator function h(i,j), establish the relationship matrix between the target and its components

[0037]

[0038] Where λ is the threshold value of 0.9. yes If the component mask is , then h(i,j)=1; otherwise h(i,j)=0; through h(i,j), we can get the relationship matrix between the target component and the whole in the mask

[0039]

[0040] Through these three closely cooperating parts, the part-whole refinement module not only effectively optimizes the SAM mask set in the training set images to obtain a higher quality pseudo-label set for the training set images, but also accurately constructs the relationship between the target parts and the whole; this module provides more reliable additional information for model training, and by utilizing this information, it enhances the model's ability to associate the overall characteristics of the target with the characteristics of the components.

[0041] Furthermore, the working process of the component perception learning module is as follows:

[0042] Step B1: extracting embedding features;

[0043] Embedding features of training set images extracted from the penultimate layer of the Transformer encoder By removing the embedded features Classify the label dimension and get the remaining embedding features

[0044] Among them, B, N, and E represent the size of the input batch, the number of image blocks into which the image is divided, and the dimension of the feature vector of each image block, respectively. represents the field of real numbers;

[0045] Step B2: Reshape the embedding feature scale;

[0046] Embed features The shape of is rearranged from B×N×E to B×C×H×W; then, the mask pseudo-label set of the training set images output from the part-whole refinement module is combined with the embedded features Point product to get the embedded feature set Then embed the feature set The shape of is rearranged from B×k×C×H×W to B×k×N×E; and Each feature in is reduced to 1 / k by a 1×1 filter layer, and the mask feature set is output.

[0047] Among them, H, W, and C represent the height, width, and feature dimension of the training set image respectively, and k represents the number of masks in the mask pseudo-label set;

[0048] Step B3: mask feature fusion;

[0049] Mask feature set Perform average pooling and normalization to obtain the weighted coefficient

[0050] Based on the relationship matrix The corresponding overall mask and component mask are obtained to obtain the corresponding overall mask feature and component mask features

[0051] Import the relationship matrix output by the component-whole refinement module When the indicator function h(i,j) in the relationship matrix is ​​1, it means that the mask is the overall mask, mask Its component mask; where i,j is the pseudo label set of the training set image Different mask indices in , representing the i-th mask and the j-th mask; due to the pseudo label set of the training set images With mask feature set One-to-one correspondence, overall mask The mask feature Denoted as the overall mask feature, and the mask feature that satisfies the number h(i,j)=1 is recorded as the component mask feature; the number of component masks associated with the overall mask feature is determined by the overall mask The corresponding entry h(i,j)=1 is determined;

[0052] By applying a shared 3×3 convolutional layer to each component mask feature, the component mask feature is updated, and all updated component mask features are added and fused with their corresponding overall mask features to obtain the fused overall mask feature, and then another shared 3×3 convolutional layer is used to update the fused overall mask feature to generate a new overall mask feature;

[0053] Integrate all updated overall and component mask features to obtain a new mask feature set

[0054] The updated mask feature set is updated by the weight coefficient W Then perform weighted concatenation to generate weighted mask features

[0055]

[0056] Among them, cat is the feature concatenation operation; W1, W2, W k represents the weighting coefficient, is the mask feature in the updated mask feature set;

[0057] Step B4: output image features;

[0058] Weighted mask features Use 1×1 convolution layer to get the features Then output features It is concatenated with the removed category token to obtain the final output feature.

[0059] By using the mask pseudo-label set of the training set images output by the part-whole refinement module and the relationship matrix between the target parts and the whole in the mask, and through the synergy of the various units and of the above component perception learning module, the model can efficiently extract and fuse the mask features of the training set images. At the same time, the relationship matrix between the part and the whole is used to strengthen the model's learning of the relationship between the target parts and the whole, further improving the performance of the model.

[0060] Furthermore, the training set images are input into the SAM segmentation model to obtain the SAM mask set of the training set images as follows:

[0061] Taking the known training set images as input, the training images are first preprocessed, including image size adjustment, image grayscale and normalization, and color space conversion operations; then the preprocessed training set images are input into the pre-trained SAM segmentation model. The pre-trained SAM segmentation model automatically segments each training image, generates segmentation masks of multiple target areas, and obtains the SAM mask set of the training set images.

[0062] Furthermore, the process of taking known training set images, image-level labels, mask pseudo-label sets of training set images, and the relationship matrix between target components and the whole in the mask as inputs of the component-aware learning segmentation network, training the component-aware learning segmentation network, and obtaining the initial component-aware learning segmentation network model is as follows:

[0063] Taking the input training set images as input, image-level labels as supervisory information, the mask pseudo-label set of the training set images and the relationship matrix between the target parts and the whole in the mask as auxiliary information, a joint loss function is used to train the component-aware learning segmentation network containing the component-aware learning module using a classifier to obtain a trained initial component-aware learning segmentation network model.

[0064] Furthermore, the joint loss function is composed of a classification loss function, an auxiliary classification loss function and a contrast loss function, and the formula is:

[0065]

[0066] Among them, λ1, λ2, λ3 are used to balance the classification loss L cls , and contrast loss L reg The regularization parameter of

[0067] L cls Represents the cross entropy loss function:

[0068]

[0069] in, is the predicted probability of category c, y c is the true label; C represents the number of categories;

[0070] Represents the auxiliary classification loss function:

[0071]

[0072] in, represents the predicted label obtained from the features of the component branch network, and softmax is the activation function;

[0073] L reg Represents the contrast loss function:

[0074]

[0075]

[0076] Among them, CosSim() represents the cosine similarity calculation, N + / N - Respectively represent the number of positive sample pairs (samples with component relationships) and negative sample pairs (samples other than positive samples), Represents the mask feature set elements.

[0077] Furthermore, the final pseudo-label set is used as supervision information to calculate the segmentation loss function, and based on the joint loss function plus the segmentation loss function, the initial component-aware learning segmentation network and the additional segmentation decoder connected in series to the back end of the component branch network are trained simultaneously to obtain a complete component-aware learning segmentation network model; finally, the image to be segmented is input into the complete component-aware learning segmentation network model to infer and obtain the semantic segmentation result in the image to be segmented.

[0078] In the second aspect, a weakly supervised semantic segmentation system based on a component-aware learning segmentation model includes:

[0079] Network building module: used to build component-aware learning segmentation network;

[0080] The component-aware learning segmentation network includes a component-whole refining module and a backbone network connected in series, as well as a classification branch network and a component branch network; the classification branch network and the component branch network are connected in parallel and then connected in series to the back end of the backbone network;

[0081] The component branch network includes a component perception learning module, a Transformer Encoder module, a fully connected layer and a classifier connected in sequence; the component perception learning module includes an embedding feature extraction unit, an embedding feature reshaping unit, a mask feature fusion unit, and an image feature output unit connected in series in sequence;

[0082] SAM mask set acquisition module: input the training set image into the SAM segmentation model to obtain the SAM mask set of the training set image;

[0083] Part-whole refinement module: It includes a small object removal mask unit, a high similarity removal mask unit, and a relationship matrix unit for recording the target parts in the mask and the whole, which are connected in series in sequence. It takes the SAM mask set as input to obtain the mask pseudo-label set of the training set image and the relationship matrix between the target parts in the mask and the whole.

[0084] Segmentation network training module: Use the known training set images, image-level labels, mask pseudo-label sets, and the relationship matrix between the target components and the whole in the mask to train the component-aware learning segmentation network and obtain the initial component-aware learning segmentation network model;

[0085] Segmentation result optimization module: Use the initial component-aware learning segmentation network model to obtain the CAM set of the target mask in the training set image, and obtain the final pseudo-label set through threshold segmentation; use the final pseudo-label set as supervision information, and simultaneously train the initial component-aware learning segmentation network and the segmentation decoder connected in series to the back end of the component branch network to obtain a complete component-aware learning segmentation network model to obtain the semantic segmentation results in the image to be segmented.

[0086] In a third aspect, a computer storage medium stores a computer program, wherein the computer program is called by a processor to implement:

[0087] The above steps are a weakly supervised semantic segmentation method based on a component-aware learning segmentation model.

[0088] Beneficial Effects

[0089] Compared with the existing method, the advantages of the present invention are:

[0090] 1. Unlike previous methods, the present invention utilizes the SAM mask set to guide the generation of more accurate CAMs, thereby producing final pseudo-labels with clearer boundaries. The present invention proposes a part-aware learning segmentation network to overcome the limitations of traditional CAM-based methods and address two challenges in SAM-based WSSS: (1) SAM generates a large number of low-quality masks and has difficulty distinguishing foreground and background under unsupervised conditions. This limitation leads to a decrease in segmentation accuracy and an inability to effectively separate overlapping targets. (2) Since SAM relies on pixel similarity for distinction, it often segments a single target into multiple parts, thereby generating complex and inaccurate pseudo-labels. Compared with existing SAM-based WSSS methods, the present invention relies on the part-whole relationship between target masks to effectively utilize the SAM mask set to generate high-quality pseudo-labels.

[0091] 2. The part-whole refinement module proposed in the present invention refines the SAM mask set and captures the part-whole relationship in the refined mask to obtain the relationship matrix between the target part and the whole in the mask. This module can provide a high-quality mask pseudo-label set and provide more reliable information for training the segmentation network model.

[0092] 3. The part-aware learning module proposed in the present invention can effectively capture the relationship between the target as a whole and its various parts, thereby avoiding the loss of target information caused by the model only focusing on the most salient areas, thereby improving the accuracy of weakly supervised semantic segmentation.

[0093] 4. The present invention preferably proposes a joint loss function, including classification loss, auxiliary classification loss and contrast loss, which can help the model better learn the relationship between the target as a whole and its components, while effectively utilizing pseudo-labels to distinguish the target foreground and background. BRIEF DESCRIPTION OF THE DRAWINGS

[0094] Figure 1 A schematic diagram of a component-aware learning segmentation network described in an embodiment of the present invention;

[0095] Figure 2 It is a schematic diagram of the component-integrated refining module mentioned in the embodiment of the present invention;

[0096] Figure 3 Schematic diagram of the structure of the component perception learning module mentioned in the embodiment of the present invention;

[0097] Figure 4 This is a schematic diagram comparing the semantic segmentation results of the method described in the present invention on the VOC and COCO datasets with the evaluation indicator mIoU (mean intersection over union); PLS is the abbreviation of the weakly supervised semantic segmentation method based on the component-aware learning segmentation model.

[0098] Figure 5This is a schematic diagram of the visualization results of the method described in the present invention on the VOC dataset. DETAILED DESCRIPTION

[0099] The technical solution of the present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0100] Example 1

[0101] The embodiment of the present invention provides a weakly supervised semantic segmentation method based on component-aware learning segmentation network, such as Figure 1 As shown, the specific process is as follows:

[0102] Step 1: Build a component-aware learning segmentation network;

[0103] The component-aware learning segmentation network includes a component-whole refinement module, a backbone network, two parallel classification branch networks and component branch networks connected in series after the backbone network, such as Figure 1 shown in Figure 1 In the above, ViT is the backbone network used to extract image features; SAM represents the SAM mask set of the training set images obtained using the SAM model, and N + represents the positive sample, including all mask features with whole-part relationship; N - Represents a negative sample, and “3×3” represents a convolutional layer with a convolution kernel of 3×3 size; among them, all component mask features share one convolutional layer; all overall mask features share another convolutional layer with a convolution kernel of 3×3 size;

[0104] The backbone network is composed of ViT (Vision Transformer Encoder);

[0105] The classification branch network includes a fully connected layer and a classifier connected in sequence; Figure 1 As shown, “FC” represents a fully connected layer; “Classifier” represents a classifier;

[0106] The component branch network includes a component perception learning module, a Transformer Encoder module, a fully connected layer and a classifier connected in sequence. Figure 1 As shown in the figure, "TEB" means Transformer Encoder module; "FC" means fully connected layer; "Classifier" means classifier;

[0107] Since the classifier is only related to the Class Token of the last Transformer Encoder module, the present invention adds an extra Transformer Encoder module after the component-aware learning module, so that the output embedding of the component-aware learning module affects the classifier.

[0108] The component-whole refinement module sequentially performs small target mask removal, high similarity mask removal, and relationship matrix recording processing between the target component and the whole in the mask, such as Figure 2 As shown;

[0109] Among them, removing small target masks and removing high similarity masks are used to refine the SAM mask set, and the data quality is improved by screening effective masks; and the relationship matrix between the target parts and the whole in the record mask is used as the core of the part-whole refinement module to construct and extract the relationship matrix between the target parts and the whole. This matrix will play a key role in the subsequent model training, helping the model to better learn the correlation between the target parts and the whole.

[0110] Step A1: remove small object mask;

[0111] The main task is to remove small target masks that do not meet the conditions from the SAM mask set of the training set images; the input is the SAM mask set of the training set images Among them, M i represents the mask in the SAM mask set of the i-th training set image, K is the total number of masks; the specific operation is to calculate The proportion of foreground pixels in each mask is U. If the proportion is lower than the set threshold τ = 0.05, the mask is removed. After this processing, the output mask set of step A1 is obtained. in Represents the number of masks in the output mask set; the formula is as follows:

[0112]

[0113] Step A2: remove high similarity masks;

[0114] Remove masks with too high similarity from the output mask set filtered in step A1 to reduce redundancy; the input is the mask set in, represents the mask in the i-th output mask set, is the total number of masks; the specific operation is to calculate the overlap between masks (i.e., intersection-over-union ratio), and for masks whose overlap exceeds the set threshold β, determine them as high-similarity masks and remove them; the output mask set consists of relatively independent valid masks, which is used as the pseudo-label set of the training set images for subsequent model training, and is recorded as the pseudo-label set of the training set images The formula is as follows:

[0115]

[0116] Among them, i,j are Different mask indices in , representing the i-th mask and the j-th mask; represents the mask in the pseudo-label set of the i-th training set image, is the total number of the mask; and Respectively represent masks and mask The size of the intersection and union area of ​​the two masks; usually the number of pixels in the overlapping area of ​​the two masks and the total area covered are calculated respectively;

[0117] f(i,j) is used to determine whether to keep the mask; if there is no other mask and If the overlap exceeds β, then f(i,j) = 1, and the mask is retained. Otherwise f(i,j)=1 and remove the mask The threshold β was set to 0.95;

[0118] Step A3: Record the relationship matrix between the target component and the whole in the mask;

[0119] The pseudo-label set of training set images As input, the relationship matrix between the target parts and the whole in the mask of the pseudo label set is constructed; the specific operation is to calculate the overlap between the masks and set the threshold to determine which masks represent the relationship between the parts and the whole; when the pseudo label set of the training set image The mask in and The overlapping part occupy When the ratio is greater than or equal to the threshold β, it is considered yes The component mask of the target can be constructed by using the indicator function h(i,j) to build the relationship matrix between the target and its components. It is defined as:

[0120]

[0121] Where λ is the threshold value of 0.9. yes If the component mask is , then h(i,j)=1; otherwise h(i,j)=0; through h(i,j), we can get the relationship matrix between the target component and the whole in the mask

[0122]

[0123] Through the close collaboration of these three steps, the part-whole refinement module not only effectively optimizes the SAM mask set in the training set images to obtain a higher quality pseudo-label set for the training set images, but also accurately constructs the relationship between the target parts and the whole. This module provides more reliable additional information for model training, and by utilizing this information, it enhances the model's ability to associate the overall characteristics of the target with the component characteristics.

[0124] like Figure 3 As shown, the working process of the component perception learning module is as follows:

[0125] Step B1: extracting embedding features;

[0126] The input is the training set image embedding features extracted from the penultimate layer of the Transformer encoder Among them, B, N, and E represent the size of the input batch, the number of image blocks into which the image is divided, and the dimension of the feature vector of each image block, respectively. Represents the real number domain; by removing the embedded features Classify the label dimension and output the remaining embedding features

[0127] Step B2: Reshape the embedding features

[0128] Embed features The shape of is rearranged from B×N×E to B×C×H×W, where H, W, and C represent the height, width, and feature dimension of the training set image, respectively; then, the masked pseudo-label set of the training set image output from the part-whole refinement module is combined with the embedded feature Point product to get the embedded feature set Wherein, k represents the number of masks (in the embodiment of the present invention, the value is 7); then the embedded feature set The shape of is rearranged from B×k×C×H×W to B×k×N×E; and Each feature in is reduced to 1 / k by a 1×1 filter layer, and the mask feature set is output.

[0129] Step B3: Mask feature fusion

[0130] right Perform average pooling and normalization to obtain the weighted coefficient Based on the relationship matrix The corresponding overall mask and component mask are obtained to obtain the corresponding overall mask feature and component mask features

[0131] According to the relationship matrix output in the component-whole refinement module When the indicator function h(i,j) in the relationship matrix is ​​1, it means that the mask is the overall mask, mask Its component mask; where i,j is the pseudo label set of the training set image Different mask indices in , representing the i-th mask and the j-th mask; due to the pseudo label set of the training set images With mask feature set One-to-one correspondence, overall mask The mask feature Denoted as the overall mask feature, and the mask feature that satisfies the number h(i,j)=1 is recorded as the component mask feature; the number of component masks associated with the overall mask feature is determined by the overall mask The corresponding entry h(i,j)=1 is determined;

[0132] By applying a shared 3×3 convolutional layer to each component mask feature, the component mask feature is updated, and all updated component mask features are added and fused with their corresponding overall mask features to obtain the fused overall mask feature, and then another shared 3×3 convolutional layer is used to update the fused overall mask feature to generate a new overall mask feature;

[0133] Integrate all updated overall and component mask features to obtain a new mask feature set

[0134] The updated mask feature set is updated by the weight coefficient W Then perform weighted concatenation to generate weighted mask features

[0135]

[0136] Among them, cat is the feature concatenation operation; W1, W2, W k represents the weighting coefficient, is the mask feature in the updated mask feature set;

[0137] Step B4: output image features;

[0138] Weighted mask features Use 1×1 convolution layer to get the features Finally, the unit will output the features It is concatenated with the removed category token to obtain the final output feature.

[0139] By using the mask pseudo-label set of the training set images output by the part-whole refinement module and the relationship matrix between the target parts and the whole in the mask, and through the synergy of the various steps of the above component-aware learning module, the model can efficiently extract and fuse the mask features of the training set images. At the same time, the relationship matrix between the part and the whole is used to strengthen the model's learning of the relationship between the target part and the whole, further improving the performance of the model.

[0140] Step 2: Input the training set images into the SAM segmentation model to obtain the SAM mask set of the training set images;

[0141] Taking the known training set images as input, the images are first preprocessed, including image resizing, image grayscale and normalization, and color space conversion operations; then the preprocessed training set images are input into the pre-trained SAM segmentation model. The model will automatically segment each training image, generate segmentation masks for multiple target areas, and obtain the SAM mask set of the training set images.

[0142] Step 3: Utilize the part-whole refinement module of the part-aware learning segmentation network and take the SAM mask set as input to obtain the mask pseudo-label set of the training set image and the relationship matrix between the target parts and the whole in the mask, including three steps: removing small target masks, removing high similarity masks, and recording the relationship matrix between the target parts and the whole in the mask.

[0143] The pseudo-label is an approximate pixel-level label in the weakly supervised semantic segmentation task, because it is not the true classification information of each pixel. Usually, the target area in the image is predicted by the currently trained model, and these prediction results are used as the pseudo pixel-level label of the target. Therefore, compared with the image-level label, the pseudo-label has a higher degree of granularity, but slightly lower accuracy, and is usually generated by a class activation map (CAM) or other feature-based strategies. This paper uses the unsupervised SAM segmentation model for prediction and preprocesses its output to generate pseudo-labels to guide the model to learn the regional boundaries of different targets in the image.

[0144] Step 4: Using the component-aware learning segmentation network containing the component-aware learning module, with the known training set images, image-level labels, the mask pseudo-label set of the training set images, and the relationship matrix between the target parts and the whole in the mask as input, the process of obtaining the initial component-aware learning segmentation network model is as follows:

[0145] Taking the input training set image as input, the image-level label as supervision information, the mask pseudo-label set of the training set image and the relationship matrix between the target component and the whole in the mask as auxiliary information, using the joint loss function, and using the classifier to train the component-aware learning segmentation network including the component-aware learning module, a trained initial component-aware learning segmentation network model is obtained;

[0146] The joint loss function is composed of the classification loss function, the auxiliary classification loss function and the contrast loss function, and the formula is:

[0147]

[0148] Among them, λ1, λ2, λ3 are used to balance the classification loss L cls , and contrast loss L reg The regularization parameter of ;

[0149] L cls Represents the cross entropy loss function:

[0150]

[0151] in, is the predicted probability of category c, y c is the true label; C represents the number of categories;

[0152] Represents the auxiliary classification loss function:

[0153]

[0154] in, represents the predicted label obtained from the features of the component branch network, and softmax is the activation function;

[0155] L reg Represents the contrast loss function:

[0156]

[0157] Among them, CosSim() represents the cosine similarity calculation, N + / N - Represent the number of positive sample pairs and negative sample pairs respectively, Represents the mask feature set elements.

[0158] Step 5: Use the initial component-aware learning segmentation network model to obtain the CAM set of the target mask in the training set image, and obtain the final pseudo-label set through threshold segmentation; use the final pseudo-label set as supervision information, calculate the segmentation loss function, and based on the joint loss function plus the segmentation loss function, simultaneously train the initial component-aware learning segmentation network model and the segmentation decoder connected in series to the back end of the component branch network to obtain a complete component-aware learning segmentation network model, and finally input the image to be segmented into the complete component-aware learning segmentation network model to obtain the semantic segmentation result in the image to be segmented.

[0159] The process of obtaining the CAM of the target mask in the image to be segmented using the initial component-aware learning segmentation network model is as follows: the image to be segmented is input into the trained initial component-aware learning segmentation network model, the category weight of the image to be segmented is obtained through model prediction, and it is multiplied and added with the last layer of features to obtain the CAM of the target mask.

[0160] like Figure 4 As shown in FIG. 1 , the method described in the embodiment of the present invention achieves the best results in the semantic segmentation evaluation index mIoU (mean intersection over union) on the VOC and COCO datasets, and the segmentation accuracy exceeds that of the existing methods. In addition, Figure 5 As shown, the visualization results of the method described in the example of the present invention on the VOC dataset are better than those of the existing methods. These results verify the effectiveness of the technical solution of the present invention in the weakly supervised semantic segmentation task.

[0161] Example 2

[0162] A weakly supervised target localization system based on a multi-layer decoupled attention localization network, comprising:

[0163] Network building module: used to build component-aware learning segmentation network;

[0164] The component-aware learning segmentation network includes a component-whole refining module and a backbone network connected in series, as well as a classification branch network and a component branch network; the classification branch network and the component branch network are connected in parallel and then connected in series to the back end of the backbone network;

[0165] The component branch network includes a component perception learning module, a Transformer Encoder module, a fully connected layer and a classifier connected in sequence; the component perception learning module includes an embedding feature extraction unit, an embedding feature reshaping unit, a mask feature fusion unit, and an image feature output unit connected in series in sequence;

[0166] SAM mask set acquisition module: input the training set image into the SAM segmentation model to obtain the SAM mask set of the training set image;

[0167] Part-whole refinement module: It includes a small object removal mask unit, a high similarity removal mask unit, and a relationship matrix unit for recording the target parts in the mask and the whole, which are connected in series in sequence. It takes the SAM mask set as input to obtain the mask pseudo-label set of the training set image and the relationship matrix between the target parts in the mask and the whole.

[0168] Segmentation network training module: Use the known training set images, image-level labels, mask pseudo-label sets, and the relationship matrix between the target components and the whole in the mask to train the component-aware learning segmentation network and obtain the initial component-aware learning segmentation network model;

[0169] Segmentation result optimization module: Use the initial component-aware learning segmentation network model to obtain the CAM set of the target mask in the training set image, and obtain the final pseudo-label set through threshold segmentation; use the final pseudo-label set as supervision information, and simultaneously train the initial component-aware learning segmentation network and the segmentation decoder connected in series to the back end of the component branch network to obtain a complete component-aware learning segmentation network model to obtain the semantic segmentation results in the image to be segmented.

[0170] It should be understood that the implementation process of each module can refer to the content of the aforementioned method. The division of the above functional modules is only a division of logical functions. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. At the same time, the above integrated units can be implemented in the form of hardware or software functional units.

[0171] Example 3

[0172] A computer-readable storage medium stores a computer program, wherein the computer program is called by a processor to implement:

[0173] The above steps are a weakly supervised target localization method based on a multi-layer decoupled attention localization network.

[0174] For the specific implementation process of each step, please refer to the description of the above method.

[0175] It should be understood that in the embodiments of the present invention, the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The memory may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the type of device.

[0176] The readable storage medium is a computer-readable storage medium, which may be an internal storage unit of the software and hardware device described in any of the aforementioned embodiments, such as a hard disk or memory of a controller. The readable storage medium may also be an external storage device of the controller, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the controller. Furthermore, the readable storage medium may also include both an internal storage unit of the controller and an external storage device. The readable storage medium is used to store the computer program and other programs and data required by the controller. The readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0177] Based on this understanding, the technical solution of the present invention, in essence or in other words, the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned readable storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., various media that can store program codes.

[0178] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. The present application is a device for realizing the function specified in one flow or multiple flows of the flow chart and / or one or multiple blocks of the block diagram with reference to the instructions executed by the processor according to the method, device (system) and computer program product of the embodiment of the present application. These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which realizes the function specified in one flow or multiple flows of the flow chart and / or one or multiple blocks of the block diagram. These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0179] It should be emphasized that the examples described in the present invention are illustrative rather than restrictive, and therefore the present invention is not limited to the examples described in the specific embodiments. Any other embodiments derived by those skilled in the art based on the technical solution of the present invention that do not depart from the purpose and scope of the present invention, whether modified or replaced, also fall within the scope of protection of the present invention.

Claims

1. A weakly supervised semantic segmentation method based on component-aware learning segmentation network, characterized in that: include: Step 1: Build a component-aware learning segmentation network; The component-aware learning segmentation network includes a component-whole refining module and a backbone network connected in series, as well as a classification branch network and a component branch network; the classification branch network and the component branch network are connected in parallel and then connected in series to the back end of the backbone network; The component-whole refining module sequentially performs small target mask removal, high similarity mask removal, and relationship matrix recording processing between the target component and the whole in the mask; The component branch network includes a component perception learning module, a Transformer Encoder module, a fully connected layer and a classifier connected in sequence; the component perception learning module sequentially performs embedding feature extraction, embedding feature scale reshaping, mask feature fusion and image feature output processing; Step 2: Input the training set images into the SAM segmentation model to obtain the SAM mask set of the training set images; Step 3: Use the SAM mask set as input to the component-whole refinement module of the component-aware learning segmentation network to obtain the mask pseudo-label set of the training set image and the relationship matrix between the target component and the whole in the mask; Step 4: Take the known training set images, image-level labels, the mask pseudo-label set of the training set images, and the relationship matrix between the target component and the whole in the mask as the input of the component-aware learning segmentation network, train the component-aware learning segmentation network, and obtain the initial component-aware learning segmentation network model; Step 5: Use the initial component-aware learning segmentation network model to obtain the CAM set of the target mask in the training set image, and obtain the final pseudo-label set through threshold segmentation; use the final pseudo-label set as supervision information, and simultaneously train the initial component-aware learning segmentation network model and the segmentation decoder connected in series to the back end of the component branch network to obtain a complete component-aware learning segmentation network model, and finally input the image to be segmented into the complete component-aware learning segmentation network model to obtain the semantic segmentation result in the image to be segmented.

2. The method according to claim 1, characterized in that The backbone network is composed of ViT; the classification branch network includes fully connected layers and classifiers connected in sequence.

3. The method according to claim 1, characterized in that The working process of the component-whole refining module is as follows: Step A1: small object mask removal; Input is the SAM mask set of the training set image calculate The proportion of foreground pixels in each mask is U. If the proportion U is lower than the set threshold τ, the mask is removed to obtain the output mask set Among them, M i represents the i-th mask in the SAM mask set of the training set image, where M represents the SAM mask set, i is the index of the mask in the set; K is the total number of masks; represents the i-th mask in the output mask set, where represents the output mask set, i is the index of the mask in the set; is the number of masks in the output mask set; Step A2: high similarity mask removal; Input is the output mask set The overlap between masks is calculated, and masks with an overlap exceeding the set threshold β are judged as high-similarity masks and removed to obtain the pseudo-label set of the training set images. Among them, i,j are Different mask indices in , representing the i-th mask and the j-th mask; represents the i-th mask in the pseudo-label set of the training set images, is the total number of masks in the pseudo-label set of the training set images; and Respectively represent masks and mask The size of the intersection and union area of ​​the two masks; usually the number of pixels in the overlapping area of ​​the two masks and the total area covered are calculated respectively; f(i,j) is used to determine whether to keep the mask; if there is no other mask, and If the overlap exceeds the threshold β, then f(i,j) = 1, and the mask is retained. Otherwise f(i,j)=1 and remove the mask Step A3: record the relationship matrix between the target part and the whole in the mask; Take the pseudo-label set of training set images As input, construct the relationship matrix between the target parts and the whole in the mask of the pseudo label set; By calculating the overlap between masks and setting a threshold, we can determine which masks represent the relationship between the parts and the whole. The mask in and The overlapping part occupy When the ratio is greater than or equal to the threshold λ, As the overall mask, identify yes Component mask; using the indicator function h(i,j), establish the relationship matrix between the target and its components Where λ is the threshold value. If yes If the component mask is , then h(i,j)=1; otherwise h(i,j)=0; through h(i,j), we can get the relationship matrix between the target component and the whole in the mask 4. The method according to claim 3, characterized in that The working process of the component perception learning module is as follows: Step B1: extract embedding features; Embedding features of training set images extracted from the penultimate layer of the Transformer encoder By removing the embedded features Classify the label dimension and get the remaining embedding features Among them, B, N, and E represent the size of the input batch, the number of image blocks into which the image is divided, and the dimension of the feature vector of each image block, respectively. represents the field of real numbers; Step B2: Reshape the embedding feature scale; Embed features The shape of is rearranged from B×N×E to B×C×H×W; then, the mask pseudo-label set of the training set images output from the part-whole refinement module is combined with the embedded features Point product to get the embedded feature set Then embed the feature set The shape of is rearranged from B×k×C×H×W to B×k×N×E; and Each feature in is reduced to 1 / k by a 1×1 filter layer, and the mask feature set is output. Among them, H, W, and C represent the height, width, and feature dimension of the training set image respectively, and k represents the number of masks in the mask pseudo-label set; Step B3: mask feature fusion; Mask feature set Perform average pooling and normalization to obtain the weighted coefficient Based on the relationship matrix The corresponding overall mask and component mask are obtained to obtain the corresponding overall mask feature and component mask features By applying a shared 3×3 convolutional layer to each component mask feature, the component mask feature is updated, and all updated component mask features are added and fused with their corresponding overall mask features to obtain the fused overall mask feature, and then another shared 3×3 convolutional layer is used to update the fused overall mask feature to generate a new overall mask feature; Integrate all updated overall and component mask features to obtain a new mask feature set The updated mask feature set is updated by the weight coefficient W Then perform weighted concatenation to generate weighted mask features Among them, cat is the feature concatenation operation; W1, W2, W k represents the weighting coefficient, is the mask feature in the updated mask feature set; Step B4: output image features; Weighted mask features Use 1×1 convolution layer to get the features Then output features It is concatenated with the removed category token to obtain the final output feature.

5. The method according to claim 1, characterized in that The process of inputting the training set images into the SAM segmentation model to obtain the SAM mask set of the training set images is as follows: Taking the known training set images as input, the training images are first preprocessed, including image size adjustment, image grayscale and normalization, and color space conversion operations; then the preprocessed training set images are input into the pre-trained SAM segmentation model. The pre-trained SAM segmentation model automatically segments each training image, generates segmentation masks of multiple target areas, and obtains the SAM mask set of the training set images.

6. The method according to claim 1, characterized in that The process of taking known training set images, image-level labels, mask pseudo-label sets of training set images, and the relationship matrix between target components and the whole in the mask as inputs of the component-aware learning segmentation network, training the component-aware learning segmentation network, and obtaining the initial component-aware learning segmentation network model is as follows: Taking the input training set images as input, image-level labels as supervisory information, the mask pseudo-label set of the training set images and the relationship matrix between the target parts and the whole in the mask as auxiliary information, a joint loss function is used to train the component-aware learning segmentation network containing the component-aware learning module using a classifier to obtain a trained initial component-aware learning segmentation network model.

7. The method according to claim 6, characterized in that The joint loss function is composed of the classification cross entropy loss function, the auxiliary classification loss function and the contrast loss function, and the formula is: Among them, L all represents the joint loss function, λ1,λ2,λ3 are used to balance L cls , and L reg The regularization parameter of L cls Represents the classification cross entropy loss function: in, is the predicted probability of category c, y c is the true label; C represents the number of categories; Represents the auxiliary classification loss function: in, represents the predicted label obtained from the features of the component branch network, and softmax is the activation function; L reg Represents the contrast loss function: Among them, CosSim() represents the cosine similarity calculation, N + and N - They represent the number of positive sample pairs, i.e., samples with component relationships, and negative sample pairs, i.e., the number of samples other than positive samples. Represents the mask feature set elements.

8. The method according to claim 7, characterized in that The final pseudo-label set is used as supervision information to calculate the segmentation loss function. Based on the joint loss function plus the segmentation loss function, the initial component-aware learning segmentation network and the additional segmentation decoder connected in series to the back end of the component branch network are trained simultaneously to obtain a complete component-aware learning segmentation network model. Finally, the image to be segmented is input into the complete component-aware learning segmentation network model to infer and obtain the semantic segmentation result in the image to be segmented.

9. A weakly supervised semantic segmentation system based on a component-aware learning segmentation model, characterized in that: include: Network building module: used to build component-aware learning segmentation network; The component-aware learning segmentation network includes a component-whole refining module and a backbone network connected in series, as well as a classification branch network and a component branch network; the classification branch network and the component branch network are connected in parallel and then connected in series to the back end of the backbone network; The component branch network includes a component perception learning module, a Transformer Encoder module, a fully connected layer and a classifier connected in sequence; the component perception learning module includes an embedding feature extraction unit, an embedding feature reshaping unit, a mask feature fusion unit, and an image feature output unit connected in series in sequence; SAM mask set acquisition module: input the training set image into the SAM segmentation model to obtain the SAM mask set of the training set image; Part-whole refinement module: It includes a small object removal mask unit, a high similarity removal mask unit, and a relationship matrix unit for recording the target parts in the mask and the whole, which are connected in series in sequence. It takes the SAM mask set as input to obtain the mask pseudo-label set of the training set image and the relationship matrix between the target parts in the mask and the whole. Segmentation network training module: Use the known training set images, image-level labels, mask pseudo-label sets, and the relationship matrix between the target components and the whole in the mask to train the component-aware learning segmentation network and obtain the initial component-aware learning segmentation network model; Segmentation result optimization module: Use the initial component-aware learning segmentation network model to obtain the CAM set of the target mask in the training set image, and obtain the final pseudo-label set through threshold segmentation; use the final pseudo-label set as supervision information, and simultaneously train the initial component-aware learning segmentation network and the segmentation decoder connected in series to the back end of the component branch network to obtain a complete component-aware learning segmentation network model, input the image to be segmented into the complete component-aware learning segmentation network model, and obtain the semantic segmentation result in the image to be segmented.

10. A computer storage medium, characterized in that: A computer program is stored, which is called by a processor to implement: The steps of the method according to any one of claims 1 to 8.

Images