A weakly supervised instance segmentation method based on peak mining and filtering

Abstract

A weakly supervised instance segmentation method based on peak mining and filtering includes the following steps: 1) sample processing stage; 2) network configuration stage; 3) training stage; 4) test stage. The peak mining and filtering strategy designed in the application introduces feature fusion, adversarial erasing and clustering analysis to enhance the diversity and integrity of the peak response map, and through iterative retrieval and confidence update, more accurate segmentation masks are retrieved. Compared with existing weakly supervised instance segmentation methods, the segmentation algorithm of the application can realize more complete and accurate segmentation of objects of different sizes in the image, thereby effectively improving the accuracy of instance segmentation.

Description

Technical Field

[0001] This invention belongs to the field of computer software technology and relates to weakly supervised instance segmentation technology, specifically a weakly supervised instance segmentation method based on peak mining and filtering. Background Technology

[0002] In computer vision, conventional supervised learning requires lengthy annotation of datasets, which is extremely resource-intensive and time-consuming. Weakly supervised learning, on the other hand, reduces the resource consumption caused by excessive annotation by weakening the supervision of the labeled information. Instance segmentation aims to accurately segment all foreground objects in an image, predicting not only the semantic category of each pixel but also determining its individual affiliation. Therefore, the goal of weakly supervised instance segmentation is to achieve instance segmentation tasks that require pixel-level annotation by reducing the amount of labeled information.

[0003] For instance segmentation tasks, image-level category labeling, compared to pixel-level category labeling which only provides the semantic categories of objects in an image but not directly information such as object location, shape, and quantity, is beneficial for constructing large-scale datasets. Current weakly supervised instance segmentation algorithms based on image-level category labeling are mainly divided into two categories: detection-based and segmentation-based. For the former, the PRM (Peak Response Map) algorithm proposes the concept of peak activation for object localization and classification, and uses network gradients to calculate the probability dependence of peak responses in the original image. Then, it uses the peak response map to retrieve candidate masks as segmentation masks. The IAM algorithm uses encoding and decoding networks to learn the filling relationship between the peak response map and the segmentation mask to enhance segmentation quality and improve model testing speed. The CountSeg algorithm combines the supervision information of the number of objects in the image with peaks as supervision to obtain the density map of each object, which is beneficial for retrieving candidate masks. The Label-PEnet algorithm uses a course-based training strategy to allow the model to learn progressively from classification, detection, and segmentation, and uses later models to provide supervision for earlier models. For the latter type, the IRN (Inter-pixel Relation Network) algorithm first learns the correlation between pixels to achieve semantic segmentation, and uses prior knowledge that the sum of the distances from each point on the object to the center point is zero and that there are boundaries between points with different semantic categories to distinguish instances.

[0004] Currently, methods based on peak values ​​as proxy for object detection have advantages in accuracy, but they are still far behind fully supervised algorithms and still suffer from problems such as localization loss, classification errors, and local segmentation, which limit the improvement of the final segmentation accuracy. Summary of the Invention

[0005] The problem this invention aims to solve is: how to train a classification network solely through image-level category label supervision, enabling the model to more comprehensively detect all objects in an image and perform more accurate classification and segmentation; and how to more accurately filter out redundant and low-quality segmentation results. Typically, classification models respond to objects only to local salient regions, failing to comprehensively and completely respond to all objects (underactivation), and suffer from blurred segmentation boundaries and background co-occurrence (overactivation), severely impacting segmentation accuracy. Therefore, the design goal of this invention is to introduce mining and filtering strategies to address the aforementioned overactivation and underactivation problems respectively, ultimately achieving more accurate weakly supervised instance segmentation.

[0006] The technical solution of this invention is as follows: a weakly supervised instance segmentation method based on peak mining and filtering, which constructs an image classification network and an instance segmentation network, using image-level category labels as supervision for training. First, the image classification network is trained, and then the training data for the instance segmentation network is obtained from the image classification network for supervised training, thereby completing instance segmentation.

[0007] Image classification networks include the following configurations:

[0008] 1) Feature fusion: ResNet50 is used as the backbone network to extract image features. Feature maps from different layers and sizes in the network are fused to generate feature maps with richer representation and semantic information.

[0009] 2) Adversarial erasure: The feature map is reduced in dimensionality and transformed into a class activation map. For the salient regions in the class activation map corresponding to the ground truth class, the feature map is erased and filled with the feature mean. Then, a convolutional layer is used to reactivate the class in the filled feature map to expand the semantic response region. The two branches before and after erasure use two classifiers with different parameters for image recognition, thereby activating and outputting the two class activation maps before and after erasure.

[0010] 3) Peak activation: Peaks are used to locate and classify objects in the image, generating local salient peaks that represent independent objects. The most salient peak in the category activation map of a certain semantic category indicates that there is an object of that category at that location. Local peaks are obtained through local max pooling and supervised by semantic annotation information. Max pooling layers are used to perform pooling operations on the category activation maps obtained before and after erasure in 2), respectively, to obtain a list of peaks from the branch activations before and after erasure.

[0011] 4) Filtering module: Add a classifier of a different type than the branches before and after erasure and the peak activation layer next to the feature map obtained in 1) to independently determine the semantic category of the object;

[0012] The instance segmentation network includes an image classification network. Based on the image classification network, cluster analysis and iterative retrieval are performed on the branch responses before and after erasure. Then, a filtering module is used to filter out segmentation masks that do not meet the confidence requirements, resulting in instance segmentation results. Specifically, after training the image classification network on the training set, the network uses the network gradient to calculate the probability dependence of the test image containing the set of classless candidate segmentation masks to be segmented, obtaining the peak response map of the peak in the original image. Cluster analysis is then performed based on the depth features of the peak, merging different peak response maps from the same object. During instance segmentation, the peak response map is used to iteratively retrieve the set of classless candidate segmentation masks corresponding to the original image, and the best match is selected as the segmentation mask for that object. At the same time, the confidence is updated by combining the object's category information and shape information obtained from the filtering module to filter out low-quality segmentation masks. Finally, the above method is used to perform instance segmentation on the training set images of the training image classification network, generating a dataset with pseudo-pixel annotations for supervised training of the fully supervised instance segmentation algorithm.

[0013] The present invention has the following advantages compared with the prior art.

[0014] This invention proposes a weakly supervised instance segmentation method based on peak mining and filtering (PMF). This method achieves more comprehensive and complete segmentation of objects in an image through peak mining and employs filtering to suppress the influence of low-quality segmentation masks, thereby significantly improving segmentation accuracy.

[0015] This invention designs a weakly supervised instance segmentation method based on peak mining and filtering. By mining more non-salient object regions, and employing filtering to integrate semantic and shape information of objects to generate a more accurate segmentation mask, it also generates a more refined pixel-level labeled dataset for fully supervised instance segmentation algorithms. Compared to existing weakly supervised instance segmentation methods, this invention demonstrates excellent segmentation capabilities for objects of different scales and categories, effectively improving segmentation accuracy.

[0016] This invention achieves good accuracy on weakly supervised instance segmentation tasks with image-level category annotation. Compared to existing methods, the PMF weakly supervised instance segmentation method proposed in this invention demonstrates good performance on recognized datasets and can even be compared with models trained with stronger supervision information. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the weakly supervised instance segmentation method based on peak mining and filtering of the present invention.

[0018] Figure 2This is a schematic diagram of feature fusion based on ResNet50 in the network structure of this invention.

[0019] Figure 3 This is a schematic diagram of the anti-erasure structure in the network structure of this invention.

[0020] Figure 4 This is a schematic diagram of the process of peak feature clustering analysis based on the present invention.

[0021] Figure 5 This is a schematic diagram of the iterative retrieval and segmentation mask process of the present invention.

[0022] Figure 6 This is a schematic diagram of the process for filtering low-quality segmentation masks according to the present invention.

[0023] Figure 7 This is a sample image of instance segmentation in a weakly supervised training scenario according to the present invention. Detailed Implementation

[0024] To address the problems of existing technologies, the goal of this invention is to enable classification networks to respond more comprehensively and completely to objects in images and filter out redundant and coarse segmentation masks, thereby improving segmentation accuracy. Specifically, this invention proposes a weakly supervised instance segmentation method based on peak mining and filtering, named PMF (Peak Mining and Filtering). Furthermore, this invention proposes the idea of ​​providing higher-precision pixel-level annotations for fully supervised instance segmentation algorithms, thereby achieving more accurate object segmentation.

[0025] The implementation and effects of the present invention will be illustrated below through specific embodiments.

[0026] This invention presents a weakly supervised instance segmentation method based on peak mining and filtering. The method trains a classification network on the PASCAL VOC 2012 dataset using a training set with only image-level category annotations. Performance evaluation of the instance segmentation task is then conducted on the same dataset with pixel-level annotations on both the training and validation sets, achieving relatively accurate segmentation. Specifically, the method utilizes Python 3.6 and the PyTorch 0.4 deep learning framework.

[0027] Figure 1This is a flowchart of the weakly supervised instance segmentation method based on peak mining and filtering used in this invention. This invention uses a ResNet50 residual network as the backbone network and achieves pixel-level instance segmentation using only image-level category annotations. The method is trained twice: once for the classification network and once for the instance segmentation network based on a newly generated dataset. This invention uses locally salient peaks and corresponding peak response maps to represent objects, and achieves more comprehensive, complete, and accurate object segmentation in images through peak mining and filtering. The entire method includes a sample processing stage, a network configuration stage, a training stage, and a testing stage. The specific implementation steps are as follows:

[0028] 1) In the sample processing stage, the training samples in this invention are RGB images with only image-level category annotations. The image size is converted to 448*448 as the network input, and a horizontal flipping data augmentation technique with a probability of 0.5 is used. In the testing stage, the samples are only RGB images, and the network input size is 448*448. A multiscale combinatorial grouping (MCG) algorithm is used to generate a set of candidate segmentation masks for each test image, and the top 100 with the highest scores are selected.

[0029] 2) In the network configuration stage, the peak mining and filtering strategy of this invention trains only one classification network, which mainly includes the following four parts: feature fusion, adversarial erasure, peak activation and filtering module.

[0030] 2.1) Feature fusion, such as Figure 2 As shown, a ResNet50 network with the last fully connected layer removed is used as the backbone network for image feature extraction. The ResNet50 consists of layer 0 (1 convolutional layer), layer 1 (9 convolutional layers), layer 2 (12 convolutional layers), layer 3 (18 convolutional layers), and layer 4 (9 convolutional layers). The network input is B*3*448*448, where B represents the batch size, and the feature map size output by each layer is F0∈R. B×64×224×224 F1∈R B ×256×112×112 F2∈R B×512×56×56 F3∈R B×1024×28×28 F4∈R B×2048×14×14 This invention fuses features from layer 3 and layer 4. Specifically, it uses a convolutional layer with a kernel size of 1*1 and a stride of 1 to activate the feature map from layer 3, increasing the number of channels from 1024 to 2048. Bilinear interpolation is then used to upsample the feature map output from layer 4. Finally, the two layers are added together to obtain the final feature map F. last ∈R B×2048×28×28 .

[0031] 2.2) Adversarial erasure: For each input sample, under the supervision of the same class label, image recognition is performed using two classifier structures with different parameters. Specifically, as follows... Figure 3 As shown, for the feature map F obtained after feature fusion in 2.1), last ∈R B×2048×28×28 First, erase the preceding branch, denoted as Branch. A A convolutional layer with a kernel size of 1*1 and a stride of 1 is used for activation, and the number of channels in the feature map is transformed from 2048 to the number of semantic categories C. The output of the network is called the category activation map, denoted as M. A Its output size is M A ∈R B×C×28×28 Each channel layer represents the response of the image to that semantic category. This invention transforms semantic categories into numerical labels, such as 20 foreground categories, represented by a 1*20 vector. A value of 1 corresponds to the index of a category, while a value of 0 corresponds to an index of an undefined category. During network training, M is selected based on the ground truth category of the image. A The corresponding number of channels and layers are used to generate a foreground saliency map, denoted as M. s Its output size is M s ∈R B×1×28×28 The value space is {0,1}, where 1 represents the foreground. This invention first activates the i-th layer category map. Normalization is performed by dividing each element in the graph by the difference between the largest and smallest elements. Then, a threshold of 0.6 is selected for filtering to obtain the saliency map for the corresponding category. During training, this invention merges the saliency maps corresponding to different ground truth categories (predicted categories during testing) in the same image to obtain a foreground saliency map for the entire image. Then, based on the foreground saliency map M... s Erase feature map F last The feature map obtained after erasing the corresponding region is denoted as F. erase For F last Only keep M s The region with a median of 0, while M s The region with a median value of 1 is filled using different methods, such as zero, maximum / minimum, and mean values. In this invention, F is used. last M corresponds to s The mean of the regions with values ​​of 0 is used to fill the gaps. Finally, after erasing, the branch is denoted as Branch. B The activation process yields a category activation diagram, denoted as M. B Specifically, a convolutional layer with a kernel size of 1*1 and a stride of 1 is used to convert F... erase The number of channels is converted from 2048 to the number of semantic categories C. The branch network structure is the same before and after erasure, but they do not share parameters, and the network parameters are all randomly initialized.

[0032] 2.3) Peak activation: This invention uses peak values ​​to locate and classify objects in an image. The peak value is the maximum value in the sampling window, representing the most salient local location. Specifically, a Maxpooling layer is used to activate the category activation maps M obtained before and after erasure in 2.2). A and M B Performing a pooling operation yields a list P of peak values ​​from branch activations before and after erasure. A and P B The pooling window size is 3*3. The specific calculation process is as follows:

[0033]

[0034] Where G represents the class activation map after peak activation, c, x, and y represent the class and the region accessible by the kernel during pooling operations, respectively, f represents the max pooling function, and N... c This represents the number of peaks generated by category c. Peak list P A and P B It is a list of peaks obtained by performing peak activation before and after erasure. Each peak includes the peak plane position (x, y) on the class activation map before and after erasure, the semantic class (c), and the numerical value.

[0035] The network not only stores the pooled values ​​of the category activation maps, i.e., the peak values, but also retains their corresponding positions, specifically represented by five parameters: s, b, c, h, and w. s represents the value corresponding to the semantic category, b represents the image index in the batch, c represents the semantic category, h represents the ordinate of the position in the feature map plane, and w represents the ordinate of the position in the feature map plane. The network filters the generated list of peak values, specifically using the median of each category for each image to select the peak values ​​generated in the feature map plane. To effectively combine image-level category annotation supervision, the mean value of the peak values ​​obtained from filtering each category in the feature map plane for each image before and after erasure is calculated to obtain the classification output S. A and S B The size is B*C, where B represents the batch size and C represents the number of semantic categories. The classification output is S. A and S B This represents the classification score of each image before and after erasing.

[0036] 2.4) Filtering module, with a classifier of a different type added alongside it. For example... Figure 6As shown, for the feature map output by layer 4 mentioned in 2.1), firstly, Average Pooling is used for pooling. Then, the pooled feature map is flattened, that is, its two-dimensional plane is transformed into a one-dimensional vector. Subsequently, a fully connected layer is used for activation with randomly initialized parameters, and a Softmax layer is used to calculate the probability S of each class. C This is used as the output of the classifier, with an output size of B*C, where C represents the number of semantic categories.

[0037] The following example illustrates the network configuration phase. Using the ResNet50 architecture (excluding the FC layer) as the backbone, the network is randomly initialized. Feature extraction is then performed on the input image, passing through layers 0, 1, 2, 3, and 4 to obtain feature maps of sizes B*64*224*224, B*128*112*112, B*512*56*56, B*1024*28*28, and B*2048*14*14, respectively. A 1x1 convolutional layer is used to increase the number of channels in the layer 3 feature map to 2048, while bilinear interpolation is used to upsample the layer 4 feature map to a planar size of 28x28. The two feature maps are then summed. Both the network branches before and after erasure use a 1x1 convolutional layer to generate class activation maps, converting the number of channels into the number of semantic classes. Based on the class activation maps obtained before erasure, the corresponding foreground salient regions M are extracted according to the class labels. s ∈R B×1×28×28 According to M s The mean of the erased foreground region feature map and the insignificant regions are used as the input features for the erased branch. The two branches generate class activation maps M. A M B ∈R B×C×28×28 All peak values ​​were obtained through a 3x3 max pooling layer and then filtered by median value according to category, denoted as P. A and P B Then, the mean value of the retained peak values ​​is calculated by category to obtain S. A S B ∈R B×C Meanwhile, for the layer4 feature map, the probability S of classifying the image into each category is obtained through mean pooling, fully connected layers, and the softmax function. C ∈R B×C S A S B S C The loss relative to the true class label is calculated using the loss function MultiLabelSoftMarginLoss.

[0038] 3) During the training phase, supervised training is performed on the image classification network with three different classifiers on the target dataset (i.e., the training set). The three classifiers are different branches, each representing a score for image classification. S... A and S B S represents the classification score for each category before and after erasure, but the input image features are derived from both before and after erasure. C The image features before erasure are input into the classifier, and the output is the probability of the image being classified into each category. This invention uses a training set of 11,540 images with only image-level category labels from the PASCAL VOC dataset for model training. The backbone network model is pre-trained on ImageNet. The batch size is 16 images. The backpropagation algorithm is used to update the network parameters, specifically using a stochastic gradient descent optimizer with momentum set to 0.9, a learning rate set to 0.01, and a weight decay set to 0.0001. The branches before and after erasure, as well as the filtering module, all use MultiLabelSoftMarginLoss as the loss function. Training is performed using only a single GTX1080Ti graphics card, with a total of 35 training epochs.

[0039] 4) During the testing phase, this invention requires post-processing to generate a final clean, complete, and accurate segmentation mask. For the image to be segmented, after passing through the trained image classification network, this invention also requires several steps in the testing phase, combined with S... C The final segmentation result is obtained by filtering the segmentation mask. The testing phase includes four simple steps: peak response, cluster analysis, iterative retrieval, and filtering strategy.

[0040] 4.1) Peak Response: The peak response is the corresponding region on the input image obtained by backpropagating the probability dependencies between layers in the network through gradient calculation, which is called the peak response map. Peaks can only locate and classify objects. To achieve complete object segmentation, this invention calculates the response region on the input image based on the category and position of the peak in the category activation map. Specifically, for each peak P... i Calculate its gradient in the network to compute the probability dependencies between layers, and gradually backpropagate from the class activation map to the response region corresponding to the original input map, thus forming the peak activation map R. i Its size is 1*448*448, the pixel value space is [0,1], and the sum of all elements is 1. For simplicity, consider a common convolutional layer, with the convolution kernel denoted as W∈R. h×w h and w represent the height and width of the convolution kernel. For a classification network model that converges using peak activation training, the specific calculation process is as follows:

[0041]

[0042] In the image classification network, the input and output of the convolutional layer correspond to U and V, respectively, where i,j and p,q represent the positions on the input and output feature map planes, and P is the probability of each position. In the above formula (U... ij |V pq The expression represents the probability that the output at position pq of the convolutional layer yields the input at position ij. The specific calculation process is as follows:

[0043]

[0044] in Z represents the activation at position ij of the network's forward propagation input U, while W+=ReLU(W) means using the ReLU activation function to preserve the non-negative values ​​of the convolution kernel parameters, i.e., only calculating the positive activation, and Z pq It is used to guarantee ∑ p,q P(U ij |V pq The regularization factor is 1. In the specific calculation of the peak response map, this invention replaces all convolutional layers in the network with the above calculation process.

[0045] 4.2) Cluster analysis: For the peak response plot calculation process described in 4.1), the peak list P generated before and after erasure is used. A and P B This involves calculating the peak response map from each peak in the peak list. First, a simple filtering of the peaks is performed using a threshold of 20. Here, the threshold represents the activation level of a certain category at that location. The probability of each peak in the list is calculated through backpropagation to obtain the peak response map list R before and after erasure. A and R B This invention performs cluster analysis on peak values, grouping peak values ​​from the same object based on feature similarity. The specific process is as follows: Figure 4 As shown, feature vectors at peak positions are extracted from the feature map plane after feature fusion, and a spectral clustering algorithm based on k-means optimization is used to implement the clustering process. The feature vectors corresponding to the peaks obtained from the activation of the branch before erasure are used as the initial cluster centers to optimize the direction of subsequent clustering iterations. The number of clusters is set to the number of peaks in the branch before erasure. The clustering objects are the feature vectors corresponding to the peaks of the two branches before and after erasure.

[0046] The spectral clustering algorithm first calculates the similarity matrix W∈R between peak vectors. n×n Then, calculate the sum of each row of W, i.e., the degree matrix D. Next, calculate the Laplace matrix according to L = DW. Then, calculate the eigenvalues ​​of L, sort the eigenvalues ​​in ascending order, and take the first k eigenvalues ​​to form a new matrix X∈R. n×kThe input to the k-means clustering algorithm is used as input, and iteration stops when the results of the previous and next iterations no longer change. The process outputs the cluster label corresponding to each peak.

[0047] After clustering, the peak response maps corresponding to the peak values ​​in each cluster are merged and renormalized (i.e., the value of each pixel in the map is divided by the sum of the values ​​of all pixels). The resulting list of more complete peak response maps after clustering is denoted as R. merge With R A R B Merging is performed to increase the diversity of peak response maps.

[0048] 4.3) Iterative retrieval: Each peak response map represents the response of an object in the image. However, overactivation and underactivation are common. Therefore, this invention utilizes the peak response map to retrieve the best match from the set of classless candidate segmentation masks generated by the MCG algorithm as the segmentation mask for the corresponding object. The matching degree is calculated as follows:

[0049]

[0050] Among them, Score ij Peak response plot R i With classless candidate segmentation mask S j The degree of matching between them Indicates the segmentation mask S j The edge of Q, and Q i The semantically salient regions corresponding to the peak response map categories are represented by normalizing a certain channel plane of the category activation map and then filtering it with a threshold of 0.5. α and β are hyperparameters, with values ​​set to 0.73 and 1.9e-5, respectively. Based on the above retrieval process, the peak response map set R = {R...} A ,R B Retrieve the corresponding segmentation mask.

[0051] The specific iterative retrieval process is as follows: Figure 5 As shown, this invention employs the Non-Maximum Suppression (NMS) algorithm, first dividing the segmentation mask set according to the matching score. ij Sort the segmentation masks in descending order of size, and continuously select the largest item in the current segmentation mask list. Filter the remaining segmentation masks in the list according to the criterion of the same category and the Intersection over Union (IoU) ≥ 0.5. Simultaneously, fuse the corresponding peak response maps to obtain a new peak response map list R. merge1 and R merge Merge R = {R merge ,R merge1Next, R is used to retrieve a set of candidate segmentation masks, and NMS is used to filter the segmentation masks and fuse the peak response maps to obtain a new list of peak response maps R. merge2 Finally, R is used. merge2 The candidate segmentation mask set is retrieved, filtered by NMS, and the final segmentation mask set is used as the instance segmentation result of the image.

[0052] 4.4) Filtering strategy: The segmentation results obtained in step 4.3) above include the segmentation mask S, the category C, and the matching score between the peak response map and the segmentation mask. match As a confidence level, this invention generates a category confidence level for the corresponding object based on this, in order to update the final segmentation confidence level. Specifically, as follows... Figure 6 As shown, the input original image is cropped using a segmentation mask and then passed through the network configured in step 2) for forward propagation. Simultaneously, the classifier, added alongside the filtering module, outputs the class corresponding to the maximum probability value of its final Softmax function. If the output class differs from the previous class, the current output class is used as the correction. The probability is the class confidence score. class This invention updates the final segmentation confidence using the following function:

[0053] Score = γ·Score match +(1-γ)·Score class ,

[0054] γ is the balance coefficient, which effectively integrates the shape and category information of the object through linear combination. In the experiment, the value was set to 0.45, and a threshold of 0.2 was used for filtering.

[0055] The following example illustrates the testing phase. The complete testing process is as follows: Figure 1 As shown, after the input image undergoes sample processing in the testing phase, the processed image and the corresponding segmentation mask set are obtained. Then, in step 2), the network configuration phase outputs a list P of peak activations of branches before and after erasure. A and P B The peak response plot list R is obtained through the peak response in step 4.1). A R B Then, based on P... A and P B The location of the mid-peak is used to extract feature vectors from the feature map, and a spectral clustering algorithm is employed with P as the eigenvector. A The peak response maps are used as initial cluster centers for clustering, and then the peak response maps from the same cluster are merged to obtain a new list of peak response maps R. mergeSubsequently, in step 4.3), the peak response map is used to retrieve the best match from the set of classless candidate segmentation masks as the object segmentation result. Based on the prior assumption that segmentation masks with high overlap in the same class correspond to the same object, NMS is used iteratively to filter and fuse the peak response maps. Finally, the original image is cropped based on the segmentation mask, reclassified, and the segmentation confidence is updated by combining the matching confidence and class confidence, and the segmentation result is filtered. Figure 7 This invention provides an example of instance segmentation in the mining and filtering steps of this invention. Furthermore, this invention performs instance segmentation on the dataset used for training the classification network, generating a dataset with pseudo-pixel annotations for supervised training of the Mask R-CNN algorithm. This approach enables more accurate instance segmentation.

[0056] This invention evaluates segmentation accuracy on training and validation sets of 1464 and 1449 images with pixel-level annotations, respectively, in the PASCAL VOC dataset. On the training set, the mAP@25, mAP@50, mAP@75, and ABO metrics reached 53.2%, 34.9%, 12.9%, and 43.3%, respectively. On the validation set, the mAP@25, mAP@50, mAP@75, and ABO metrics reached 52.1%, 32.9%, 14.7%, and 43.4%, respectively. Specifically, the segmentation mask regenerated from the training set samples used for classification using the method of this invention was used as pseudo-pixel annotations to supervise the training of the instance segmentation algorithm Mask R-CNN with a ResNet 50-FPN backbone network. This algorithm achieved mAP@50 and mAP@75 metrics of 45.6% and 21.2%, respectively. Compared to current weakly supervised instance segmentation algorithms, the method of this invention also demonstrates better segmentation accuracy.

Claims

1. A weakly supervised instance segmentation method based on peak mining and filtering, characterized in that An image classification network and an instance segmentation network are constructed. Image-level category labels are used as supervision for training. First, the image classification network is trained. Then, the training data for the instance segmentation network is obtained from the image classification network for supervised training, thereby completing instance segmentation. Image classification networks include the following configurations: 1) Feature fusion: ResNet50 with the last fully connected layer removed is used as the backbone network for image feature extraction. ResNet50 consists of layer 0 with 1 convolution, layer 1 with 9 convolutions, layer 2 with 12 convolutions, layer 3 with 18 convolutions, and layer 4 with 9 convolutions. Features from layer 3 and layer 4 are fused. A convolutional layer with a kernel size of 1*1 and a stride of 1 is used to activate the feature map from layer 3, which increases the number of channels from 1024 to 2048. Bilinear interpolation is used to upsample the feature map output from layer 4, and then the two are added together to obtain the final feature map. 2) Adversarial erasure: The feature map is reduced in dimensionality and transformed into a class activation map. For salient regions in the class activation map corresponding to the ground truth class, erasure is performed on the feature map, and the feature mean is used for padding. Then, a convolutional layer is used to reactivate the class in the padded feature map to expand the semantic response region. Specifically, for each input sample, under the supervision of the same class label, two classifier structures with different parameters are used for image recognition. For the feature map obtained after feature fusion... The branch before erasing is denoted as A convolutional layer with a kernel size of 1*1 and a stride of 1 is used for activation, transforming the number of channels in the feature map from 2048 to the number of semantic categories C. The output is called the category activation graph, denoted as The semantic categories are converted into numerical labels. For N foreground categories, a 1*N vector is used to represent them. If the image contains a category, the corresponding index position has a value of 1; otherwise, the value is 0. During supervised training, the ground truth category of the image is selected. The corresponding number of channels is layered, and a foreground saliency map is generated, denoted as... First, the activation graph of the i-th layer category is... Normalization is performed by dividing each element in the graph by the difference between the maximum and minimum elements. Then, a threshold of 0.6 is selected for filtering to obtain the saliency map for the corresponding category. During training, saliency maps corresponding to different ground truth classes in the same image are merged to obtain the foreground saliency map of the entire image. Then, based on the foreground saliency map... Erasure feature map The corresponding region is used to obtain the erased feature map, denoted as . ,for Only keep The region with a median value of 0, and For regions with a median value of 1, use Chinese correspondence The mean of the region with an upper value of 0 is used to fill the gap, and the branch after erasure is denoted as . The activation process yields a category activation diagram, denoted as follows: ,Will The number of channels is changed from 2048 to the number of semantic categories C. The branch network structure is the same before and after erasure, but the parameters are not shared, and the network parameters are all randomly initialized. 3) Peak activation: Peaks are used to locate and classify objects in the image, generating local salient peaks that represent independent objects. The most salient peak in the category activation map of a certain semantic category indicates that there is an object of that category at that location. Local peaks are obtained through local max pooling and supervised by semantic annotation information. Max pooling layers are used to perform pooling operations on the category activation maps obtained before and after erasure in 2), respectively, to obtain a list of peaks from the branch activations before and after erasure. 4) Filtering module: Add a classifier of a different type than the branches before and after erasure and the peak activation layer next to the feature map obtained in 1) to independently determine the semantic category of the object; The instance segmentation network includes an image classification network. Based on the image classification network, clustering analysis and iterative retrieval are performed on the branch responses before and after erasure. A filtering module then filters out segmentation masks that do not meet the confidence requirements, yielding the instance segmentation result. Specifically, after training the image classification network on the training set, the network calculates the probability dependence of the test image containing the set of classless candidate segmentation masks for instance segmentation, using the network gradient to obtain the peak response map in the original image. Specifically, the response region in the input image is calculated based on the class and position of the peak in the class activation map. For each peak... Calculate its gradient in the network to compute the probability dependencies between layers, and gradually backpropagate from the class activation map to the response region corresponding to the original input map, thus forming the peak activation map. For a classification network model that converges using peak activation training, the calculation process is as follows: , In the image classification network, the input and output of the convolutional layer correspond to U and V, respectively, where i, j and p, q represent the positions on the input and output feature map planes, and P is the probability of each position. and Indicates the height and width of the convolution kernel. This represents the probability that the output at position pq of the convolutional layer yields the input at position ij. The specific calculation process is as follows: , wherein denotes the activation of the network forward propagation input U at position ij, denotes the ReLU activation function is used to reserve non-negative values for the convolution kernel parameters, i.e. only the positive activations are calculated, for ensuring a regularization factor; in the specific calculation of the peak response map, all the convolution layers in the network are replaced by the above calculation process; Cluster analysis is performed based on the depth features of the peak values, merging different peak response maps from the same object. The resulting list of peak response maps after clustering is denoted as... During instance segmentation, the peak response map is used to iteratively retrieve the set of classless candidate segmentation masks corresponding to the original image, and the best match is selected as the segmentation mask for the object. Simultaneously, the confidence score is updated by combining the object's category and shape information obtained from the filtering module to filter out low-quality segmentation masks. Specifically: The best matching item is retrieved from the set of classless candidate segmentation masks generated by the MCG algorithm using the peak response map, and is used as the segmentation mask for the corresponding object. The matching degree is calculated as follows: , in, Peak response plot With classless candidate segmentation mask The degree of matching between them Represents segmentation mask The edge, This represents the semantically salient region corresponding to the peak response map category. It is obtained by normalizing a certain channel plane of the category activation map and then filtering it with a threshold of 0.

5. and These are hyperparameters, and their values ​​are set to 0.73 and 1.9e-5 respectively; The iterative retrieval process employs a non-maximum suppression algorithm. First, the segmentation mask set is divided according to the matching score. Sort the segmentation masks in descending order of size, and continuously select the largest item in the current segmentation mask list. Then, for the remaining segmentation masks in the list, sort them according to the same category and the intersection-union ratio between them. The peak response maps are filtered according to the standard and then merged with the corresponding peak response maps to obtain a new list of peak response maps. and with merge Then use A candidate segmentation mask set is retrieved, and the segmentation masks are filtered using NMS and the peak response maps are fused to obtain a new list of peak response maps. Finally utilize The candidate segmentation mask set is retrieved, filtered by NMS, and the final segmentation mask set is used as the instance segmentation result of the image. The obtained segmentation results include segmentation masks. ,category And the matching score between the peak response map and the segmentation mask. The process generates category confidence scores for corresponding objects to update the final segmentation confidence score: the input original image is cropped using a segmentation mask and passed through the forward propagation of the image classification network. Simultaneously, a classifier added alongside the filtering module outputs the category corresponding to the highest probability value. If the category differs from the previous category, the current output category is used as a correction; the probability is the category confidence score. The final segmentation confidence is updated using the following function: , wherein is a balance coefficient, which effectively integrates the shape information and category information of the object by linear combination, and is set to 0.45 in the experiment, and a threshold value of 0.2 is used for filtering; Finally, the above method is used to perform instance segmentation on the training set images of the training image classification network to generate a dataset with pseudo-pixel annotations, which is used for supervised training of the fully supervised instance segmentation algorithm.

2. The weakly supervised instance segmentation method based on peak mining and filtering according to claim 1, characterized in that Includes the following steps: 1) In the sample processing stage, the training samples include RGB images and their corresponding class labels, where the input images are horizontally flipped for data augmentation. The test samples include RGB images and a set of classless candidate segmentation masks generated for each image using the MCG algorithm. 2) In the network configuration phase, using RGB images as input and image-level category labels as supervision, the image classification network includes the following configuration: 2.1) Feature fusion, using ResNet50 as the backbone network to extract image features, Resnet50 includes layer0, layer1, layer2, layer3 and layer4, fusing features from layer3 and layer4 to obtain a fused feature map ; 2.2) Anti-erase, reduce the dimension of feature map and transform it into class activation map, erase the significant region in the class activation map corresponding to the true value class on the feature map and fill it with feature mean, output two class activation maps before and after erasing and ; 2.3) Peak activation, using Maxpooling layer to get the peak list from the class activation map of 2.2) before and after erasing and performing pooling operation to get the peak list from the branch activation before and after erasing and The window size of the pooling operation is 3*3, and the specific calculation process is as follows: = wherein G represents a class activation map after peak activation, c, x, y respectively represent a class and a region accessible by a kernel in a pooling operation, represents a max-pooling function, represents a number of peaks generated by the cth class. The pooled values ​​of the category activation maps are saved as peak values, and their corresponding positions are preserved. Five parameters, s, b, c, h, and w, represent the peak values: s represents the value of the corresponding semantic category, b represents the image index in the batch, c represents the semantic category, h represents the ordinate of the position in the feature map plane, and w represents the ordinate of the position in the feature map plane. The generated peak value list is filtered by using the median of each category for each image to select the peak values ​​generated in the feature map plane. Specifically, the mean of the peak values ​​obtained from filtering each category in the feature map plane for each image from both the before and after erasure branches is calculated to obtain the classification output. and ; 2.4) Filter module, for the feature map output in 2.1), first adopt Average Pooling to pool, then flat the pooled feature map to convert its two-dimensional plane to a one-dimensional vector, then adopt a fully connected layer to activate and parameter random initialization, at the same time, adopt Softmax layer to calculate the probability of each category and as the output of the classifier ; 3) Training phase, for the above network configuration, using the pre-trained backbone network on the Imagenet dataset and the response parameters, the classification network with three different classifiers is supervised trained on the target dataset; in the training, , , respectively via the loss function MultiLabelSoftMarginLoss to calculate the loss relative to the true class label. 4) During the testing phase, for the image to be segmented, the converged model trained above is used to generate segmentation masks for the objects in the image, including the following process: 4.1) Peak response: For the set of peaks generated by the network branches before and after erasure, the response region that generated the peak is obtained in the original graph by calculating the probability dependence of the gradient. 4.2) Cluster analysis: Cluster analysis is performed based on the depth features of peak locations to merge different peak response maps from the same object to increase its diversity and integrity; 4.3) Iterative retrieval: Based on the principle that different segmentation masks of the same category with a high intersection-union ratio (IUU) are likely to correspond to the same object, the best matching item is retrieved from the set of candidate segmentation masks without category using the peak response map iteratively as the segmentation mask. 4.4) Filtering strategy: Combining the matching score between the peak response map and the segmentation mask in 4.3), and the classification score of the segmentation mask region by the classifier added in addition to the network branches before and after erasure, the final confidence is updated, and segmentation masks with low confidence are filtered to obtain the final instance segmentation result.

Images