A semi-supervised instance segmentation method based on feature transfer

Abstract

The application discloses a semi-supervised instance segmentation method based on feature migration. The method is based on a Mask R-CNN network, and a feature migration module based on an inverse convolutional neural network is constructed, so that image visual features learned by a Mask R-CNN target detection branch are converted into features used for segmentation prediction. In order to solve the problem that the feature resolution output by the Mask R-CNN target detection branch does not match the feature resolution of a segmentation prediction branch, an inverse convolutional network layer is used in the feature migration module of the method to perform up-sampling on input features. Through the feature migration process, the semi-supervised instance segmentation precision of a Mask R-CNN benchmark model can be effectively improved.

Description

Technical Field

[0001] This invention relates to the field of image segmentation technology, and more particularly to instance segmentation in image segmentation, specifically a semi-supervised instance segmentation method based on feature transfer. Background Technology

[0002] Instance segmentation is a key technology in research fields such as autonomous driving, biomedical image processing, and robot vision control. It requires the simultaneous localization, classification, and region segmentation of specified target instances within an image. Instance segmentation is a popular research area with great application potential, aiming to improve people's daily lives, and has many important applications in real-world scenarios and technological research. Examples include segmenting aerial images of urban landscapes for sports venue improvement, segmenting microscopic images in biomedicine, using machine vision to control robots to grasp objects, and segmenting in-vehicle images for autonomous driving to distinguish pedestrians, roads, and vehicles.

[0003] In recent years, deep learning has achieved groundbreaking progress in many image processing tasks by leveraging its advantage of automated feature extraction. Regarding instance segmentation algorithms, supervised learning-based segmentation models, such as Mask R-CNN (HeK, Gkioxari G, Dollár P, et al. Mask r-cnn[C] / / Proceedings of the IEEE international conference on computer vision. 2017: 2961-2969.) and MaskLab (ChenL C, Hermans A, Papandreou G, et al. Masklab: Instance segmentation by refining object detection with semantic and direction features[C] / / Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018: 4013-4022.), have achieved significant segmentation results on public datasets such as COCO and Cityscapes. Supervised learning instance segmentation models require image datasets with complete instance segmentation annotations to achieve good segmentation results. However, collecting instance segmentation labels requires significant manpower and time. For example, on average, it takes about 79 seconds for a single person to annotate the segmentation label for an object on the COCO dataset, and additional personnel are needed to check the segmentation labels afterward. The segmentation annotation time on the COCO dataset is about 15 times that of object bounding box annotation.

[0004] To address the challenge of collecting instance segmentation annotations, some studies have attempted to combine "weak" (relatively easy to collect) labels such as object bounding boxes, image categories, and the number of objects to achieve weakly supervised instance segmentation. While the performance of weakly supervised algorithms has improved in recent years, the best current weakly supervised instance segmentation algorithms, such as IRNet (Laradji IH, Vazquez D, Schmidt M. Where are the Masks: InstanceSegmentation with Image-level Supervision[J]. arXiv preprint arXiv:1907.01430, 2019.), still lag significantly behind fully supervised instance segmentation algorithms (such as Mask R-CNN).

[0005] In recent years, semi-supervised instance segmentation algorithms have combined the advantages of fully supervised and weakly supervised algorithms. They utilize semi-supervised instance segmentation datasets (datasets where some image data has weakly supervised labels or no labels at all, while the remaining image data has complete instance segmentation labels). This reduces the difficulty of dataset labeling while improving the prediction accuracy of the instance segmentation algorithm model. Semi-supervised instance segmentation algorithms are divided into two categories: 1) algorithms based on pseudo-segmentation label generation; and 2) algorithms based on transfer learning. Algorithms based on pseudo-segmentation label generation, such as the one proposed by Li et al. (Li Q, Arnab A, Torr PH S. Weakly-and semi-supervised panoptic segmentation[C] / / Proceedings of the European Conference on Computer Vision (ECCV). 2018: 102-118.) and the one proposed by Bellver et al. (Bellver M, Salvador A, Torrres J, et al. Budget-aware semi-supervised semantic and instance segmentation[J]. arXiv preprint arXiv:1905.05880, 2019.), require special procedures to generate instance segmentation pseudo-labels, and they are only designed for certain types of "weak labels." Therefore, the scalability of these algorithms for image datasets with different weak label types is not high. Algorithms based on transfer learning, such as the one proposed by Hu et al. (Hu R, Dollár P, He K, et al. Learning tosegment every thing[C] / / Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018: 4233-4241.), can "transfer" knowledge learned by the model from different source tasks to improve the accuracy of instance segmentation prediction, and the algorithm has good scalability. However, due to the relatively late start of research, the design ideas of semi-supervised instance segmentation algorithms based on transfer learning are still relatively simple. Summary of the Invention

[0006] This invention aims to expand research on semi-supervised instance segmentation. By designing a semi-supervised instance segmentation algorithm based on transfer learning, a feature transfer branch is established on the Mask R-CNN benchmark model, and the visual features of objects learned by the object detection branch are used to improve segmentation prediction performance.

[0007] The objective of this invention is achieved as follows:

[0008] A semi-supervised instance segmentation model with feature transfer is established by connecting the last convolutional network layer of the object detection branch of the Mask R-CNN model with the original deconvolutional network layer of the segmentation prediction branch through deconvolutional (and convolutional) network modules. This model is then trained using a semi-supervised instance segmentation dataset to improve the performance of segmentation prediction by utilizing the object visual features learned from the object detection branch.

[0009] The specific steps are as follows:

[0010] (1) Establish a semi-supervised instance segmentation model based on feature transfer

[0011] The semi-supervised instance segmentation model proposed in this invention is called FT-Mask (Feature TransferMask R-CNN). Figure 1 The overall structure of the FT-Mask model is shown. The model is based on Mask R-CNN, a high-performing fully supervised model in instance segmentation, with the central feature transfer module representing an innovative network structure proposed in this invention. As shown in the figure, the entire network is mainly divided into a shared network front section, an upper object detection branch, a lower segmentation prediction branch, and the central feature transfer module. The shared network front section extracts visual features from the image, and the RoIAlign layer then extracts these visual features as Region of Interest (RoI) features. The object detection branch and the segmentation prediction branch use the RoI features for object detection prediction and object segmentation mask prediction, respectively.

[0012] The object detection branch of the model is trained using the A∪B dataset with detection labels, where the full instance segmentation annotations in dataset A include object detection annotations. The segmentation prediction branch is trained using dataset A with segmentation label annotations. The FT-Mask model utilizes both fully labeled and weakly labeled data during training, thus its training process is semi-supervised.

[0013] exist Figure 1In this model, the feature transfer module takes the feature map output from the convolutional network layer of the object detection branch as input, and "transfers" the visual knowledge in the feature map to obtain the output feature map. The feature map output by the feature transfer module and the original segmentation prediction feature map are concatenated through corresponding channel addition operations and then used as input to the deconvolutional network layer in the segmentation branch. The deconvolutional layer of the segmentation prediction branch enlarges the output feature map to twice its original size, followed by a 1×1 convolutional layer for segmentation prediction. After semi-supervised training of the FT-Mask model, its feature transfer module can effectively transform the knowledge learned in the object detection branch into knowledge used for segmentation prediction.

[0014] (2) Establish a feature transfer module

[0015] This invention proposes to use a feature transfer method to achieve knowledge transfer, and designs a specific feature transfer module to simulate the knowledge transfer function. Figure 2 This is a schematic diagram of the feature transfer module. The upper part of the diagram is a basic feature transfer module composed of multiple neural network layers. The two ends of the basic feature transfer module are deconvolutional network layers, and the ellipsis in the middle represents multiple convolutional network layers. Figure 2 The lower half consists of a segmentation aggregation module comprised of a single neural network layer. This layer has only one output channel, and its single-channel feature map "aggregates" information from all channels of the output features of the object detection branch. When the output features of the upper and lower parts are concatenated, the single-channel feature map output by the segmentation aggregation module is added to each corresponding channel of the feature map output by the basic feature transfer module. To address the mismatch between the feature resolution of the object detection branch and the segmentation prediction branch in the Mask R-CNN base network, the first network layer of the basic feature transfer module and each individual network layer of the segmentation aggregation module are implemented using deconvolutional layers.

[0016] The feature transfer module simulates a special feature knowledge transfer function, as shown in Equation 1. In Equation 1, Tbase and Tagr represent the basic transfer module and the segmentation aggregation module, respectively, while fdet and fseg represent the detection features and segmentation features, respectively.

[0017] (1)

[0018] Figure 2The design of the segmentation aggregation module is inspired by the "class-agnostic" segmentation prediction branch in the MaskX R-CNN model, but it primarily implements class-agnostic feature transfer rather than segmentation prediction. Class-agnostic means that the segmentation prediction branch of the Mask R-CNN model outputs a 1×M×M segmentation map, where M represents the height and width of the segmentation map, and 1 indicates that the segmentation prediction map only generates a single overall foreground segmentation result for the total number of different target classes. The segmentation aggregation module of the FT-Mask model generates a 1×h×w output feature map for the feature transfer process, where h and w represent the height and width of the feature map, respectively. The output feature map of the segmentation aggregation module increases the diversity of knowledge transformation in the feature transfer process.

[0019] The feature transfer module design addresses the issue of mismatched resolution between the target detection branch output features and the segmentation prediction branch features in the base network Mask R-CNN of the FT-Mask model. Figure 3 The diagram illustrates the structure of the object detection branch network head of the Mask R-CNN model. In the diagram, 7×7 and 1×1 represent the height × width of different feature maps or different convolutional kernels, and C1 and C2 represent the number of channels in different feature maps. Figure 3 As shown, the first convolutional network layer of the object detection branch downsamples the input feature map with a resolution of 7×7 to a resolution of 1×1 using a 7×7 convolutional kernel. Then, a convolutional network layer with a kernel size of 1×1 transforms the features and outputs a feature map with a resolution of 1×1. Figure 3 The 1×1×C2 feature map output by the convolutional network layer in the object detection branch serves as the input feature map for the feature transfer module. The feature transfer module transforms the output features of the object detection branch into features for segmentation prediction. However, segmentation prediction requires a high-resolution feature map (typically 14×14), so the feature transfer module needs further design to address the resolution mismatch issue.

[0020] To address the aforementioned issues, the feature transfer branch upsamples the input features to the size required for segmentation prediction using a deconvolutional network layer. Deconvolution, also known as de-convolution or transposed convolution, is a commonly used feature map upsampling method. Other common feature map upsampling methods include bilinear interpolation, upsampling, and unpooling. While these other methods obtain the output feature map through fixed computation methods, the deconvolutional network layer obtains the output feature map through adaptive network parameters and convolution operations during upsampling prediction. During network model training, the parameters of the deconvolutional network layer are updated through backpropagation (BP), thus achieving higher accuracy compared to other upsampling methods. Attached Figure Description

[0021] Figure 1 Overall structure of the FT-Mask model

[0022] Figure 2 Feature transfer module structure

[0023] Figure 3 Schematic diagram of the object detection branch network head of the Mask R-CNN benchmark model Detailed Implementation

[0024] This invention requires first establishing a neural network model, then training it on a semi-supervised instance segmentation dataset, and finally performing instance segmentation on the image. The model can be implemented using neural network frameworks such as Keras and TensorFlow, and trained using a high-performance GPU (Graphics Processing Unit) server.

[0025] The algorithm model's backbone convolutional neural network consists of a ResNet50 (residual networks) head and a Feature Pyramid Network (FPN) network. The ResNet50 head uses network parameters trained on the ImageNet image classification dataset. The ResNet50 convolutional network extracts various visual features from the input image, and the Feature Pyramid Network fuses these visual features at different levels and resolutions. The feature transfer branch connects the last convolutional network layer of the object detection network branch and the deconvolutional network layer of the segmentation network branch. Subsequently, the feature output of the transfer branch, along with the original segmentation prediction features, participates in segmentation prediction.

[0026] The basic feature transfer module and the segmentation aggregation module are integrated through a layer-by-layer addition operation. The basic feature transfer module outputs a feature map of size h×w×C, while the segmentation aggregation module outputs a feature map of size h×w×1. h and w represent the height and width of the feature map, respectively, and C represents the number of channels. The layer-by-layer addition operation means that for each channel of the feature map output by the basic transfer module, the single-channel feature map output by the segmentation aggregation module is added to it. The total size of the feature map output by the transfer module is h×w×C. In the specific implementation of the FT-Mask model, h and w are both set to 14, and C is set to 256.

[0027] Following the segmentation prediction branch structure of the Mask R-CNN model, the FT-Mask model adds a BatchNorm network layer and a ReLU activation layer after the deconvolutional and convolutional network layers in the feature transfer module. The BatchNorm network layer addresses the problem of internal covariate shift (ICS) in the probability distribution of neural network output features, making the neural network easier to train. The ReLU activation layer provides a non-linear transformation to the neural network output and suppresses negative activation of output features.

[0028] In the encoding implementation of the FT-Mask model, the inference stage needs to address the issue arising from the discrepancy between the number of output features from the object detection branch and the number of input features from the segmentation prediction branch of the Mask R-CNN base model. During its inference phase, the Mask R-CNN model first filters the RoIs output by the RPN network using the NMS algorithm based on the object detection results. Then, it uses the corresponding RoI features as the feature input to the segmentation prediction branch (the number of features is limited to no more than 100 by default), while its object detection branch inputs a normal number of features (1000 by default). The feature transfer branch of the FT-Mask model uses the object detection features as input, and its output feature count is consistent with that of the object detection branch. Therefore, there is a mismatch in the number of output features between the transfer branch and the segmentation prediction branch.

[0029] The FT-Mask model incorporates a custom feature filtering layer during the inference phase to filter the output features of the transfer branch. This feature filtering layer employs the same strategy as the RoI filtering process in Mask R-CNN, filtering output features at the same indices. The number and indices of the transfer features retained by the feature filtering layer are consistent with the number of input features in the segmentation prediction branch; therefore, adding this feature filtering layer solves the aforementioned problem.

[0030] Semi-supervised dataset partitioning:

[0031] The COCO instance segmentation dataset can be divided into semi-supervised datasets, followed by semi-supervised instance segmentation training. For example, 25% of the COCO training set data with complete instance segmentation labels can be selected as sub-training set A, and an additional 25% of the COCO training set data with object detection labels can be randomly added as sub-training set B. The COCO instance segmentation validation set can be used as the test dataset for the experiments in this chapter.

[0032] Model training methods:

[0033] The FT-Mask model's phased training process first uses the merged A and B datasets to train the model's object detection branch, FPN, and RPN, enabling the object detection branch to output better visual features (Phase 1). Then, the model's segmentation prediction branch and feature transfer branch are trained using dataset A (Phase 2). In the second phase of the phased training process, the segmentation branch benefits from the multi-task learning shared network features, and the object detection branch, being trained first, is more conducive to the feature transformation learning of the feature transfer module.

[0034] The second step in training the FT-Mask model employs a joint training strategy. This strategy jointly trains the object detection branch, segmentation prediction branch, feature transfer branch, FPN, and RPN of the FT-Mask model on dataset A to optimize the overall performance of the network model. The base network of the FT-Mask model adopts a multi-task learning structure and shares network parameters; therefore, the joint training strategy allows multiple network branches to mutually reinforce each other during learning.

[0035] Examples of user scenarios for this invention:

[0036] Scenario 1: In instance segmentation application development, users do not need to perform complete instance segmentation annotation on newly collected image data. They can annotate some images with instance segmentation labels and the remaining images with object detection labels. In this scenario, the dataset annotation cost for application development can be reduced while achieving good image instance segmentation results.

Claims

1. A semi-supervised instance segmentation method based on feature transfer, characterized in that, The method includes the following steps: Step 1: Establish a semi-supervised instance segmentation model FT-Mask (Feature Transfer Mask R-CNN). The entire model is divided into a shared network front section, an upper object detection branch, a lower segmentation prediction branch, and a middle feature transfer module. A feature transfer module is established between the last convolutional network layer of the object detection branch and the original deconvolutional network layer of the segmentation prediction branch in the Mask R-CNN model. The shared network front section is used to extract visual features of the image, and then the RoIAlign layer extracts these visual features as Region of Interest (RoI) features. The object detection branch and the segmentation prediction branch use the RoI features for object detection prediction and object segmentation mask prediction, respectively. Step 2: The object detection branch of the model is trained using the A∪B dataset with detection labels. The complete instance segmentation label of dataset A includes object detection labels, while dataset B only contains detection label labels. The segmentation prediction branch is trained using dataset A with segmentation label labels. The FT-Mask model utilizes both data with complete instance segmentation labels and weakly labeled data during training. Step 3: The model feature transfer module takes the feature map output by the convolutional network layer of the object detection branch as input, and transfers the visual knowledge in the feature map to obtain the output feature map. The feature map output by the feature transfer module and the original segmentation prediction feature map are concatenated by adding the corresponding channels and then used as the input of the deconvolutional network layer in the segmentation branch. The deconvolutional layer of the segmentation prediction branch enlarges the size of the output feature map to twice its original size, followed by a convolutional layer with a kernel size of 1×1 used for segmentation prediction.

2. The semi-supervised instance segmentation method based on feature transfer as described in claim 1, characterized in that, The feature transfer module in step 1 consists of a basic feature transfer module and a segmentation aggregation module. The basic feature transfer module has deconvolutional network layers at both ends and multiple convolutional network layers in the middle. The network layer of the segmentation aggregation module has only one output channel, and its output single-channel feature map aggregates the information of all channels of the output features of the object detection branch. The output features of the upper and lower parts are connected, and the single-channel feature map output by the segmentation aggregation module is added to each channel of the output feature map of the basic feature transfer module. The first network layer of the basic feature transfer module and the individual network layers of the segmentation aggregation module are both implemented by deconvolutional layers. The segmentation and aggregation module of the FT-Mask model generates a 1×h×w output feature map for the feature transfer process, where h and w represent the height and width of the feature map, respectively. In the structure of the object detection branch network head of the Mask R-CNN model, 7×7 and 1×1 represent the height × width of different feature maps or different convolutional kernels, and C1 and C2 represent the number of channels of different feature maps. The first convolutional network layer of the object detection branch downsamples the input feature map with a resolution of 7×7 to a resolution of 1×1 through a 7×7 convolutional kernel. Then, the convolutional network layer with a 1×1 kernel transforms the features and outputs a feature map with a resolution of 1×1. The 1×1×C2 feature map output by the convolutional network layer of the object detection branch is the input feature map of the feature transfer module. The feature transfer branch upsamples the input features to the size required for segmentation prediction through the deconvolutional network layer; the deconvolutional network layer obtains the output feature map through adaptive network parameters and convolution operations during upsampling prediction; during the training of the network model, the parameters of the deconvolutional network layer are updated through back propagation (BP).

Images