Semi-supervised zero sample semantic segmentation method

Abstract

The invention provides a semi-supervised zero sample semantic segmentation method, and the method provided by the invention comprises four neural network models which are respectively a ViT (Vision Transform) encoder, a pre-training image encoder, a pixel level decoder and a Transform decoder. Meanwhile, the method consists of two modules: a self-supervised mask generation module and a pre-training module of the semantic segmentation decoder. Comprising the following steps: preparing data, and carrying out image serialization to convert an input image X into feature embedding; mask feature extraction is carried out, data are input to a first layer Transform, and each Transform layer is composed of a multi-head attention layer, a layer normalization layer and a feedforward network layer; and then, respectively inputting the features into a Transform decoder and a pixel-level decoder, and performing matrix product on each output feature to predict N masks and complete matching to obtain a segmentation result. In addition, a decoder also needs to learn specific image semantic information. The features of the positive samples and the negative samples are compared by using self-supervised contrast learning, the similarity between the positive samples is increased, and the similarity between the negative samples and the positive samples is reduced.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of computer vision, and in particular to a method for semantic segmentation based on contrastive learning. Background Art

[0002] Semantic segmentation is one of the core research tasks in the field of computer vision, and its goal is to classify images at the pixel level. Semantic segmentation is widely used in various tasks in real-world scenarios, such as obtaining driving areas in the field of autonomous driving, diagnosing lesions in the medical field, and estimating gender, age, and expression in the field of face recognition.

[0003] In the field of computer vision, semantic segmentation is the task of classifying images into different object or background categories at the pixel level. The introduction of the Visual Transformer has greatly improved the learning ability of the backbone network. The performance of networks such as ViT and SwinTransformer far exceeds convolutional neural networks such as VGGNet and ResNet.

[0004] Contrastive learning is an excellent strategy to improve the discreteness of feature space. A more discrete feature space can make classification easier. The idea of ​​applying contrastive learning to semi-supervised semantic segmentation is inspired by self-supervised contrastive learning, which compares the features of positive and negative samples to increase the similarity between positive samples and reduce the similarity between negative and positive samples. Summary of the invention

[0005] At present, the semi-supervised semantic segmentation methods based on contrastive learning are all based on pixel-level contrastive learning. Due to the large consumption of computing resources, it is impossible to use all pixels for contrastive training. The research direction of this invention is semi-supervised semantic segmentation. This paper aims to propose a semi-supervised learning framework of a hierarchical Transformer encoder based on multi-scale features.

[0006] This paper designs a mask generation module based on the self-supervised Transformer semantic segmentation decoder, which provides a new perspective on the semantic segmentation problem. On the ImageNet dataset, the multi-scale feature layered ViT is used as the backbone network, the attention matrix is ​​regarded as an attention map, and the weights of the map are cut in an unsupervised manner to generate a mask; the method includes the following steps:

[0007] (1) Input the image into ViT to complete feature extraction;

[0008] (2) Using Transformer-based visual encoder to learn feature representation;

[0009] (3) The features are input into the self-supervised mask generation module to obtain the image segmentation result and divide it into masks, and output the feature f.

[0010] The present invention aims to solve the problems of unclear segmentation areas and inability to identify unclassified images. It proposes adding a contrastive learning pre-training module to the model, using the generated masks as labels, freezing the parameter weights of the backbone network, and using the semantic segmentation network to complete pre-training on the ImageNet dataset, thereby improving the training difficulty of the Transformer-based semantic segmentation decoder and reducing the training time. Specifically, it includes the following steps:

[0011] (1) Input the features output by Vision Transformer into the decoder to obtain the output features F;

[0012] (2) Input the output f of the Transformer decoder into the MLP, and obtain the output category probability distribution through average pooling features;

[0013] (3) Input the image into CLIP encoding to obtain the feature probability representation of the image;

[0014] (4) Use semantic contrast loss to optimize model parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0016] Figure 1 It is the overall model diagram of the present invention;

[0017] Figure 2 The present invention generates a feature mask structure diagram based on Transformer;

[0018] Figure 3 This is a diagram of the contrastive learning pre-training structure added by the present invention. DETAILED DESCRIPTION

[0019] The specific implementation of the present invention is given below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present invention.

[0020] For details, see Figure 1, a semi-supervised zero-shot semantic segmentation method, comprising the steps of:

[0021] S1: Data preparation, image serialization is performed to convert the input image X into feature embedding. The production steps are as follows:

[0022] S11: Use convolution to divide the H×W×3 image into multiple patch tokens in the form of a grid, and the size of each patch is 16×16;

[0023] S12: Add a class token for classification training of the final model. The class token is directly concatenated with the previous image patch token, so a matrix with dimension T = 197 × d is obtained, where d refers to the channel dimension of the matrix.

[0024] S2: Get the image mask, add position encoding, and input data to the first layer of Transformer. Each Transformer layer consists of multi-head attention, layer normalization, and feedforward network layers. Input E obtained in the previous step into 24 series-connected Transformers, so that the receptive field of each Transformer is the entire image. Including steps:

[0025] S21: Given a one-dimensional embedding sequence E as input, a Transformer-based encoder is used to learn feature representations;

[0026] S22: Transformer encoder consists of L e The network consists of a multi-head self-attention layer and a feed-forward network layer. In each layer, we first need to obtain Q, K and V, which represent query, key and value respectively, by linearly transforming the input X:

[0027] query=XW Q ,key=XW K ,value=XW V

[0028] Where W Q , W K , W V ∈R C×d is the parameter of the linear transformation, and the fully connected layer is used here to complete the linear transformation;

[0029] S23: Attention first performs matrix product on query and key, then uses softmax to activate to get the attention map, and finally multiplies the result of the attention map with the value, as shown in the formula:

[0030]

[0031] S24: Multi-head self-attention is an extension of m independent self-attention operations, dividing the d dimension into d′=d / h, where h is the number of division heads. The output of the multi-head self-attention is then converted to FFNs with residual skip as output:

[0032] MSA(Z l-1 )=[SA1(Z l-1 );SA2(Z l-1 );…;SA m (Z l-1 )]W O

[0033] S25: FFNs previously used dropout to improve generalization and used the GeLU function to complete nonlinearity. The FFNs process formula is as follows:

[0034] FFN(t)=drop(FC2(GeLU(drop(FC1(t)))))

[0035] Among them, FC1 and FC2 represent two fully connected layers. FC1 is used to expand the original channel dimension d to 4d, while FC2 is used to convert 4d;

[0036] S3: The features are then input into the Transformer decoder and the pixel-level decoder respectively, and matrix products are performed on each output feature to predict N masks and complete the matching. The steps are as follows:

[0037] S31: Pre-training, extracting the features F ViT As input, it is input into the decoder to obtain F and f, which are matrix-multiplied and activated with sigmoid to obtain N queries and the corresponding mask M. ；

[0038] S32: Set N queries and compare N queries with M ViT Perform bipartite graph matching to obtain different masks that need to be trained. Use binary mask loss between matched masks;

[0039] S33: Input f output by the Transformer decoder into the MLP, and obtain p after average pooling features. t Input the image x into the adapted CLIP code to obtain the image feature representation p c , using semantic contrastive loss to optimize model parameters;

[0040] S4: Generate segmentation results, and send the new feature vector output by the cross-attention module acting as the decoder to the segmentation layer to predict the final segmentation result.

Claims

1. A semi-supervised zero-shot semantic segmentation method, comprising the steps of: S1: Data preparation, image serialization is performed to convert the input image X into feature embedding. The production steps are as follows: S11: Use convolution to divide the H×W×3 image into multiple patch tokens in the form of a grid, and the size of each patch is 16×16; S12: Add a class token for classification training of the final model. The class token is directly concatenated with the previous image patch token, so a matrix with dimension T = 197 × d is obtained, where d refers to the channel dimension of the matrix. S2: Get the image mask, add position encoding, and input data to the first layer of Transformer. Each Transformer layer consists of multi-head attention, layer normalization, and feedforward network layers. Input E obtained in the previous step into 24 series-connected Transformers, so that the receptive field of each Transformer is the entire image. Including steps: S21: Given a one-dimensional embedding sequence E as input, a Transformer-based encoder is used to learn feature representations; S22: Transformer encoder consists of L e The network consists of a multi-head self-attention layer and a feed-forward network layer. In each layer, we first need to obtain Q, K and V, which represent query, key and value respectively, and are obtained by linearly transforming the input X: query=XW Q ,key=XW K ,value=XW V Where W Q , W K , W V ∈R C×d is the parameter of the linear transformation, and the fully connected layer is used here to complete the linear transformation; S23: Attention first performs matrix product on query and key, then uses softmax to activate to get the attention map, and finally multiplies the result of the attention map with the value, as shown in the formula: S24: Multi-head self-attention is an extension of m independent self-attention operations, dividing the d dimension into d′=d / h, where h is the number of division heads. The output of the multi-head self-attention is then converted to FFNs with residual skip as output: MSA(Z l-1 )=[SA1(Z l-1 );SA2(Z l-1 );…;ARE m (WITH l-1 )]IN O S25: FFNs previously used dropout to improve generalization and used the GeLU function to complete nonlinearity. The FFNs process formula is as follows: FFN(t)=drop(FC2(GeLU(drop(FC1(t))))) Among them, FC1 and FC2 represent two fully connected layers. FC1 is used to expand the original channel dimension d to 4d, while FC2 is used to convert 4d; S3: The features are then input into the Transformer decoder and the pixel-level decoder respectively, and matrix products are performed on each output feature to predict N masks and complete the matching. The steps are as follows: S31: Pre-training, extracting the features F ViT As input, it is input into the decoder to obtain F and f, which are matrix-multiplied and activated with sigmoid to obtain N queries and corresponding masks M; S32: Set N queries and compare N queries with M ViT Perform bipartite graph matching to obtain different masks that need to be trained. Use binary mask loss between matched masks; S33: Input f output by the Transformer decoder into the MLP, and obtain p after average pooling features. t Input the image x into the adapted CLIP code to obtain the image feature representation p c , using semantic contrastive loss to optimize model parameters; S4: Generate segmentation results, and send the new feature vector output by the cross-attention module acting as the decoder to the segmentation layer to predict the final segmentation result.

Images