A generalized small sample semantic segmentation method based on word vector embedding-information self-complement

Abstract

The application discloses a generalized small sample semantic segmentation method based on word vector embedding-information self-completion, comprising: a base class learning stage: training a backbone network, a semantic word vector embedding module and a base class classifier using base class label samples to generate a base class prototype; a new class updating stage: training a new class classifier using one or five new class label samples to generate a new class prototype; an evaluation stage: inputting the base class prototype obtained in the base class learning stage and the new class prototype obtained in the new class updating stage into a self-completion module, using the base class prototype and the new class prototype to generate pseudo labels through initial prediction of the query image by the self-completion module, then extracting a query prototype in the query image through the pseudo labels, calculating the cosine similarity between the query prototype and the query image to generate an auxiliary prediction; finally, weighting the auxiliary prediction and the initial prediction to obtain a final prediction result. The application increases the inter-class distance of the model and excavates more information, and further improves the segmentation performance of the model.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of semantic segmentation, in particular to a generalized few-shot semantic segmentation method based on word vector embedding-information self-complement. BACKGROUND

[0002] For the generalized few-shot semantic segmentation problem, the usual solution method is divided into two categories: one is to increase the distance between classes, which uses loss function and orthogonal prototype to increase the distance between the class prototype vectors learned by the model. The second is to use optimization strategies, such as: fuse the prediction scores of new class classifier and base class classifier and use calibration network to make the fusion score not biased towards base class or new class, and use the InfoMax principle to maximize the mutual information between the learned features and their corresponding predictions. However, the first method cannot well learn the distance between classes when only a small number of samples (one or five) are used, so the trained model cannot distinguish between classes with similar semantic features, further leading to model segmentation errors, which is called inter-class interference. The second method improves the performance of the model using optimization strategies, but since the generalized few-shot segmentation model uses class prototype vectors to store class information, this approach will result in the loss of semantic information of the class, especially the image space information of the class. The missing valuable class semantic information makes the model unable to accurately divide the boundaries of the class and even produces large area segmentation errors, thereby damaging the segmentation accuracy of the model for new classes and base classes. SUMMARY

[0003] In order to overcome the above-mentioned shortcomings and deficiencies of the prior art, the purpose of the present application is to provide a generalized few-shot semantic segmentation method based on word vector embedding-information self-complement.

[0004] The present application uses semantic word vector embedding to assist the generalized few-shot semantic segmentation model to better learn the distance between the class prototype vectors of the image samples. And uses a self-complement module to mine information in the query image to make up for the loss of information due to the use of prototype vectors to represent classes in the final evaluation stage.

[0005] The method model proposed in the present application achieves excellent segmentation performance on the problem of generalized few-shot semantic segmentation, and for the first time introduces additional information (semantic word vector) to assist in training the model while solving the problem of information loss caused by using prototype vectors to represent classes.

[0006] The purpose of the present application is achieved by the following technical solutions:

[0007] A generalized few-shot semantic segmentation method based on word vector embedding-information self-complement for computing the pixels of unlabeled images and assigning corresponding class labels, comprising:

[0008] The base class learning stage: training the backbone network, the semantic word vector embedding module and the base class classifier using the base class label samples to generate base class prototypes;

[0009] The new class updating stage: training the new class classifier using one or five new class label samples to generate new class prototypes;

[0010] The evaluation stage: inputting the base class prototypes obtained in the base class learning stage and the new class prototypes obtained in the new class updating stage into the self-completion module, using the base class prototypes and the new class prototypes to generate pseudo labels through initial prediction of the query image by the self-completion module, then extracting query prototypes in the query image through the pseudo labels, calculating the cosine similarity between the query prototypes and the query image to generate auxiliary prediction; finally, weighting the auxiliary prediction and the initial prediction to obtain the final prediction result.

[0011] Further, the base class learning stage has the following specific process:

[0012] Inputting M base class label samples into the backbone network to obtain support features, inputting class words into a pre-trained word2vec to obtain word vectors;

[0013] Inputting the support features, the word vectors and the image corresponding mask into the semantic word vector embedding module to generate base class models.

[0014] Further, the inputting of the support features, the word vectors and the image corresponding mask into the semantic word vector embedding module to generate base class models has the following specific process:

[0015] Using mask average pooling on the support features to obtain image prototypes;

[0016] Inputting the image prototypes and the word vectors into a word vector prototype generator to generate word vector prototypes;

[0017] Updating the word vector prototypes using the newly obtained word vector prototypes;

[0018] Inputting the word vector prototypes and the class prototypes into a correlation estimator to obtain adaptive weights;

[0019] Using the adaptive weights to weight and integrate the word vector prototypes and the class prototypes to obtain integrated prototypes, i.e., base class prototypes;

[0020] Inputting the integrated prototypes and image features into a base class classifier to perform segmentation prediction and calculate training loss.

[0021] Further, the word vector model is updated using the newly obtained word vectors, specifically, the word vectors are updated under the supervision of a triplet loss function, and then the prototype integration generator is used to complete the integration of the word vector prototypes and the class prototypes.

[0022] Further, the word vector is converted from semantic space to visual space using a triplet loss function, specifically:

[0023] The word vector prototype is set as an anchor point, and then positive and negative pairs are sampled from the general features obtained by the general feature generator to calculate the triplet loss.

[0024] Further, the general feature generator is composed of a convolutional block to reduce the support features by 4 times to obtain general features.

[0025] Further, the evaluation stage is specifically:

[0026] The base class prototype and the new class prototype input corresponding classifier generates the initial prediction y of the query image qry

[0027] The threshold τ is applied to select the high confidence area to generate the pseudo label M qry , y qry The values greater than τ are set to 1, and the values less than τ are set to 0, τ = 0.8;

[0028] Mask global average pooling is performed using the pseudo label and the query feature to generate the query prototype p query :

[0029] The cosine distance between the query prototype p q and the query feature f aux is calculated to generate the auxiliary prediction y qry :

[0030]

[0031] The initial prediction y αux and the auxiliary prediction y final are combined to obtain the final prediction result y final :

[0032] y qry = (1-ε) * y aux + ε * y seg

[0033] Where ε is the weight.

[0034] Further, in the base class learning stage, the loss function adopts the base class segmentation loss and the triplet loss, L = L triplet + α * L s , set α = 0.2;

[0035] In the new class update stage, the base class segmentation loss function is used.

[0036] Further, set ε = 0.3.

[0037] Further, the word vector prototype generator is composed of two fully connected layers.

[0038] Compared with the prior art, the present application has the following advantages and beneficial effects:

[0039] 1. Compared with the generalized small sample semantic segmentation method using loss function and orthogonal prototype to increase the distance between classes, the present application uses the distance between word vectors to enable the model to learn more effective and more practical inter-class distance, thereby effectively alleviating the inter-class interference problem.

[0040] 2. The semantic word vector embedding module and the triplet loss function designed in the present application can well complete the embedding of semantic word vectors and map the semantic word vectors from the text space to the visual space.

[0041] 3. The self-complementing module designed in the present application can effectively mine information from the query image, thereby making up for the information loss caused by using prototype to represent classes.

[0042] 4. The method model designed in the present application is a general model that can be applied to different data sets (CT data set and remote sensing data set), and the unified end-to-end network model can be used for generalized small sample semantic segmentation learning for different types of data.

[0043] 5. The method model designed in the present application can well maintain the performance of the base class, and even after the model completes the new class update, the performance of the base class will not decrease significantly, and the case of losing the performance of the base class for learning the new class will not occur.

[0044] 6. The method model designed in the present application achieves excellent performance on the commonly used generalized small sample semantic segmentation data sets PASCAL-5i and COCO-20i, and is better than the methods of the same period. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 is a structural schematic diagram of the base class learning stage and the new class updating stage of the present application;

[0046] Figure 2 is a structural schematic diagram of the semantic word vector embedding module of the present application;

[0047] Figure 3 is a structural schematic diagram of the self-complementing module of the present application. DETAILED DESCRIPTION

[0048] The present application will be further described in detail below in conjunction with embodiments, but the embodiments of the present application are not limited thereto.

[0049] EMBODIMENT

[0050] AsFigures 1-3 As shown, this embodiment provides a generalized small-sample semantic segmentation method based on word vector embedding and information self-completion, which is used for computer recognition of unlabeled image pixels and assignment of corresponding class labels, including:

[0051] S1 base class learning phase: Use a large number of base class label samples to train the model's backbone network, semantic word vector embedding module, and base class classifier to generate base class prototypes. Specifically:

[0052] S1.1 Input two base classes: images of people and cats into the backbone network to extract support features f s , and then input the class word (e.g., "cat, person") into the pre-trained word2vec to obtain the word vector w:

[0053] w=F word2vec (word)

[0054] F word2vec (*) indicates a pre-trained word2vec model that generates corresponding word vectors based on class words.

[0055] S1.2 will support feature f s , word vector w and image corresponding mask Masks are input into the semantic word vector embedding module to generate the base class prototype. Specifically, Figure 2 As shown:

[0056] S1.2.1, for support feature f s Use mask average pooling to obtain image prototypes n is the number of categories contained in the image:

[0057]

[0058] Among them F pool represents average pooling. is obtained by interpolation and expansion techniques. Adjust to f s Functions of the same shape. It is class C i In f s The mask on.

[0059] S1.2.2 Image prototype and word vector w i Input into the word vector prototype generator to generate the word vector prototype

[0060]

[0061] in, is the cascade operation along the channel dimension, FWVPG (*) denotes the word vector prototype generator, which is implemented by two fully connected layers with the parameter settings shown in Table 1.

[0062] Table 1. Parameter settings of word vector prototype generator

[0063] Layer Batch Size Channel Width Input 2 512 256 Linear 2 512 256 LeakyReLU 2 256 256 Dropout 2 256 256

[0064] S1.2.3, using new word vector prototypes updating word vector prototypes

[0065]

[0066]

[0067] where F d is the cosine similarity function, word vector prototype P em is a global parameter and is randomly initialized, and t denotes the number of updates.

[0068] In this embodiment, the word vector is updated under the supervision of a triplet loss function, and then the prototype integration generator is used to complete the integration of the word vector prototype and the class prototype.

[0069] In order to apply the distance information carried by the word vector in the visual space, we designed a triplet loss function to convert the word vector from the semantic space to the visual space. Among them, we set as the anchor point, and then sample positive and negative pairs from the general feature to calculate the triplet loss. The general feature is a further abstraction of the image feature, representing the general semantic information of the category, such as tail, head, torso and other features. We designed a general feature generator (GFG) composed of convolutional blocks, and reduced the size of the support feature by 4 times to obtain the general feature f gen , whose parameter settings are shown in Table 2.

[0070] f gen = F GFG (f s )

[0071] where F GFG (*) denotes the general feature generator. Then, we use the mask for feature selection, which separates the class c i and other region vectors into two different sets, that is:

[0072]

[0073]

[0074]

[0075] If is equal to 1, then is added to V i , otherwise it is added to V other . The positive sample is the universal vector of class c i in V i . We compute the hardest sample, i.e., the sample farthest from the anchor, to obtain the positive sample in order to better optimize.

[0076]

[0077] where F d is the l2 distance function. We average V other to obtain the negative sample because the region excluding class c i contains multiple classes.

[0078]

[0079] The triplet loss L triplet is:

[0080]

[0081] where margin is a fixed value to keep the negative sample far away, and margin is 0.5 in this embodiment. The triplet loss aligns the semantic information of with the visual information of the universal feature.

[0082] Table 2 Universal feature generator parameter settings

[0083] Layer In_channels Out_channels Kernel_size stride padding bias Input 256 Conv2d 256 256 3 2 0 false ReLU 256 256 Conv2d 256 256 3 2 1 false ReLU 256 256 Conv2d 256 256 3 1 1 false

[0084] Further, the integration of the word vector prototype and the class prototype is completed using the prototype integration generator, specifically:

[0085] First, we input the word vector prototype p em and the class prototype p into the correlation estimator Φ to obtain the adaptive weight γ. Then, we use the adaptive weight γ to weight and integrate the word vector prototype p em and the class prototype p.

[0086]

[0087]

[0088] During the base class learning phase, K is M, while during the new class registration phase, K is M+N. The adaptive weight γ balances the influence of word vector prototypes and class prototypes. This is because γ is data-dependent and its value depends on each pair of word vector prototypes and class prototypes. The correlation estimator Φ consists of linear layers, with parameter settings shown in Table 3.

[0089] S1.2.4 We will integrate the prototype and image features f s The input is fed into the base classifier ψ for segmentation prediction, and the training loss is calculated using the cross entropy loss function:

[0090] y cls =ψ(p cls ,f s )

[0091] L seg =CE(y cls ,M s )

[0092] Where ψ represents the base class classifier in the base class learning phase and represents the base class and new class classifier in the new class updating phase. CE(·) represents the cross entropy loss function. cls The predictions generated by the classifier, M s is the true mask.

[0093] Table 3 Correlation estimator parameter settings

[0094] Layer In_size Out_size Input 1024 Linear 1024 512 ReLU 512 512 Linear 512 1

[0095] S2 New Class Update Phase: Use limited (one or five) new class label samples to train the model's new class classifier and new class prototype. Specifically:

[0096] exist Figure 1 In the new class update phase, the model processing flow is similar to the base class learning phase. The difference between the new class update and base class learning phases is: 1. The loss function in the base class learning phase is L = L seg +α*L triplet , set α = 0.2, and in the new class update stage the loss function is L seg . 2. The base class learning phase trains the model’s backbone network, semantic word vector embedding module, base class classifier, and base class prototype, while the new class updating phase trains the model’s new class classifier and new class prototype.

[0097] S3 evaluation stage: The base class and new class prototypes learned in the previous two stages are input into the self-completion module, which uses the base class and new class prototypes to make an initial prediction on the query image to generate pseudo labels, and then extracts query prototypes in the query image through the pseudo labels. Next, the cosine similarity between the query prototypes and the query image is calculated to generate an auxiliary prediction. Finally, the auxiliary prediction and the initial prediction are weighted to obtain the final prediction result.

[0098] Specifically, we first use the base class, new class prototypes, and classifier to generate an initial prediction y qry :

[0099]

[0100] Then, a threshold τ is applied to select high-confidence regions to generate pseudo labels M qry :

[0101] M qry = F index (y qry , τ)

[0102] The function F index sets the values greater than τ in y qry to 1 and the values less than τ to 0, and we set τ = 0.8.

[0103] Subsequently, we use the pseudo labels and query features to generate query prototypes p query :

[0104]

[0105] n is the number of categories contained in the query image, and MAP(·) represents the mask global average pooling.

[0106] Finally, we calculate the cosine distance between the query prototype and the query feature f q to generate a self-completion auxiliary prediction y aux :

[0107]

[0108] We combine the prediction y qry and the self-completion auxiliary prediction y aux by weighting to obtain the final prediction result y final :

[0109] y final = (1-ε) * y qry + ε * y aux

[0110] where ε is the weight, we set ε = 0.3.

[0111] In this embodiment, both the auxiliary prediction and the initial prediction are for image prediction categories, that is, class labels. The class label generally refers to what category each pixel point in the image belongs to. For example, for an image containing a dog, the class label is the image region containing the dog.

[0112] The above method of the application is specifically described using the commonly used generalized small sample semantic segmentation dataset PASCAL-5i. PASCAL-5i is created according to PASCAL VOC 2012 with additional annotations of SBD. There are a total of 20 classes in the dataset, which are evenly divided into 4 folds, and each fold contains 5 classes. For each training, the classes in three folds are selected as base classes, and the remaining folds are used as new classes. In this way, each training contains 15 base classes, 5 new classes and 1 background class. The training process of the model is divided into base class learning and new class updating. Among them, in the base class learning stage, we select images containing base classes from the training set of the dataset, and these images contain at least one base class for training the model. The part of the selected image containing the new class is regarded as background. In the new class updating stage, according to the experimental setting (1-shot or 5-shot), 1 or 5 images containing new classes are randomly selected to complete the learning of new classes. In the final evaluation stage, we use the validation set of the dataset to complete the segmentation evaluation of the base class and the new class. In the experiment, the average intersection over union (mIoU) of the base class, the new class and all classes is used as the performance measurement.

[0113] In order to make a fair comparison between our model and other generalized small sample semantic segmentation models, we use PSPNet in ResNet50 pre-trained on ImageNet as the feature encoder. The encoder output feature map of the original image 1 / 8 size is input into ESS to generate the corresponding prediction. ESS uses the learned class prototype to generate the prediction, and the number of learned class prototypes is related to the specific dataset. For example: in PASCAL-5i, 15 base classes and 5 new classes are learned. In the base class training stage, we use a batch size of 2, and use the SGD optimizer, set the initial learning rate of the optimizer to 1x10 -2 , the momentum to 0.9, and the weight decay to 1x10 -4 . For the two datasets, each model needs to be trained for 50 cycles. For new class updating, we update the model for 500 cycles at a fixed learning rate of 0.01, and set the batch size to 2. We use the word2vec model pre-trained on Google to obtain the d(300) dimensional word vector representation. For the word embedding of a class containing multiple words, the embedding of each individual word is obtained by averaging.

[0114] One specific application of the method is as follows:

[0115] In this example, the generalized few-shot semantic segmentation method based on word vector embedding-information self-complementing includes the following steps:

[0116] 1) Input the training images of 15 base classes into the backbone network to extract features f s , and input the class word (for example, "cat") into the pre-trained word2vec to obtain the word vector w:

[0117] w=F word2vec (word)

[0118] 2) Update the semantic word vector and class prototype for each training image through the semantic word vector embedding module, specifically:

[0119] Use mask average pooling on the features f s to obtain the image prototype n is the number of classes contained in the image:

[0120]

[0121] where F pool represents average pooling. is a function that adjusts to the same shape as f s through interpolation and expansion techniques. is the mask of class c i on f s .

[0122] Then, and the word vector w i are input into the word vector prototype generator, thereby generating the word vector prototype

[0123]

[0124] where, is a concatenation operation along the channel dimension, F WVPG (*) represents the word vector prototype generator, which is implemented by two fully connected layers.

[0125] Finally, the newly obtained word vector prototype updates the word vector prototype

[0126]

[0127]

[0128] Among them F d is the cosine similarity function, the word vector prototype P em is a global parameter and is randomly initialized, and t represents the number of updates.

[0129] 3) For the 5 new classes, use the same method as the base class 2) to complete the update learning of the new class word vector prototype and class prototype.

[0130] 4) Through 2) and 3), the model learned word vector prototypes and class prototypes for 5 new classes and 15 base classes. In the final evaluation phase, the model needs to use the prototypes learned in the first two phases to perform semantic segmentation on the test samples, while using the self-complementing module to supplement the information. Specifically:

[0131] We first use the base class, the new class prototype and the classifier to generate an initial prediction y for the query image qry ,The classifier parameter settings are shown in Table 4.

[0132]

[0133] Where M is 15 and N is 5.

[0134] Then, a threshold τ is applied to select high confidence regions to generate pseudo labels M qry :

[0135] M qry =F index (y qry ,τ)

[0136] Function F index y qry The values ​​greater than τ are set to 1, and the values ​​less than τ are set to 0. We set τ = 0.8.

[0137] We then perform masked global average pooling using the pseudo-labels and query features to generate the query prototype p query :

[0138]

[0139] n is the number of categories contained in the query image, and MAP(·) represents mask global average pooling.

[0140] Finally, we compute the query prototype With query feature f q The cosine distance between them to generate the self-complementary auxiliary prediction y aux :

[0141]

[0142] We predict y qryand self-supplementary auxiliary prediction y aux The weighted combination is performed to obtain the final prediction result y final :

[0143] y final = (1 - epsilon) * y qry + epsilon * y aux

[0144] Wherein epsilon is a weight, and we set epsilon = 0.3.

[0145] Table 4 Base class and new class classifier parameter settings.

[0146] Layer In_channels Out_channels Kernel_size biase Input 512 512 Conv2d 512 512 1 False Relu Conv2d 512 512 1 False Relu Conv2d 512 1 1 False

[0147] The algorithm model of the present application is written using Python 3.5, based on the deep learning framework Pytorch, and the GPU model for experiment running is 1 NVIDIA GeForce GTX 3080Ti, with a total of 16GB of video memory.

[0148] Other generalized small sample semantic segmentation methods are similar to this method.

[0149] In summary, the present application models the generalized small sample semantic segmentation problem by taking advantage of embedding semantic word vectors and mining query image information, introduces the distance between semantic word vectors to assist the model in learning the inter-class distance of class prototypes through embedding semantic word vectors, and ensures that the model can correctly distinguish between similar classes. Moreover, the present application uses a self-supplementary module to mine the information contained in the query image in the final evaluation stage, effectively alleviating the information loss problem caused by using class prototypes to represent classes, and further improving the performance of the model. The method proposed in the present application achieves good performance on the commonly used generalized small sample semantic segmentation datasets PASCAL-5i and COCO-20i, and for the first time proposes embedding semantic word vectors to assist the model in learning class prototypes, so the present application has practical application value and is worth popularizing.

[0150] The above embodiments are preferred embodiments of the present application, but the embodiments of the present application are not limited by the above embodiments, and any changes, modifications, substitutions, combinations and simplifications made without departing from the spirit and principles of the present application shall be equivalent replacement methods and shall be included in the protection scope of the present application.

Claims

1. A generalized few-shot semantic segmentation method based on word vector embedding-information self-complement, used for computer recognition of unlabeled image pixels and assignment of corresponding class labels, characterized in that, Comprise: Base class learning stage: use base class label samples to train the backbone network, semantic word vector embedding module and base class classifier, generate base class prototype; New class update stage: use one or five new class label samples to train new class classifier, generate new class prototype; Evaluation stage: input the base class prototype obtained in the base class learning stage and the new class prototype obtained in the new class update stage into the self-completion module, the self-completion module uses the base class prototype and the new class prototype to generate pseudo labels through initial prediction of query images, then extracts query prototypes in query images through pseudo labels, calculates the cosine similarity between query prototypes and query images to generate auxiliary prediction; Finally, the final prediction result is obtained by weighting the auxiliary prediction and the initial prediction; The base class learning stage, the specific process is: Input M base class label samples into the backbone network to obtain support features, and input class words into the pre-trained word2vec to obtain word vectors; Input support features, word vectors and image corresponding masks into semantic word vector embedding module to generate base class model; The evaluation stage is specifically: The base class prototype and the new class prototype input correspond to the classifier generating the initial prediction y of the query image qry ; An application threshold τ is used to select high-confidence regions to generate pseudo-labels M qry y qry Values greater than τ are set to 1 and values less than τ are set to 0, τ = 0.8; query prototype p is generated using pseudo-labels and masked global average pooling on query features query : Computing query prototypes Cosine distance between query features f q to generate auxiliary predictions y aux : The initial prediction y qry and the side prediction y aux are combined to obtain the final prediction result y final : y final = (1 - ε) * y qry + ε * y aux Wherein ε is the weight.

2. The method of claim 1, wherein, The support features, word vectors and image corresponding masks are input into the semantic word vector embedding module to generate the base class model, specifically: Mask average pooling is used on support features to obtain image prototype; Input image prototype and word vector into word vector prototype generator to produce word vector prototype; Update the word vector prototype using the newly obtained word vector prototype; Input the word vector prototype and the class prototype into the correlation estimator to obtain the adaptive weight; Use the adaptive weight to weight and integrate the word vector prototype and the class prototype to obtain the integrated prototype, that is, the base class prototype; Input the integrated prototype and image features into the base class classifier for segmentation prediction, and calculate the training loss.

3. The method of claim 2, wherein, Update the word vector model using the newly obtained word vector, specifically update the word vector under the supervision of the triple loss function, and then use the prototype integration generator to complete the integration of the word vector prototype and the class prototype.

4. The method of claim 3, wherein, Use the triple loss function to convert the word vector from the semantic space to the visual space, specifically: Set the word vector prototype as the anchor point, then sample positive and negative pairs from the general feature to calculate the triple loss, wherein the general feature is obtained by a general feature generator.

5. The method of generalized small sample semantic segmentation according to claim 4, characterized in that, The general feature generator is composed of convolution blocks, which are used to reduce the support features by 4 times to obtain general features.

6. The method of claim 1, wherein, In the base class learning stage, the loss function adopts base class segmentation loss and triplet loss, L = L seg + a * L triplet , where a = 0.2, L triplet is the triplet loss, and L seg is the base class segmentation loss; In the new class update stage, use the base class segmentation loss function.

7. The method of claim 1, wherein, Set ε = 0.

3.

8. The method of claim 2, wherein, The word vector prototype generator is composed of two fully connected layers.

Images