Open vocabulary instance segmentation method based on image title alignment and generation

Abstract

The invention discloses an open vocabulary instance segmentation method based on image title alignment and generation. An open vocabulary instance segmentation model based on image title alignment and generation is established; a loss function of model training comprises image title alignment loss and image title generation loss; in the image title alignment loss, only using a noun representing an object as an alignment object; according to the image title generation loss, a training mode based on an image title generation task is adopted, an image title predicted by a model is compared with a real image title, and then the loss is calculated; generating a mask of the object according to the potential features of the object; predicting the object category according to the potential features of the object and the similarity of the category vocabularies; and the mask and the prediction category are combined to form an instance segmentation result, namely, open vocabulary instance segmentation based on image title alignment and generation is realized. The method can improve the segmentation accuracy, is simple and efficient, and is convenient to reproduce.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, and relates to an open vocabulary instance segmentation technology of an image, and in particular to an open vocabulary instance segmentation method based on image title alignment and generation. Background Art

[0002] Instance segmentation is a core task in the field of computer vision. It is to segment and classify each object in an image. It goes beyond object detection in granularity. Although there is still a lot of research work devoted to solving the instance segmentation problem, the current solutions mainly focus on the closed semantic recognition of a predefined set of object categories. That is, for a number of predefined categories in the data set, such as 80 categories, the model predicts a category among these 80 categories for each identified instance, such as a cat or a dog.

[0003] However, in practice, many applications require detecting and segmenting new categories. In order to avoid annotating new object categories, zero-shot instance segmentation is proposed, where the zero-shot instance segmentation model has the ability to be trained on the base categories and then segment new categories. However, zero-shot instance segmentation suffers from low performance on novel categories because high-level word embeddings cannot effectively encode fine-grained visual information.

[0004] To address this problem, recent work has proposed an open vocabulary setting, which adds image-related text corpus (image captions, word embeddings of instance categories, etc.) as additional training data, so that the model has the ability to train on the basic categories and predict novel categories, and the effect is better than the zero-shot setting due to the introduction of effective additional data. Several previous methods, such as ViLD, proposed effective methods to extract knowledge from pre-trained visual language models (VLMs) into detectors. At the same time, some work decomposes the learning of open vocabulary classification and detection / segmentation into a two-stage process. Such open vocabulary detection / segmentation solutions usually try to use larger-scale datasets for pre-training, with the help of VLMs knowledge. For example, Detic uses the ImageNet-21k dataset in a weakly supervised way to expand the data range, while PromptDet uses image-caption pairs obtained from the Internet to expand the detection dataset. The recent XPM also pre-trains its model on a large-scale image caption dataset.

[0005] The disadvantages of the above methods include: 1) complex architecture design is usually required to utilize additional data sets. Although the final performance is improved, compared with the cost of obtaining these large-scale data, these methods are not cost-effective in data utilization. 2) After introducing VLMs into the model, the runtime and memory requirements of the model are greatly increased, which is not conducive to the development and utilization of real scenarios.

[0006] There are also some works, such as OVR-CNN and OpenSeg, which use image caption alignment methods to train models to link instance categories with natural language words. But these methods still have a core problem, that is, they use the alignment loss between words and masks, implying that each word (or noun) should correspond to an area in the image. However, in the caption of the image, it is obvious that not every word corresponds to an area representing an object in the image, such as `messy' and 'room'. Aligning the mask of the object with the meaningless words, this inappropriate training method leads to a decrease in the accuracy of the final instance segmentation model. Summary of the invention

[0007] In order to overcome the shortcomings of the above-mentioned prior art, the present invention proposes an open vocabulary instance segmentation method based on image caption alignment and generation to better utilize the knowledge in image caption data. The method of the present invention only uses image captions as additional data, thus avoiding the problem of data collection in pre-training using large-scale data, and does not use pre-trained visual language models (VLMs) as part of the model, thus avoiding the cost of memory and computational complexity. In addition, in the image caption alignment loss, it only uses nouns representing objects as alignment objects, thus avoiding the problem of ambiguous alignment.

[0008] The technical solution provided by the present invention is:

[0009] An open vocabulary instance segmentation method based on image caption alignment and generation. The method establishes an open vocabulary instance segmentation model based on image caption alignment and generation, referred to as a CGG (Caption Grounding and Generation) model. CGG includes a visual backbone network based on resnet-50, a multi layer transformer decoder, an image caption alignment module, an image caption generation module, an object mask prediction module, and an object category prediction module. In the image caption alignment module, the present invention improves the shortcomings of the previous methods (convolutional neural network method for open vocabulary object detection, OVR-CNN; semantic segmentation method for open vocabulary, OpenSeg), and adopts a method of using only nouns representing objects as alignment objects. The image caption generation module is newly proposed by us in the instance segmentation for open vocabulary, which allows the CGG model to predict the image caption according to the image content and the predicted object features, and then compares the predicted image caption with the real caption (original title), and then calculates the loss. The above two modules are the core innovations of this method. Training the two modules simultaneously can effectively promote the significant improvement of the accuracy indicators of the CGG model in the instance segmentation task for open vocabulary.

[0010] The method of the present invention comprises the following steps:

[0011] 1) Extract image features using the visual backbone network:

[0012] 2) Extract features of all potential objects in the image using a multi-layer transformer decoder;

[0013] 3) Align the features of the potential object with the noun representing the object in the real title (original title) of the image and calculate the loss function; including:

[0014] Align and pair all the latent features of the objects with the nouns representing the objects in the image captions, without matching all the words in the image captions;

[0015] After the alignment and pairing is completed, the alignment loss function is calculated. Through the gradient descent algorithm, the relationship between the category of the object and the natural language is established through model learning. Based on the relationship between the learned natural language and the novel category vocabulary, it is determined whether the object belongs to the novel category;

[0016] 4) Generate image captions based on image features and potential object features, and calculate loss functions with real image captions; learn the relationship between each word in the image caption and the image content; including:

[0017] The image features and the latent features of all objects are input into another multi-layer transformer decoder, which predicts and generates an image caption;

[0018] Compare the title predicted by the multi-layer transformer decoder with the true title to calculate a loss function;

[0019] The relationship between each word in the image title and the image content is learned through the gradient descent algorithm;

[0020] 5) Generate the mask of the object based on the potential object features:

[0021] 6) The category of the object is predicted based on the similarity between the potential object features and the category vocabulary.

[0022] Compared with the prior art, the present invention has the following beneficial effects:

[0023] 1) Segmentation accuracy is greatly improved:

[0024] Compared with the existing best method XPM (Open-vocabularyinstance segmentation via robust cross-modal pseudo-labeling) in the direction of open vocabulary instance segmentation, the present invention and XPM are compared on the COCO dataset. The accuracy of CGG on novel categories is improved from 21.6 of the XPM method to 28.4. This is a significant improvement in this field.

[0025] 2) The present invention can also be applied to multiple open vocabulary segmentation tasks:

[0026] Due to the robustness of the designed structure, the present invention can not only perform open vocabulary instance segmentation, but also open vocabulary panoptic segmentation and open vocabulary object detection tasks. The above two tasks have been experimentally verified and compared with the previous good work in related fields, both have achieved better results.

[0027] 3) The structure of the present invention is simple and effective, and easy to reproduce:

[0028] Compared with the previous work, the present invention has two improvements: first, only nouns representing objects are used as alignment objects; second, a new image title generation module is added. Based on the public code, it is relatively easy to make changes to reproduce the present invention.

[0029] 4) The present invention does not require the use of large visual language models (VLMs) or large pre-training datasets, and the cost in time and data annotation is significantly reduced:

[0030] Most previous works use large visual language models, which greatly increases the time complexity of the model. Some works also pre-train the model on large data sets, and labeling such data sets itself requires a lot of cost. The present invention overcomes these two shortcomings, so it is very cost-effective. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a flowchart of the present invention.

[0032] The input image is extracted from the visual backbone network to extract image features, and then the multi-layer transformer decoder extracts the features of the potential object. The potential object features are 1) trained by aligning with the nouns representing the objects in the title and calculating their loss function; 2) trained by generating image captions and comparing them with the real image captions to calculate the generated loss function. After training, the potential object features can predict masks and categories to form the final open vocabulary instance segmentation results. DETAILED DESCRIPTION

[0033] The present invention will be further described below by way of embodiments in conjunction with the accompanying drawings, but the scope of the present invention is not limited in any way.

[0034] An open vocabulary instance segmentation method based on image caption alignment and generation. During the training process, it includes an image caption alignment loss and an additional image caption generation loss. In the image caption alignment loss, we improved the shortcomings of the previous method and adopted a method that only uses nouns representing objects as alignment objects. In addition, we newly proposed a training method based on the image caption generation task, that is, let CGG generate a natural language describing the image content (i.e., image title) based on the image content and predicted object features, and then compare the title predicted by CGG with the real title (original title), and then calculate the loss. Training with the two methods at the same time can effectively promote a significant improvement in the performance of CGG. The method of the present invention comprises the following steps:

[0035] 1) Extract image features using the visual backbone network:

[0036] Input an image and its real title. First, use the visual backbone network ResNet-50 (Deepestial learning for image recognition) to extract the features of the image.

[0037] 2) Use a multi-layer transformer decoder to extract the potential features of all objects in the image:

[0038] The extracted features are input into the multi-layer transformer decoder to extract the features of all potential objects in the image. The query of the multi-layer transformer decoder is a string of randomly initialized vectors, and the key and value are image features. In this step, we extract the features of all potential objects in the image, regardless of whether these objects belong to the basic category or the novel category.

[0039] 3) Align the latent features of the object with the noun representing the object in the real title (original title) of the image:

[0040] Align and pair the features of all potential objects with the nouns representing the objects in the image captions. For example, words such as "person" and "dog" are nouns that need to be paired. The difference from the existing methods is that this method does not pair all the words in the title, but only pairs nouns representing visible objects, because other words cannot find a corresponding visible object in the image, and forced pairing causes the models of other methods to learn incorrect representations. After the alignment is completed, the alignment loss function (from Open-vocabulary object detection using captions) is calculated. Through the gradient descent algorithm, CGG learns to establish a relationship between the category of the object and the natural language. What CGG learns is no longer a meaningless label of 1 or 2 representing the category, but a semantic word such as "person" and "dog". In this way, when classifying novel categories, such as "cat", the model of the present invention can be used to judge whether an object belongs to a novel category based on the connection between the natural language and the novel category vocabulary that has been learned.

[0041] 4) Generate an image caption based on the image features and the features of all potential objects, compare it with the real image caption, and calculate the loss function (Show and tell: A neural image caption generator); learn the relationship between each word in the image caption and the image content:

[0042] The image features and the features of all potential objects are input into another multi-layer transformer decoder, which predicts and generates an image title. The title predicted by the decoder is compared with the real title to calculate the loss function. The greater the difference between the two, the greater the loss function. Through the gradient descent algorithm, the method of the present invention can fully learn the relationship between each word in the image title and the image content, and achieve interactive understanding of images and language.

[0043] 5) Generate the mask of the object based on the potential object features:

[0044] The potential object features are input into a multi-layer linear network (MLP) structure, and the masks of all objects are output.

[0045] 6) The category of the object is predicted based on the similarity between the potential object features and the category vocabulary:

[0046] The dot product of the latent object features and the features of the vocabulary representing the category (basic category vocabulary and novel category vocabulary) is performed to obtain the similarity between the latent object features and the category vocabulary; the largest result indicates that the latent object feature has the greatest similarity with the corresponding category, and CGG determines that the category of the object feature is that category.

[0047] 7) Combine the mask and predicted category together to form the final instance segmentation result:

[0048] Instance segmentation requires the segmentation of the mask and category of each foreground object in the image. Step 5) obtains the mask of the object, and step 6) obtains the category prediction of the object. CGG combines the two results according to the number of the potential object features to obtain the mask and category of each potential object. Then, the predictions with overlapping masks are deleted to obtain the mask and predicted category of instance segmentation, completing the task of instance segmentation.

[0049] A specific implementation method for realizing the present invention is provided below.

[0050] Based on Mask2Former, this method is an instance segmentation method from Masked-attentionmask transformer for universal image segmentation. First, its training and testing methods are changed to a classification architecture that supports open vocabulary. That is, the model predicts the category of the object not as a simple category number, but as the category representation of the object. After calculating the similarity between the representation and the category language feature, the largest one is taken as the category of the object. In this way, the category set for each classification is variable. Whether it is 80 classes or 800 classes, the most similar one can be found to predict its category, achieving the purpose of open vocabulary. To achieve this, you only need to change the classification head of the model to predict a category representation, and the category representation should be aligned with the language representation. During training, it should be allowed to calculate the loss function based on similarity with the language representation of the corresponding category.

[0051] The second step is to add the object noun title alignment module of the present invention. First, we extract the titles of the images in the training data, and then extract the nouns representing the objects from the titles. This step can be implemented based on the category names of the LVIS dataset. All the words in the image titles that contain the LVIS dataset category names and their synonyms are extracted, and then the nouns representing the objects are extracted according to the attached Figure 1 As shown in the figure, the extracted nouns are used together with the latent object features extracted by the model through the multi-layer transformer decoder to calculate the alignment loss function, so that the latent object features will only be aligned with the nouns representing the objects, which is more in line with the requirements of open vocabulary instance segmentation.

[0052] After the second step, the model has cross-modal capabilities and can efficiently complete the task of open vocabulary instance segmentation with high accuracy. The third step is to add an additional module, namely the title generation module. We use a multi-layer transformer decoder as a title generator with 4 layers. Experiments show that increasing or decreasing the number of layers is not beneficial to improving the performance of the model. Input the latent object features into the module, and the module can generate an image title. Then compare the generated title with the real title word by word, calculate a loss function, and then train the module.

[0053] After the object-noun-title alignment module and the title generation module are trained simultaneously, the model is trained and can be run for testing. The expected results are compared with the previous methods on the COCO dataset, and the results are shown in Table 1:

[0054] Table 1 Accuracy comparison

[0055]

[0056] Table 1 shows that the accuracy of the method of the present invention exceeds that of the previous four methods in the tests of basic categories and novel categories separately and in the tests of basic categories and novel categories combined. This method is currently the most accurate open vocabulary instance segmentation method on the COCO dataset.

Claims

1. An open vocabulary instance segmentation method based on image caption alignment and generation, which is characterized by, Build an open vocabulary instance segmentation model CGG based on image caption alignment and generation; including: Visual backbone network, multi-layer transformer decoder, image caption alignment module, image caption generation module, object mask prediction module, object category prediction module; During the model training process, the loss function of the CGG model includes image caption alignment loss and image caption generation loss. In the image caption alignment loss, only nouns representing objects are used as alignment objects. The image caption generation loss adopts a training method based on the image caption generation task, that is, the natural language describing the image content, i.e., the image caption, is generated based on the image content and the predicted object features, and then the image caption predicted by the model is compared with the real image caption, and then the loss is calculated. The method comprises the following steps: 1) Extract image features using the visual backbone network: 2) Extract latent features of all objects in the image using a multi-layer transformer decoder; all objects include those belonging to base classes and novel classes; 3) Align the potential features of the object with the noun representing the object in the real title of the image and calculate the loss function; including: Align and pair all the latent features of the objects with the nouns representing the objects in the image captions, without matching all the words in the image captions; After the alignment and pairing is completed, the alignment loss function is calculated. Through the gradient descent algorithm, the relationship between the category of the object and the natural language is established through model learning. Based on the relationship between the learned natural language and the novel category vocabulary, it is determined whether the object belongs to the novel category; 4) Generate an image title based on the image features and the potential features of the object, and calculate the loss function with the real image title; learn the relationship between each word in the image title and the image content; including: The image features and the latent features of all objects are input into another multi-layer transformer decoder, which predicts and generates an image caption; Compare the title predicted by the multi-layer transformer decoder with the true title to calculate a loss function; The relationship between each word in the image title and the image content is learned through the gradient descent algorithm; 5) Generate a mask of the object based on the potential features of the object; 6) Predict the category of the corresponding object based on the potential features of the object and the similarity of the category vocabulary; 7) Combine the mask and predicted category together to form the final instance segmentation result: Through the above steps, open vocabulary instance segmentation based on image-caption alignment and generation is achieved.

2. The open vocabulary instance segmentation method based on image title alignment and generation as claimed in claim 1, characterized in that: The visual backbone network is specifically based on the visual backbone network of resnet-50.

3. The open vocabulary instance segmentation method based on image title alignment and generation as claimed in claim 1, characterized in that: The image caption alignment module is specifically based on the convolutional neural network method OVR-CNN for open vocabulary object detection or the semantic segmentation method OpenSeg for open vocabulary, and only uses nouns representing objects as alignment objects.

4. The open vocabulary instance segmentation method based on image title alignment and generation as claimed in claim 1, characterized in that: The specific process of image title alignment is: First, extract the captions of the images in the training data; Then, nouns representing objects are extracted from the title, which can be achieved based on the category names of the LVIS dataset; Extract all the words in the image title that contain the LVIS dataset category name and its synonyms; The extracted nouns are combined with the latent object features extracted by the multi-layer transformer decoder to calculate the alignment loss function, so that the latent object features are only aligned with the nouns representing objects.

5. The open vocabulary instance segmentation method based on image title alignment and generation as claimed in claim 1, characterized in that: Step 5) specifically inputs the potential object features into the multi-layer linear network MLP structure and outputs the masks of all objects.

6. The open vocabulary instance segmentation method based on image title alignment and generation as claimed in claim 1, characterized in that: Step 6) specifically performs a dot product of the latent object feature and the feature of to obtain the similarity between the latent object feature and the category vocabulary; when the latent object feature has the greatest similarity with the corresponding category, the category of the object feature is judged to be the corresponding category, that is, the category of the object is predicted based on the similarity between the latent object feature and the category vocabulary; wherein the vocabulary representing the category includes basic category vocabulary and novel category vocabulary.

7. The open vocabulary instance segmentation method based on image title alignment and generation as claimed in claim 1, characterized in that: Step 7) specifically combines the mask and the predicted category according to the number of the potential object features to obtain the mask and category of each potential object; then deletes the predictions with overlapping masks to obtain the mask and predicted category of instance segmentation, thereby completing the instance segmentation task.

8. The open vocabulary instance segmentation method based on image caption alignment and generation as claimed in claim 1, characterized in that: Specifically, a multi-layer transformer decoder with 4 layers is used as the header generator.

Images