An open vocabulary segmentation method based on a multi-base large model

Abstract

The application discloses an open vocabulary segmentation method based on a multi-base large model, and comprises the following steps: step one, training on panoramic segmentation training set samples after establishing two interaction modes on CLIP and SAM image features, and obtaining an open vocabulary segmentation model; and step two, adopting a SAM uniform sampling point prompt generated mask prediction to correct the final open vocabulary segmentation result in the test stage. The application uses CLIP as a main network for image coding, uses SAM as an auxiliary model to provide fine-grained feature understanding ability, and increases the generalization of the open vocabulary segmentation model to the generated mask proposal through the interaction between the features of the base model, so that the pre-trained base model is applied to downstream fine-grained semantic understanding and recognition tasks. The application injects the features of SAM into the internal Transformer decoder and interacts and learns with the features of CLIP, so that the result of the mask query is more accurate.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and relates to an open vocabulary segmentation method, specifically an open vocabulary segmentation method based on a multi-foundation large model. Background Technology

[0002] Open-vocabulary segmentation is a learning problem that matches real-world scene understanding. It focuses on the model's segmentation performance for both known and unknown classes of new objects in the application scenario. Specifically, the model is trained on a segmentation dataset with a limited number of classes. This allows the model to acquire the ability to recognize not only the classes seen during training (those within the vocabulary) and their segmentation masks, but also the ability to recognize classes not seen during training (those outside the vocabulary) and their segmentation masks, thus providing open-aware segmentation capabilities for different test scenarios.

[0003] However, most traditional segmentation methods are designed and trained only on specific datasets or segmentation tasks, ultimately limiting their ability to segment and recognize only a limited number of categories encountered during training. This significantly restricts the application of segmentation models in real-world scenarios. With the emergence of multiple large-scale foundational models, pre-trained on vast amounts of data, they possess more generalized knowledge. However, due to the singular nature of their pre-training tasks, they cannot effectively generalize to downstream tasks, such as fine-grained category-aware segmentation. Therefore, utilizing multiple large-scale foundational models simultaneously can complement their fundamental capabilities, addressing both task-type generalization and category-aware generalization. Summary of the Invention

[0004] To address the limitation of downstream adaptability caused by the gap between the pre-trained classification task and the downstream segmentation task in existing open vocabulary segmentation methods based on a single base model, this invention introduces an additional segmentation base model to facilitate feature interaction between models, providing an open vocabulary segmentation method based on multiple base models.

[0005] The objective of this invention is achieved through the following technical solution:

[0006] An open vocabulary segmentation method based on a multi-foundation large model includes the following steps:

[0007] Step 1: After establishing two interaction methods for CLIP and SAM image features, train the model on the panoptic segmentation training set to obtain an open vocabulary segmentation model. The specific steps are as follows:

[0008] Step 11: Construct an interaction framework. The interaction framework uses CLIP image encoder as the backbone image encoder of Mask2Former and SAM image encoder as the auxiliary image encoder of Mask2Former.

[0009] Steps 1 and 2: Construct a fusion module, which includes a feature injector and a query injector. For the l-th layer of the Transformer decoder, the feature injector contains a multi-head cross-attention layer. Features from the CLIP layer are used as queries, and SAM features are used as keys and values, outputting features. The query injector contains a mask pooling layer and a feature mapping layer. The feature mapping layer consists of a LayerNorm layer and a linear mapping layer. Finally, it outputs the mask query x of SAM at this layer. l ;

[0010] Step 13: During training, each training image sample is processed by a CLIP image encoder and a pixel decoder to obtain multi-layer CLIP image features, and then by a SAM image encoder and a pixel decoder to obtain SAM image features.

[0011] Step 1, Part 4: Multi-layer CLIP image features and SAM image features interact with each other through cross-attention within the inner layers of the Transformer decoder;

[0012] Step 15: Initialize the k mask query vectors of Mask2Former. When the Transformer decoder passes through the inner layer, the mask prediction of the intermediate layer is obtained from the output of the inner layer. The mask prediction is used to depool the SAM image features to obtain k SAM query vectors, which are then added to the original mask query vectors after passing through this layer.

[0013] Step 16: The k SAM query vectors interact with the multi-layer CLIP image features inside the decoder. At the end of each layer of the Transformer decoder, a mask prediction is obtained through mapping, and the cosine similarity score with the training class text vectors output by the CLIP text encoder is calculated as the class prediction. The Hungarian matching algorithm is used to match the corresponding label to each query vector.

[0014] Step 17: For each layer of mask prediction and mask label, calculate the Dice loss and Sigmoid cross-entropy loss. For class prediction and class label, calculate the cross-entropy loss. Minimize the total loss during training. Use the backpropagation algorithm to optimize the parameters of the k query vectors, Transformer decoder and pixel decoder to obtain the optimized query vectors and the trained open vocabulary segmentation model.

[0015] Step 2: During the testing phase, the mask prediction generated by SAM uniform sampling point prompts is used to correct the final open vocabulary segmentation result. The specific steps are as follows:

[0016] Step 2: 1. Feed the test data into the trained open vocabulary segmentation model to obtain the mask prediction set of all mask query vectors. and category prediction set

[0017] Step 22: After binarizing the mask prediction, pool the largest feature map from the CLIP multi-layer features and calculate the similarity score with the CLIP-encoded text features to obtain the CLIP's class prediction for the mask.

[0018] Steps 2 and 3: Sample points evenly across the test image according to a grid, obtain cue vectors via the SAM cue encoder, and interact with the SAM image features in the mask decoder to obtain the SAM mask prediction set. n represents the number of SAM prediction results;

[0019] Step 24: Use the SAM mask to predict the image features of the binarized pooled CLIP, and calculate the similarity score with the CLIP-encoded text features to obtain the SAM category prediction set.

[0020] Step 2.5: Final mask prediction set based on open vocabulary segmentation model, CLIP, and SAM. and category prediction set Obtain the final open vocabulary segmentation result.

[0021] Compared with the prior art, the present invention has the following advantages:

[0022] 1. This invention uses the visual-language multimodal pre-trained basic model CLIP as the main network for image encoding, and uses the visual cue segmentation model SAM as an auxiliary model to provide fine-grained feature understanding capabilities. By increasing the generalization of the open vocabulary segmentation model to generate mask proposals through the interaction between features of the basic model, the pre-trained basic model can be applied to downstream fine-grained semantic understanding and recognition tasks.

[0023] 2. This invention freezes the parameters of two basic models, CLIP and SAM, and proposes two important feature fusion methods. These methods fuse features in mask lookup and segmentation image features respectively. The features of SAM are injected into the Transformer decoder to interact and learn with the features of CLIP, resulting in more accurate mask lookup results. This invention does not require additional training of image and text encoders, thus significantly reducing training resource consumption.

[0024] 3. This invention utilizes the mask prediction set generated by the grid point sampling prompts of SAM to correct the semantic segmentation results of the open vocabulary. The mask prediction of SAM itself does not have semantic information. This invention integrates the semantic information of SAM mask prediction and the mask pooling of corresponding CLIP image features as the semantic mask prediction of SAM, and integrates it with the segmentation results of the trained model as a supplement to the mask prediction results of out-of-vocabulary categories. Attached Figure Description

[0025] Figure 1 This is the overall framework of the open vocabulary segmentation method based on a multi-foundation large model in this invention;

[0026] Figure 2 It is a network structure that integrates two feature modules;

[0027] Figure 3 This is a flowchart illustrating the process of fusing SAM mask prediction during the inference stage to obtain the final semantic segmentation prediction.

[0028] Figure 4 This is a comparison of the segmentation results of the method of this invention and the benchmark method on the panoramic segmentation dataset;

[0029] Figure 5 This is a comparison of the segmentation results of the method of this invention and the benchmark method on a semantic segmentation dataset. Detailed Implementation

[0030] The technical solution of the present invention will be further described below with reference to the accompanying drawings, but it is not limited thereto. Any modifications or equivalent substitutions to the technical solution of the present invention that do not depart from the spirit and scope of the technical solution of the present invention should be covered within the protection scope of the present invention.

[0031] This invention provides an open vocabulary segmentation method based on a multi-foundation large model, such as... Figure 1 As shown, the method includes the following steps:

[0032] Step 1: After establishing two interaction methods for CLIP and SAM image features, train the model on the panoptic segmentation training set to obtain an open vocabulary segmentation model. The specific steps are as follows:

[0033] Step 11: Construct an interaction framework. The interaction framework uses CLIP image encoder as the backbone image encoder of Mask2Former and SAM image encoder as the auxiliary image encoder of Mask2Former.

[0034] Steps one and two: Constructing the fusion module, which includes a feature injector and a query injector, such as... Figure 2 As shown, for the l-th layer of the Transformer decoder, the feature injector contains a multi-head cross-attention layer. The features of the CLIP layer are used as queries, and the SAM features are used as keys and values. The output features are... The query injector contains a mask pooling layer and a feature mapping layer. The feature mapping layer consists of a LayerNorm layer and a linear mapping layer. Finally, it outputs the mask query x of SAM at this layer. l .

[0035] Step 13: During training, each training image sample obtains multi-layer CLIP image features through a CLIP image encoder and a pixel decoder, and obtains SAM image features through a SAM image encoder and a pixel decoder.

[0036] In this step, the method for obtaining SAM image features through SAM image encoder and pixel decoder is as follows: each training sample is processed by SAM image encoder to obtain SAM encoded features, which are then resized by one convolutional layer and then processed by multiple deformable convolutional layers to obtain SAM image features.

[0037] Step 1, Part 4: Multi-layer CLIP image features and SAM image features interact with each other through cross-attention within the inner layers of the Transformer decoder.

[0038] like Figure 1 and Figure 2 As shown on the left, the specific method is as follows: the features of CLIP layer are used as the query, and the image features of SAM are used as the key and value. At the beginning of each layer of the Transformer decoder, the interaction features are obtained by passing through a cross-attention layer. The mask query and the interaction features are then cross-attention learned.

[0039] Step 1, Fifth: Initialize the k mask query vectors of Mask2Former. When the Transformer decoder passes through the inner layer, the mask prediction of the intermediate layer is obtained from the output of the inner layer. The mask prediction is used to depool the SAM image features to obtain k SAM query vectors, which are then added to the original mask query vectors after passing through this layer.

[0040] like Figure 1 and Figure 2 As shown on the right, the specific method is as follows: k mask query vectors are multiplied with the largest feature map of CLIP at the end of each layer of the Transformer decoder to obtain k mask predictions of the intermediate layer; the k mask predictions are converted into binary masks according to the threshold of 0, and the SAM image features are pooled to obtain k SAM mask query vectors, which are then directly added to the k original query vectors.

[0041] Step 16: The k SAM query vectors interact with the multi-layer CLIP image features within the decoder. At the end of each layer of the Transformer decoder, a mask prediction is obtained through mapping, and the cosine similarity score with the training category text vectors output by the CLIP text encoder is calculated as the category prediction. The Hungarian matching algorithm is used to match the corresponding label to each query vector.

[0042] Step 17: For each layer's mask prediction and mask label, calculate the Dice loss and Sigmoid cross-entropy loss; for class prediction and class label, calculate the cross-entropy loss. Minimize the total loss during training. Use the backpropagation algorithm to optimize the parameters of the k query vectors, Transformer decoder, and pixel decoder to obtain the optimized query vectors and the trained open vocabulary segmentation model.

[0043] In this step, the Dice loss for mask prediction and mask label calculation at each layer is... Sigmoid cross-entropy loss They are respectively:

[0044]

[0045] In the formula This represents the binary mask label assigned to mask query i, where σ represents the Sigmoid activation function; m i This indicates the mask prediction for mask query i.

[0046] In this step, the cross-entropy loss is calculated for each layer's category prediction and category label. for:

[0047]

[0048] In the formula This indicates the number of classes in the training dataset. This represents the value of the category label assigned to i in class c using a mask query, p i This represents the category prediction for mask query i.

[0049] In this step, the final total training loss for:

[0050]

[0051] In the formula, L represents the number of layers in the Transformer decoder.

[0052] Step 2: During the testing phase, the mask prediction generated by SAM uniform sampling point prompts is used to correct the final open vocabulary segmentation result. The specific steps are as follows:

[0053] Step 2: 1. Feed the test data into the trained open vocabulary segmentation model to obtain the mask prediction set of all mask query vectors. and category prediction set

[0054] Step 22: After binarizing the mask prediction, pool the largest feature map from the CLIP multi-layer features and calculate the similarity score with the CLIP-encoded text features to obtain the CLIP's class prediction for the mask.

[0055] Steps 2 and 3: Sample points evenly across the test image according to a grid, obtain cue vectors via the SAM cue encoder, and interact with the SAM image features in the mask decoder to obtain the SAM mask prediction set. n represents the number of mask results predicted by SAM.

[0056] Step 24: Use the SAM mask to predict the image features of the binarized pooled CLIP, and calculate the similarity score with the CLIP-encoded text features to obtain the SAM category prediction set. n represents the number of category results predicted by SAM.

[0057] like Figure 3 As shown, the specific method for calculating the similarity score is as follows: SAM's mask prediction is converted into a binary mask based on a threshold of 0. After pooling the largest feature map of CLIP, the semantic vector is obtained. Cosine similarity is then calculated between this semantic vector and the text encodings of all categories output by the CLIP text encoder to obtain the category score.

[0058] Step 2.5: Final mask prediction set based on open vocabulary segmentation model, CLIP, and SAM. and category prediction set The final open vocabulary segmentation result is obtained. The specific method is as follows:

[0059] Predict the model category p i and CLIP category prediction p' i The category prediction for masked query i is obtained by weighting the index values ​​according to whether the category is within the vocabulary:

[0060]

[0061] In the formula express The value at the j-th position, C train These are the in-vocabulary categories seen in the training set, and α and β are different exponential weights.

[0062] For semantic segmentation tasks, all category predictions Multiplying the mask prediction m by matrix yields the semantic prediction graph r, which is the set of category predictions for SAM. and mask prediction set Perform matrix multiplication to obtain the masked prediction semantic graph of SAM.

[0063]

[0064] For categories outside the vocabulary, r and The values ​​for the corresponding categories are summed according to their weights to obtain the final semantic prediction score. For categories within the vocabulary, the value for the corresponding category r is used as the final semantic prediction score.

[0065]

[0066] For panoramic segmentation tasks, The category score prediction is used as the mask query i, and then the segmentation result corresponding to each mask query is obtained directly using the Mask2Former method.

[0067] In this invention, the CLIP and SAM models are loaded with pre-trained loss weights, which are derived from the official weights released by CLIP and SAM, respectively. The mask query vector, Transformer decoder, and pixel decoder all need to be initialized. During the training phase, the CLIP image, CLIP text encoder, and SAM image encoder are frozen to complete the training of the mask query vector and the two decoders.

[0068] In this invention, each category name c is combined with multiple text descriptors to form category text descriptors such as 'a photo of a{c}' and 'This is a photo of a{c}'. After being fed into the CLIP text encoder, the descriptor vector set is averaged and standardized to obtain the category text vector T. c ; Mask prediction m for each mask query i i The pooling vector F is obtained by pooling the maximum feature map of CLIP. i Finally, the category prediction c is obtained. i =cos <T i， F i >, cos<·, ·> represent the cosine similarity between two vectors.

[0069] This invention was trained on the training set of the COCO panorama dataset and tested on multiple datasets. In the open-vocabulary panorama segmentation task, the test datasets included the COCO panorama validation set, Cityscapes, Mapillary Vistas, and BDD 100K. In the open-vocabulary semantic segmentation task, the test datasets covered PASCAL-Context (PC-459, containing 459 categories), PASCAL VOC (PAS-21, containing 21 categories), and ADE20K (A-847, containing 847 categories). As shown in Tables 1 and 2, the results of this invention on various datasets for both open-vocabulary panorama segmentation and semantic segmentation tasks are superior to existing methods. Figure 4 and Figure 5 The segmentation results of the method of this invention and the benchmark method are shown on the panoptic segmentation dataset and the semantic segmentation dataset, respectively.

[0070] Table 1: Performance of different methods on different datasets in the open vocabulary panoptic segmentation task

[0071]

[0072] Table 2: Performance of different methods on different datasets for open vocabulary semantic segmentation tasks

[0073] method PC-459 PAS-21 A-847 MaskCLIP 10.0 - 8.2 Ovseg 12.4 - 9.0 Cat-Seg - - 12.4 SAN 17.1 - 13.0 MasQCLIP 18.2 - 10.7 FC-CLIP 18.2 81.8 14.4 This method 19.7 82.5 14.8

Claims

1. An open vocabulary segmentation method based on a multi-foundation large model, characterized in that... The method includes the following steps: Step 1: After establishing two interaction methods for CLIP and SAM image features, train the model on the panoptic segmentation training set to obtain an open vocabulary segmentation model. The specific steps are as follows: Step 11: Construct an interaction framework. The interaction framework uses CLIP image encoder as the backbone image encoder of Mask2Former and SAM image encoder as the auxiliary image encoder of Mask2Former. Steps one and two: Constructing the fusion module, which includes a feature injector and a query injector, for the Transformer decoder... The feature injector layer contains a multi-head cross-attention layer. Features from the CLIP layer are used as queries, and SAM features are used as keys and values, outputting features. The query injector contains a mask pooling layer and a feature mapping layer. The feature mapping layer contains a LayerNorm layer and a linear mapping layer. Finally, it outputs the mask query of SAM at this layer. ; Step 13: During training, each training image sample is processed by a CLIP image encoder and a pixel decoder to obtain multi-layer CLIP image features, and then by a SAM image encoder and a pixel decoder to obtain SAM image features. Step 1, Part 4: Multi-layer CLIP image features and SAM image features interact with each other through cross-attention within the inner layers of the Transformer decoder; Step 15: Initialize Mask2Former Each mask query vector, when passed through the Transformer decoder, yields the mask prediction for the intermediate layer from the output of the inner layer. This mask prediction is then used to depool the SAM image features to obtain... Each SAM query vector is added to the original mask query vector passed through this layer; Step 16 Each SAM query vector interacts with multiple CLIP image features within the decoder. At the end of each layer of the Transformer decoder, a mask prediction is obtained through mapping, and the cosine similarity score with the training class text vectors output by the CLIP text encoder is calculated as the class prediction. The Hungarian matching algorithm is used to match the corresponding label to each query vector. Step 17: For each layer's mask prediction and mask label, calculate the Dice loss and Sigmoid cross-entropy loss; for class prediction and class label, calculate the cross-entropy loss. Minimize the total loss during training by using the backpropagation algorithm. The parameters of the query vector, Transformer decoder, and pixel decoder are optimized to obtain the optimized query vector and the trained open vocabulary segmentation model. Step 2: During the testing phase, the mask prediction generated by SAM uniform sampling point prompts is used to correct the final open vocabulary segmentation result. The specific steps are as follows: Step 2:

1. Feed the test data into the trained open vocabulary segmentation model to obtain the mask prediction set of all mask query vectors. and category prediction set ; Step 22: After binarizing the mask prediction, pool the largest feature map from the CLIP multi-layer features and calculate the similarity score with the CLIP-encoded text features to obtain the CLIP's class prediction for the mask. ; Steps 2 and 3: Sample points evenly across the test image according to a grid, obtain cue vectors via the SAM cue encoder, and interact with the SAM image features in the mask decoder to obtain the SAM mask prediction set. , This represents the number of results predicted by SAM; Step 24: Use the SAM mask to predict the image features of the binarized pooled CLIP, and calculate the similarity score with the CLIP-encoded text features to obtain the SAM category prediction set. ; Step 2.5: Final mask prediction set based on open vocabulary segmentation model, CLIP, and SAM. and category prediction set This yields the final open vocabulary segmentation result.

2. The open vocabulary segmentation method based on a multi-foundation large model according to claim 1, characterized in that... In steps one and three, the method for obtaining SAM image features through the SAM image encoder and pixel decoder is as follows: each training sample is processed by the SAM image encoder to obtain the SAM encoded features, which are then resized by a convolutional layer and then processed by multiple deformable convolutional layers to obtain the SAM image features.

3. The open vocabulary segmentation method based on a multi-foundation large model according to claim 1, characterized in that... The specific method of step one and four is as follows: use the layer features of CLIP as the query, the image features of SAM as the key and value, and at the beginning of each layer of the Transformer decoder, first go through a layer of cross attention to obtain the interaction features, and then perform cross attention learning with the mask query and the interaction features.

4. The open vocabulary segmentation method based on a multi-foundation large model according to claim 1, characterized in that... The specific method for step one five is as follows: Each mask query vector is multiplied by the largest feature map of CLIP at the end of each layer of the Transformer decoder to obtain the intermediate layer. Predicting a mask; Each mask prediction is converted into a binary mask based on a threshold of 0, and then the SAM image features are obtained through pooling. Each SAM mask query vector, and The original query vectors are directly added together.

5. The open vocabulary segmentation method based on a multi-foundation large model according to claim 1, characterized in that... In step one seven, the Dice loss for mask prediction and mask label calculation at each layer is... Sigmoid cross-entropy loss They are respectively: In the formula Indicates mask lookup The assigned binary mask label, This represents the Sigmoid activation function; Indicates mask lookup Mask prediction.

6. The open vocabulary segmentation method based on a multi-foundation large model according to claim 1, characterized in that... In step one seven, the cross-entropy loss for category prediction and category label calculation at each layer is... for: In the formula This indicates the number of classes in the training dataset. Indicates mask lookup The assigned category label is in Values ​​on a class Indicates mask lookup Category prediction.

7. The open vocabulary segmentation method based on a multi-foundation large model according to claim 1, characterized in that... In step one seven, the final total training loss for: In the formula , , These are Dice loss, Sigmoid cross-entropy loss, and cross-entropy loss, respectively. This indicates the number of layers in the Transformer decoder.

8. The open vocabulary segmentation method based on a multi-foundation large model according to claim 1, characterized in that... In step two of the above steps, the specific method for calculating the similarity score is as follows: the SAM mask prediction is converted into a binary mask based on a threshold of 0. After pooling the largest feature map of CLIP, the semantic vector is obtained. Cosine similarity is then calculated between this semantic vector and the text encodings of all categories output by the CLIP text encoder to obtain the category score. .

9. The open vocabulary segmentation method based on a multi-foundation large model according to claim 1, characterized in that... The specific method for step two five is as follows: Predicting model categories CLIP category prediction The mask lookup is obtained by weighting the index values ​​according to whether the category is within the vocabulary. Category prediction: In the formula express The The value at each position, These are the in-vocabulary categories seen in the training set. and These are different index weights; For semantic segmentation tasks, all category predictions and mask prediction Matrix multiplication yields the semantic prediction graph. The category prediction set of SAM and mask prediction set Perform matrix multiplication to obtain the masked prediction semantic graph of SAM. : For categories outside the vocabulary, and Values ​​in the corresponding category are weighted The summation serves as the final semantic prediction score for categories within the vocabulary. The value corresponding to the category is used as the final semantic prediction score: For panoramic segmentation tasks, As a mask query The category scores are predicted, and then the segmentation result corresponding to each mask is obtained directly using the Mask2Former method.

Images