Open world semantic segmentation automatic concept completion method based on pre-training

Abstract

The invention discloses an open world semantic segmentation automatic concept completion method based on pre-training. Comprising the steps of calculating image coding features of a teacher model and a student model, and transmitting knowledge of the teacher model to the student model through image feature distillation; a preset visual language pre-training model is used, similarity comparison is carried out on image coding features generated by the large-scale image-text data and image coding features generated by the teacher model, sub-titles are generated, automatic concept completion is carried out, and a completed title is obtained; the student model is promoted to realize cross-modal semantic information alignment by utilizing the coding feature of the completed title, the coding feature of the subtitle and image coding feature comparison loss generated by the student model; and performing effective segmentation and identification on the image by using the trained student model. Through image feature distillation, it is ensured that the image features and the text features are effectively aligned in the same semantic space. And automatic concept completion is used, so that the consistency and accuracy of the text labels are ensured.

Description

Technical Field

[0001] The present invention belongs to the fields of computer vision, deep learning and natural language processing, and more specifically, to an open-world semantic segmentation automatic concept completion method based on pre-training. Background Art

[0002] The success of vision-language pre-training (VLP) models has greatly promoted the development of text-supervised open-world semantic segmentation, which exploit the correlation between images and text for segmentation. However, current models mainly rely on large-scale web-based image-text datasets, which have limited text descriptions and high training cost for task-specific text generators. In addition, transferring open-world semantic knowledge from pre-trained VLP models to semantic segmentation tasks faces challenges in patch grouping and cross-modal fine-grained semantic alignment.

[0003] As one of the specific implementations of the VLP model, the existing CLIP-based open-world semantic segmentation models usually use retraining for patch self-grouping GroupViT, or use a mask generator to fine-tune the semantic alignment module for semantic classification OVSeg. Specifically, GroupViT performs patch self-grouping by retraining, divides the image into several fragments, and extracts and processes features for each fragment. OVSeg uses a mask generator to fine-tune the semantic alignment module and achieves semantic classification by generating masks. In addition, in order to improve the quality of text supervision information, previous methods often solve the ambiguity problem of text supervision information by introducing a semantic dictionary built by LLM (such as ChatGPT). However, the retrained CLIP-like framework may destroy the original semantic alignment relationship, resulting in poor performance of the model when dealing with unseen new categories; on the other hand, the image-level semantic alignment based on CLIP is not suitable for fine-grained semantic segmentation tasks, and the fine-tuning process may introduce noise and affect the segmentation accuracy. In addition, the cost of building and maintaining a semantic dictionary through LLM (such as ChatGPT) is high, and it cannot effectively solve the label missing and standardization problems in the dataset.

[0004] The invention patent with the prior art publication number CN118072332A proposes a self-evolving zero-shot target recognition method based on dual prompts of sketches and text. The method includes using CLIP as the backbone network, designing a visual prompt module and a text prompt module in the DP-CLIP framework, using the Transformer layer to extract category knowledge from the image and transfer it to a set of learnable vectors, and then using these vectors as visual prompts for the category, using CLIP's text encoder to embed the target category label into the text features, using these text features to generate channel scaling vectors, applying the scaling vectors to the hidden features of the CLIP visual encoder, and guiding the model to focus on the channels related to the current category. The invention uses a retrained CLIP-like framework that may destroy the original semantic alignment relationship, resulting in poor performance of the model when dealing with new categories that have not been seen. In addition, the cost of building and maintaining a semantic dictionary is high, and it cannot effectively solve the problems of missing labels and standardization in the data set. Summary of the invention

[0005] The present invention provides an open-world semantic segmentation automatic concept completion method based on pre-training to improve fine-grained segmentation capabilities, reduce the impact of noise, and enhance the performance of the model when processing new categories that have not been seen before; solve the problems of label missing and standardization, reduce costs, and improve the quality of text supervision information and the applicability of the model.

[0006] The present invention provides an open world semantic segmentation automatic concept completion method based on pre-training, comprising the following steps:

[0007] Calculate the image encoding features of the teacher model and the student model, and transfer the knowledge of the teacher model to the student model through image feature distillation, thereby enhancing the student model's grasp of global semantic information;

[0008] Using the preset visual language pre-training model, the image encoding features generated by large-scale image-text data are compared with the image encoding features generated by the teacher model to generate subtitles and perform automatic concept completion to obtain the completed title;

[0009] The encoding features of the title after automatic concept completion, the title encoding features of the subtitle generated by the visual language pre-training model, and the contrast loss of the image encoding features generated by the student model are used to promote the student model to achieve cross-modal semantic information alignment.

[0010] The trained student model is used to effectively segment and recognize images.

[0011] Furthermore, the preset visual language pre-training model is a CLIP model.

[0012] Furthermore, the image feature distillation comprises the following steps:

[0013] Use the image encoder of the vision-language pre-trained model as the teacher model to calculate the image encoding features; use GoupViT's segment generator as the student model to segment the image into multiple segment tokens and calculate the image encoding features;

[0014] The image encoding features of the teacher model and the student model are normalized separately, and then their similarity is measured by calculating their total distillation contrast loss;

[0015] The total distillation contrastive loss is used to improve the student model so that the features it generates are closer to those generated by the teacher model.

[0016] Furthermore, the parameters of the teacher model are fixed.

[0017] Furthermore, the step of calculating the image coding features of the student model comprises the following steps:

[0018] Use GoupViT’s image encoder as a segment generator to segment the image into multiple segment tokens;

[0019] Perform an average pooling operation on the segmented multiple segment tokens;

[0020] The pooling results are transformed through a multi-layer perceptron to obtain the encoded features.

[0021] Furthermore, the formula for the total distillation contrast loss is:

[0022]

[0023] Among them, L is the contrast loss function used, I is the image, and I s and I t denote the image feature embeddings encoded by the student model and the teacher model, respectively.

[0024] Furthermore, the automatic concept completion comprises the following steps:

[0025] Use large-scale image-text data to construct label sets, including basic label sets and open label sets;

[0026] Use the visual language pre-trained model text encoder to encode the label set to obtain feature encoding;

[0027] Calculate the cosine similarity between the feature code and the image feature code generated by the teacher model, and take out the label with high similarity as a supplementary label;

[0028] Generate subtitles by combining supplementary tags with prompting engineering in visual language pre-training models;

[0029] The generated sub-captions and the original caption are merged into an overall caption, which is then trained to align with the corresponding image semantic information.

[0030] Furthermore, there is no intersection between the basic tag set and the open tag set, and the open tag set is not fixed.

[0031] Furthermore, the parameters of the text encoder of the visual language pre-training model are fixed.

[0032] Furthermore, the cross-modal semantic information alignment is implemented, comprising the following steps:

[0033] Calculate the encoding features of the title after automatic concept completion, the title encoding features of the subtitle generated by the visual language pre-training model, and the image encoding features generated by the student model;

[0034] The contrast loss between the encoding features of the title after automatic concept completion, the title encoding features of the subtitle generated by the visual language pre-training model, and the image encoding features generated by the student model is calculated respectively to obtain two cross-modal semantic alignment contrast losses;

[0035] Calculate the total loss, which is the sum of the total distillation contrast loss and the two cross-modal semantic alignment contrast losses;

[0036] The total loss is used to promote the student model to achieve cross-modal semantic information alignment.

[0037] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0038] The present invention directly uses the pre-trained VLP model to perform image feature distillation, and uses contrast loss to promote the model to achieve cross-modal semantic information alignment, ensuring that image features and text features are effectively aligned in the same semantic space, reducing the impact of noise, improving segmentation accuracy, and improving the performance of the model when processing new categories that have not been seen. The automatic concept completion module uses the pre-trained VLP model to complete missing or ambiguous concept information, ensure the consistency and accuracy of text labels, solve the problem of missing labels and standardization, reduce costs, and improve the quality of text supervision information and the applicability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to make the purpose and technical solution of the present invention clearer, the present invention provides the following drawings and descriptions:

[0040] Figure 1 A flow chart of a method provided by an embodiment of the present invention;

[0041] Figure 2 A schematic diagram of functional modules provided by an embodiment of the present invention;

[0042] Figure 3 Examples of ambiguity in text descriptions provided for embodiments of the present invention;

[0043] Figure 4 Examples of incomplete text descriptions of embodiments of the present invention;

[0044] Figure 5 Comparison of the PASCAL VOC and COCO semantic segmentation visualization results provided by the embodiments of the present invention;

[0045] Figure 6 Examples of open world semantic segmentation in some rare scenarios provided by embodiments of the present invention;

[0046] Figure 7 An example of open world semantic segmentation based on rare object categories provided by an embodiment of the present invention; DETAILED DESCRIPTION

[0047] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0048] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0049] The present invention provides an open world semantic segmentation automatic concept completion method based on pre-training, such as Figure 1 The figure shows a flow chart of an automatic concept completion method for open-world semantic segmentation based on pre-training. Figure 2 The figure shows a functional module diagram of an open world semantic segmentation automatic concept completion method based on pre-training provided by the present invention, and the specific steps are as follows:

[0050] S1: Calculate the image encoding features of the teacher model and the student model, and transfer the knowledge of the teacher model to the student model through image feature distillation, thereby enhancing the student model's grasp of global semantic information.

[0051] The image feature distillation comprises the following steps:

[0052] Use the image encoder of the vision-language pre-trained model as the teacher model to calculate the image encoding features; use GoupViT's segment generator as the student model to segment the image into multiple segment tokens and calculate the image encoding features;

[0053] It should be noted that the parameters of the teacher model are fixed.

[0054] More specifically, the step of calculating the image encoding features of the student model comprises the following steps:

[0055] Use GoupViT’s image encoder as a segment generator to segment the image into multiple segment tokens;

[0056] Perform an average pooling operation on the segmented multiple segment tokens;

[0057] The pooling results are transformed through a multi-layer perceptron to obtain the encoded features.

[0058] The image encoding features of the teacher model and the student model are normalized separately, and then their similarity is measured by calculating their total distillation contrast loss InfoNCE;

[0059] The total distillation contrastive loss is used to improve the student model so that the features it generates are closer to those generated by the teacher model.

[0060] In a specific embodiment, Figure 2 As shown in Figure 2, a strong image classifier is used as the teacher model ft, whose parameters are fixed. For the student network f s , using GoupViT’s image encoder as the segment generator, I is split into G segment tokens as follows:

[0061]

[0062] in, is the g-th segment feature, d' is the dimension of the segment feature, and R is the real number space. Further, the encoded feature embedding is obtained as follows:

[0063]

[0064] Among them, AvgPool is The average pooling operation is performed by first performing average pooling on the segmentation features and then transforming them through a multi-layer perceptron MLP.

[0065] Accordingly, the image encoder of the vision-language pre-trained model is used as the teacher network f t , so we have:

[0066]

[0067] Finally, the standard contrast loss function InfoNCE is used for image feature distillation. In InfoNCE, for a coded query q and a set of coded samples k = {k0, f1, k2, ...} M , the formula is:

[0068]

[0069] where M is the size of the dictionary, τ is a learnable temperature parameter, q and k + are positive pairs, and the other (M-1) pairs are negative pairs.

[0070] In a batch of B image-text pairs For image I i , through teacher f t and students s Encode them separately and embed them into vectors and And normalize them respectively. Then their similarity is measured by calculating their dot product. The total distillation contrast loss is defined as:

[0071]

[0072] Among them I s and I t denote the embedding of image I encoded by the student network and the teacher network, respectively.

[0073] S2: Use the visual language pre-training model to compare the image encoding features generated by large-scale image-text data with the image encoding features generated by the teacher model, generate subtitles, perform automatic concept completion, and obtain the completed title.

[0074] It should be noted that the preset visual language pre-training model is the CLIP model.

[0075] More specifically, the automatic concept completion in step S2 includes the following steps:

[0076] Use large-scale image-text data to construct label sets, including basic label sets and open label sets;

[0077] It should be noted that there is no intersection between the basic tag set and the open tag set, and the open tag set is not fixed.

[0078] Use the visual language pre-trained model text encoder to encode the label set to obtain feature encoding;

[0079] It should be noted that the parameters of the text encoder of the visual language pre-training model are fixed.

[0080] Calculate the cosine similarity between the feature code and the image feature code generated by the teacher model, and take out the label with high similarity as a supplementary label;

[0081] Generate subtitles by combining supplementary tags with prompting engineering in visual language pre-training models;

[0082] The generated sub-titles and the original title are combined into a total title, and the title is trained to align with the corresponding image semantic information, such as Figure 3 The example shown is an example of using the present invention to generate a general title when there is ambiguity in the text description, such as Figure 4 Shown is an example of using the present invention to generate a general title when the text description is incomplete.

[0083] In a specific embodiment, Figure 2 As shown in the figure, the pre-trained VLP model is used for concept completion. Specifically, two label sets are first constructed: a basic category set and an open category set. The former is used to ensure that the label set covers the most common object categories, and the latter is used to maintain the open attributes of object categories. For the basic label set, 389 object categories are first collected from the MMSegmentation dataset library, including commonly used segmentation datasets such as COCO, ADE20K, PASCALVOC, and PASCAL Context. In addition, RAM, the Recognize Anything model, a powerful basic model for image tagging, is used to add 4K+ labels collected therein to the basic category set. An open label set is also constructed by extracting nouns from the original titles of a batch of image-text pairs containing 500 different nouns. The open label set has no intersection with the basic category set. Note that the open label set is not fixed, it will change with the change of the batch dataset. These two label sets are merged into the total label set N is the number of tags, set N to 5263.

[0084] Then, we start to complete the concept of image I. Specifically, we use the pre-trained visual language pre-trained model text encoder with frozen parameters to encode the label set and obtain the corresponding label embedding, the formula is:

[0085]

[0086] Then, by calculating the teacher model image feature encoding With embedded Z L The cosine similarity between the label sets of image I is used to complete the concept. The process of concept completion L cc It can be expressed as

[0087]

[0088] Where θ represents the similarity threshold, top kUsed to control the number of supplementary labels, in one embodiment, k is set to 3. The generated labels are combined with prompting engineering in the visual language pre-training model to generate additional captions. Finally, the generated and original captions are merged into a total caption, and the caption is trained to align with the corresponding image semantic information.

[0089] S3: Use the encoding features of the title after automatic concept completion, the title encoding features of the subtitle generated by the visual language pre-training model, and the contrast loss of the image encoding features generated by the student model to promote the student model to achieve cross-modal semantic information alignment.

[0090] More specifically, the cross-modal semantic information alignment is implemented in step S3, including the following steps:

[0091] Calculate the encoding features of the title after automatic concept completion, the title encoding features of the subtitle generated by the visual language pre-training model, and the image encoding features generated by the student model;

[0092] The contrast loss between the encoding features of the title after automatic concept completion, the title encoding features of the subtitle generated by the visual language pre-training model, and the image encoding features generated by the student model is calculated respectively to obtain two cross-modal semantic alignment contrast losses;

[0093] Calculate the total loss, which is the sum of the total distillation contrast loss and the two cross-modal semantic alignment contrast losses;

[0094] The total loss is used to promote the student model to achieve cross-modal semantic information alignment.

[0095] In a specific embodiment, Figure 2 As shown, for each given image-text pair (I, T), the concept-completed title is encoded as:

[0096] Z T =MLP(E T (AutoCC(T)))∈R d

[0097] Similarly, we can get the subtitles generated by prompting engineering in the visual language pre-training model:

[0098]

[0099] Where Q is the number of subtitles, set Q to 3. Use the text encoder E T Encode them, i.e.:

[0100]

[0101] Calculate the student network f in the image branch s Output and text encoder E T The output in the text branch {Z T ,Z T'}contrast loss.

[0102] Finally, calculate {T,T'} and I respectively s Specifically, Z T and The contrast loss between them is defined as follows:

[0103]

[0104] In addition, Z T' and The multi-label contrast loss between is defined as follows:

[0105]

[0106] in,

[0107]

[0108] The total loss of the present invention is the sum of the image feature distillation contrast loss and the two cross-modal semantic alignment contrast losses:

[0109]

[0110] The total loss is used to promote the student model to achieve cross-modal semantic information alignment.

[0111] S4: Use the trained student model to effectively segment and recognize images.

[0112] The AutoCC-Seg of the present invention shows good performance on PASCAL VOC, PASCAL Context, COCO and ADE20K datasets. Compared with the previous open world semantic segmentation model (such as GroupViT), this patent has achieved the best results, as shown in Table 1, Table 2, and Table 3.

[0113] On PASCAL VOC, PASCAL Context datasets:

[0114] Table 1

[0115]

[0116]

[0117] On the COCO dataset:

[0118] Table 2

[0119]

[0120]

[0121] On the ADE20K dataset:

[0122] Table 3

[0123]

[0124] Figure 5 The visualization comparison results of semantic segmentation on PASCAL VOC and COCO datasets are shown. Among them, GroundTruth represents the standard answer. The comparison results show that the segmentation results of the AutoCC-Seg solution proposed in this paper are closer to Ground Truth than those of the GroupViT model.

[0125] Figure 6 The comparative results of open world semantic segmentation in some rare scenarios are presented, and the red boxes mark the areas where segmentation errors are more obvious. Figure 7 It shows the open world semantic segmentation comparison results based on rare object categories, and also marks the obvious segmentation error areas with red boxes.

[0126] A comprehensive analysis of these results shows that compared with the GroupViT model, AutoCC-Seg shows stronger segmentation ability in semantic segmentation tasks, especially when dealing with rare scenes and rare object categories.

[0127] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A pre-trained open-world semantic segmentation automatic concept completion method, characterized in that: The steps include: Calculate the image encoding features of the teacher model and the student model, and transfer the knowledge of the teacher model to the student model through image feature distillation, thereby enhancing the student model's grasp of global semantic information; Using the preset visual language pre-training model, the image encoding features generated by large-scale image-text data are compared with the image encoding features generated by the teacher model to generate subtitles and perform automatic concept completion to obtain the completed title; The encoding features of the title after automatic concept completion, the title encoding features of the subtitle generated by the visual language pre-training model, and the image encoding features generated by the student model are compared with the loss to promote the student model to achieve cross-modal semantic information alignment; The trained student model is used to effectively segment and recognize images.

2. The method for automatic concept completion based on pre-training in open world semantic segmentation according to claim 1, characterized in that: The preset visual language pre-training model is the CLIP model.

3. The method for automatic concept completion based on pre-training open world semantic segmentation according to claim 1, characterized in that: The image feature distillation comprises the following steps: Use the image encoder of the vision-language pre-trained model as the teacher model to calculate the image encoding features; use GoupViT's segment generator as the student model to segment the image into multiple segment tokens and calculate the image encoding features; The image encoding features of the teacher model and the student model are normalized separately, and then their similarity is measured by calculating their total distillation contrast loss; The total distillation contrastive loss is used to improve the student model so that the features it generates are closer to those generated by the teacher model.

4. The method for automatic concept completion based on pre-training open world semantic segmentation according to claim 2, characterized in that: The parameters of the teacher model are fixed.

5. The method for automatic concept completion based on pre-training open world semantic segmentation according to claim 2, characterized in that: The step of calculating the image coding features of the student model comprises the following steps: Use GoupViT’s image encoder as a segment generator to segment the image into multiple segment tokens; Perform an average pooling operation on the segmented multiple segment tokens; The pooling results are transformed through a multi-layer perceptron to obtain the encoded features.

6. The method for automatic concept completion based on pre-training open world semantic segmentation according to claim 2, characterized in that: The formula for the total distillation contrast loss is: Among them, L is the contrast loss function used, I is the image, and I s and I t denote the image feature embeddings encoded by the student model and the teacher model, respectively.

7. The method for automatic concept completion based on pre-training open world semantic segmentation according to claim 1, characterized in that: The automatic concept completion comprises the following steps: Use large-scale image-text data to construct label sets, including basic label sets and open label sets; Use the visual language pre-trained model text encoder to encode the label set to obtain feature encoding; Calculate the cosine similarity between the feature code and the image feature code generated by the teacher model, and take out the label with high similarity as a supplementary label; Generate subtitles by combining supplementary tags with prompting engineering in visual language pre-training models; The generated sub-captions and the original caption are merged into an overall caption, which is then trained to align with the corresponding image semantic information.

8. The method for automatic concept completion based on pre-training open world semantic segmentation according to claim 7, characterized in that: There is no intersection between the basic tag set and the open tag set, and the open tag set is not fixed.

9. The method for automatic concept completion based on pre-training in open world semantic segmentation according to claim 7, characterized in that: The parameters of the text encoder of the visual language pre-training model are fixed.

10. The method for automatic concept completion based on pre-training in open world semantic segmentation according to claim 1, characterized in that: The cross-modal semantic information alignment is implemented, comprising the following steps: Calculate the encoding features of the title after automatic concept completion, the title encoding features of the subtitle generated by the visual language pre-training model, and the image encoding features generated by the student model; The contrast loss between the encoding features of the title after automatic concept completion, the title encoding features of the subtitle generated by the visual language pre-training model, and the image encoding features generated by the student model is calculated respectively to obtain two cross-modal semantic alignment contrast losses; Calculate the total loss, which is the sum of the total distillation contrast loss and the two cross-modal semantic alignment contrast losses; The total loss is used to promote the student model to achieve cross-modal semantic information alignment.

Images