An automatic concept completion method based on pre-training open world semantic segmentation
By using image feature distillation and automatic concept completion, and leveraging a pre-trained visual language pre-trained model for cross-modal semantic information alignment, this approach addresses the issues of poor model performance and missing labels in open-world semantic segmentation, achieving higher segmentation accuracy and lower training costs.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SUN YAT SEN UNIV
- Filing Date
- 2024-08-16
- Publication Date
- 2026-05-29
AI Technical Summary
Existing technologies in open-world semantic segmentation suffer from problems such as poor model performance when handling unseen new categories, significant noise impact, low quality of text supervision information, missing labels, and difficulties in standardization, as well as high training costs.
By employing image feature distillation and automatic concept completion, a pre-trained visual language pre-trained model is used to align cross-modal semantic information. The automatic concept completion module is used to complete missing or ambiguous conceptual information. The feature generation of the student model is improved by combining total distillation contrast loss and cross-modal semantic alignment contrast loss.
It improves the segmentation accuracy of the model when dealing with unseen new categories, reduces the impact of noise, solves the problems of missing labels and standardization, reduces training costs, and improves the quality of text supervision information and model applicability.
Smart Images
Figure CN119206205B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the fields of computer vision, deep learning and natural language processing, and more specifically, relates to a pre-trained open-world semantic segmentation automatic concept completion method. Background Technology
[0002] The success of Visual Language Pre-trained (VLP) models has greatly advanced the development of text-supervised open-world semantic segmentation, leveraging the correlation between images and text for segmentation. However, current models primarily rely on large-scale web-based image-text datasets, which have limitations in text description, and task-specific text generators are expensive to train. Furthermore, transferring open-world semantic knowledge from pre-trained VLP models to semantic segmentation tasks faces challenges in patch grouping and fine-grained cross-modal semantic alignment.
[0003] As one specific implementation of VLP models, existing CLIP-based open-world semantic segmentation models typically employ retraining for patch self-grouping (GroupViT) or using a mask generator to fine-tune the semantic alignment module for semantic classification (OVSeg). Specifically, GroupViT performs patch self-grouping through retraining, dividing the image into segments and extracting and processing features for each segment. OVSeg, on the other hand, uses a mask generator to fine-tune the semantic alignment module, generating masks to achieve semantic classification. Furthermore, to improve the quality of text supervision information, previous methods often introduced semantic dictionaries built using LLMs (such as ChatGPT) to address ambiguity issues. However, retraining a CLIP-like framework may disrupt the original semantic alignment relationships, leading to poor model performance when handling unfamiliar categories. On the other hand, CLIP-based image-level semantic alignment is unsuitable for fine-grained semantic segmentation tasks, and the fine-tuning process may introduce noise, affecting segmentation accuracy. Moreover, building and maintaining semantic dictionaries using LLMs (such as ChatGPT) is costly and cannot effectively address the issues of missing labels and standardization in datasets.
[0004] The prior art patent CN118072332A proposes a self-evolving zero-shot target recognition method based on dual sketch and text cues. This method includes using CLIP as the backbone network, designing visual and text cues modules within the DP-CLIP framework, extracting category knowledge from images using Transformer layers and transferring it to a set of learnable vectors, which are then used as visual cues for that category. A CLIP-based text encoder embeds target category labels into text features, generates channel scaling vectors using these text features, and applies these scaling vectors to the hidden features of the CLIP visual encoder, guiding the model to focus on channels related to the current category. However, this invention utilizes a retrained CLIP-like framework, which may disrupt the original semantic alignment, leading to poor model performance when handling unseen new categories. Furthermore, the cost of constructing and maintaining a semantic dictionary is high, and it cannot effectively address the issues of missing labels and standardization in the dataset. Summary of the Invention
[0005] This invention aims to improve fine-grained segmentation capabilities, reduce the impact of noise, enhance model performance when handling unseen new categories, address label missing and standardization issues, reduce costs, improve the quality of text supervision information and the applicability of the model, and provide a pre-trained open-world semantic segmentation automatic concept completion method.
[0006] This invention provides an automatic concept completion method for open-world semantic segmentation based on pre-training, comprising the following steps:
[0007] The image coding features of the teacher model and the student model are calculated. Through image feature distillation, the knowledge of the teacher model is transferred to the student model, thereby enhancing the student model's grasp of global semantic information.
[0008] Using a pre-trained visual language model, the similarity between the image coding features generated from large-scale image-text data and the image coding features generated by the teacher model is compared to generate subheadings. Automatic concept completion is then performed to obtain the completed title.
[0009] The coding features of the title after automatic concept completion, the title coding features of the subtitle generated by the visual language pre-trained model, and the image coding features generated by the student model are compared using a loss function to promote cross-modal semantic information alignment of the student model.
[0010] The trained student model is used to effectively segment and recognize images.
[0011] Furthermore, the preset visual language pre-trained model is the CLIP model.
[0012] Furthermore, the image feature distillation includes the following steps:
[0013] The image encoder, a pre-trained visual language model, is used as the teacher model to calculate image encoding features; the segment generator of GoupViT is used as the student model to segment the image into multiple segment tokens and calculate image encoding features.
[0014] The image encoding features of the teacher model and the student model are normalized respectively, and their similarity is measured by calculating their total distillation contrast loss.
[0015] The student model is improved by using total distillation contrast loss, so that the features it generates are closer to the features generated by the teacher model.
[0016] Furthermore, the parameters of the teacher model are fixed.
[0017] Furthermore, the calculation of the image coding features of the student model includes the following steps:
[0018] The image encoder of GoupViT is used as a segment generator to segment the image into multiple segment tokens;
[0019] Perform an average pooling operation on the divided token segments;
[0020] The pooling results are transformed using a multilayer perceptron to obtain encoded features.
[0021] Furthermore, the formula for the total distillation contrast loss is as follows:
[0022]
[0023] Where L is the contrast loss function used, I is the image, and I0 is the image value. s and I t These represent the image feature embeddings encoded by the student model and the teacher model, respectively.
[0024] Furthermore, the automatic concept completion includes the following steps:
[0025] A label set is constructed using large-scale image-text data, including a basic label set and an open label set;
[0026] The label set is encoded using a visual language pre-trained model text encoder to obtain feature codes;
[0027] Calculate the cosine similarity between the feature encoding and the image feature encoding generated by the teacher model, and extract the labels with high similarity as supplementary labels;
[0028] Subheadings are generated by combining supplementary labels with prompting engineering in a visual language pre-trained model;
[0029] The generated subtitles and the original title are merged into a single overall title, and this title is trained to align with the corresponding image semantic information.
[0030] Furthermore, there is no overlap between the basic tag set and the open tag set, and the open tag set is not fixed.
[0031] Furthermore, the parameters of the text encoder in the visual language pre-trained model are fixed.
[0032] Furthermore, the process of achieving cross-modal semantic information alignment includes the following steps:
[0033] Calculate the encoding features of the title after automatic concept completion, the title encoding features of the subtitles generated by the visual language pre-trained model, and the image encoding features generated by the student model;
[0034] The contrast loss between the encoding features of the title after automatic concept completion, the title encoding features of the subtitle generated by the visual language pre-trained model, and the image encoding features generated by the student model is calculated separately to obtain two cross-modal semantic alignment contrast losses.
[0035] Calculate the total loss, which is the sum of the total distillation contrastive loss and the two cross-modal semantic alignment contrastive losses;
[0036] The total loss is used to facilitate cross-modal semantic information alignment in student models.
[0037] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:
[0038] This invention utilizes a pre-trained VLP model for image feature distillation, leveraging contrastive loss to facilitate cross-modal semantic information alignment. This ensures effective alignment of image and text features within the same semantic space, reducing noise impact, improving segmentation accuracy, and enhancing model performance when handling unfamiliar categories. An automatic concept completion module is employed to fill in missing or ambiguous concept information using the pre-trained VLP model, ensuring consistency and accuracy of text labels. This addresses label missing and standardization issues, reduces costs, and improves the quality of text supervision information and the model's applicability. Attached Figure Description
[0039] To make the objectives and technical solutions of this invention clearer, the following drawings are provided and described:
[0040] Figure 1 A flowchart of the method provided in an embodiment of the present invention;
[0041] Figure 2 This is a schematic diagram of the functional modules provided in an embodiment of the present invention;
[0042] Figure 3 Examples of ambiguity in the text description provided in the embodiments of the present invention;
[0043] Figure 4 This is an example of an incomplete text description provided in the embodiments of the present invention;
[0044] Figure 5 A comparison of the semantic segmentation visualization results of PASCALVOC and COCO provided in the embodiments of the present invention;
[0045] Figure 6 Examples of open-world semantic segmentation in some rare scenarios provided for embodiments of the present invention;
[0046] Figure 7 This is an example of open-world semantic segmentation based on rare object categories provided in an embodiment of the present invention; Detailed Implementation
[0047] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.
[0048] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.
[0049] This invention provides an automatic concept completion method based on pre-trained open-world semantic segmentation, such as... Figure 1 The diagram shows a flowchart of an automatic concept completion method for open-world semantic segmentation based on pre-training. Figure 2 The diagram shows a functional module of an automatic concept completion method for open-world semantic segmentation based on pre-training provided by the present invention. The specific steps are as follows:
[0050] S1: Calculate the image coding features of the teacher model and the student model. Through image feature distillation, transfer the knowledge of the teacher model to the student model, thereby enhancing the student model's grasp of global semantic information.
[0051] The image feature distillation includes the following steps:
[0052] The image encoder, a pre-trained visual language model, is used as the teacher model to calculate image encoding features; the segment generator of GoupViT is used as the student model to segment the image into multiple segment tokens and calculate image encoding features.
[0053] It should be noted that the parameters of the teacher model are fixed.
[0054] More specifically, the calculation of the image coding features of the student model includes the following steps:
[0055] The image encoder of GoupViT is used as a segment generator to segment the image into multiple segment tokens;
[0056] Perform an average pooling operation on the divided token segments;
[0057] The pooling results are transformed using a multilayer perceptron to obtain encoded features.
[0058] The image encoding features of the teacher model and the student model are normalized respectively, and their similarity is measured by calculating their total distillation contrast loss InfoNCE.
[0059] The student model is improved by using total distillation contrast loss, so that the features it generates are closer to the features generated by the teacher model.
[0060] In a specific embodiment, such as Figure 2 As shown, a strong image classifier is used as the teacher model ft, and its parameters are fixed. For the student network f... s Using the GoupViT image encoder as a segment generator, I is segmented into G segment tokens, as shown in the following formula:
[0061]
[0062] in, Let d' be the feature of the g-th segment, d' be the dimension of the segment feature, and R be the real number space. Further, the encoded feature embedding is obtained as follows:
[0063]
[0064] Where AvgPool is a pair The average pooling operation is performed first on the segmentation features, and then transformed by a multilayer perceptron (MLP).
[0065] Accordingly, a visual language pre-trained model image encoder is used as the teacher network f. t Therefore:
[0066]
[0067] Finally, image feature distillation is performed using the standard contrastive loss function InfoNCE. In InfoNCE, for a coded query q and a set of coded samples k = {k0, f1, k2, ...} M The formula is:
[0068]
[0069] Where M is the size of the dictionary, τ is a learnable temperature parameter, and q and k + It is a positive pair, and the other (M-1) pairs are negative pairs.
[0070] In a batch of B image-text pairs In the middle, for image I i Through teacher f t and student f s Encode them separately and embed them into vectors and They are then normalized separately. Their similarity is then measured by calculating their dot product. The total distillation contrastive loss is defined as:
[0071]
[0072] Where I s and I t Let I represent the embedding of image I encoded by the student network and the teacher network, respectively.
[0073] S2: Using a visual language pre-trained model, the image coding features generated from large-scale image-text data are compared with the image coding features generated by the teacher model to generate subheadings, perform automatic concept completion, and obtain the completed title.
[0074] It should be noted that the preset visual language pre-trained model is the CLIP model.
[0075] More specifically, the automatic concept completion in step S2 includes the following steps:
[0076] A label set is constructed using large-scale image-text data, including a basic label set and an open label set;
[0077] It should be noted that there is no overlap between the basic tag set and the open tag set, and the open tag set is not fixed.
[0078] The label set is encoded using a visual language pre-trained model text encoder to obtain feature codes;
[0079] It should be noted that the parameters of the text encoder in the visual language pre-trained model are fixed.
[0080] Calculate the cosine similarity between the feature encoding and the image feature encoding generated by the teacher model, and extract the labels with high similarity as supplementary labels;
[0081] Subheadings are generated by combining supplementary labels with prompting engineering in a visual language pre-trained model;
[0082] The generated subtitles and the original title are merged into a single overall title, and this title is trained to align with the corresponding image semantic information, such as... Figure 3 The image shows an example of using this invention to generate a main title when there is ambiguity in the text description, such as... Figure 4 The image shows an example of using this invention to generate a general title when the text description is incomplete.
[0083] In a specific embodiment, such as Figure 2 As shown, concept completion is performed using a pre-trained VLP model. Specifically, two label sets are constructed: a base category set and an open category set. The former ensures that the label set covers the most common object categories, while the latter maintains the open attribute of object categories. For the base label set, 389 object categories are collected from the MMSegmentation dataset, which includes commonly used segmentation datasets such as COCO, ADE20K, PASCALVOC, and PASCAL Context. Furthermore, using RAM (Anything Recognition Model), a powerful foundational model for image labeling, 4K+ labels collected therein are added to the base category set. An open label set is also constructed by extracting nouns from a batch of original captions containing 500 different nouns from image-text pairs. This open label set has no overlap with the base category set. Note that the open label set is not fixed; it changes with the batch dataset. These two label sets are merged into a total label set. N is the number of tags; set N to 5263.
[0084] Then, we begin to complete the concept of image I. Specifically, we encode the label set using a pre-trained visual-language pre-trained model text encoder with frozen parameters to obtain the corresponding label embeddings, as shown in the formula:
[0085]
[0086] Then, by calculating the image feature encoding of the teacher model. With embedded Z L The cosine similarity between the label sets is used to complete the concept of image I. The concept completion process L... cc It can be represented as
[0087]
[0088] Where θ represents the similarity threshold, top kTo control the number of supplementary labels, k is set to 3 in one embodiment. The generated labels are combined with prompting engineering in the visual language pre-trained model to generate additional captions. Finally, the generated and original captions are merged into a single overall caption, which is then trained to align with the corresponding image semantic information.
[0089] S3: The student model is encouraged to achieve cross-modal semantic information alignment by using the encoding features of the title after automatic concept completion, the title encoding features of the subtitle generated by the visual language pre-trained model, and the image encoding features generated by the student model as contrast loss.
[0090] More specifically, step S3, which aligns cross-modal semantic information, includes the following steps:
[0091] Calculate the encoding features of the title after automatic concept completion, the title encoding features of the subtitles generated by the visual language pre-trained model, and the image encoding features generated by the student model;
[0092] The contrast loss between the encoding features of the title after automatic concept completion, the title encoding features of the subtitle generated by the visual language pre-trained model, and the image encoding features generated by the student model is calculated separately to obtain two cross-modal semantic alignment contrast losses.
[0093] Calculate the total loss, which is the sum of the total distillation contrastive loss and the two cross-modal semantic alignment contrastive losses;
[0094] The total loss is used to facilitate cross-modal semantic information alignment in student models.
[0095] In a specific embodiment, such as Figure 2 As shown, for each given image-text pair (I,T), the title after concept completion is encoded as follows:
[0096] Z T =MLP(E T (AutoCC(T)))∈R d
[0097] Similarly, subheadings generated by prompting engineering in the visual language pre-trained model can be obtained:
[0098]
[0099] Where Q represents the number of subheadings, and Q is set to 3. Use text encoder E. T Encode them, that is:
[0100]
[0101] Computational image branch of student network f s Output and text encoder E T Output in the text branch {Z T Z T' Contrast loss.
[0102] Finally, calculate {T,T'} and I respectively. s The contrast loss between them. Specifically, Z T and The contrast loss between them is defined as follows:
[0103]
[0104] In addition, Z T' and The multi-label contrast loss between them is defined as follows:
[0105]
[0106] in,
[0107]
[0108] The total loss in this invention is the sum of the image feature distillation contrast loss and the two cross-modal semantic alignment contrast losses:
[0109]
[0110] The total loss is used to facilitate cross-modal semantic information alignment in student models.
[0111] S4: Use the trained student model to effectively segment and recognize images.
[0112] The AutoCC-Seg invention demonstrates excellent performance on the PASCAL VOC, PASCAL Context, COCO, and ADE20K datasets. Compared to previous open-world semantic segmentation models (such as GroupViT), this patent achieves the best results, as shown in Tables 1, 2, and 3.
[0113] On the PASCALVOC, PASCAL Context dataset:
[0114] Table 1
[0115]
[0116]
[0117] On the COCO dataset:
[0118] Table 2
[0119]
[0120]
[0121] On the ADE20K dataset:
[0122] Table 3
[0123]
[0124] Figure 5 This paper presents a visual comparison of semantic segmentation results on the PASCALVOC and COCO datasets. Ground Truth represents the standard answer. The comparison results show that the proposed AutoCC-Seg scheme achieves segmentation results closer to the Ground Truth than the GroupViT model.
[0125] Figure 6 The results show comparative results of open-world semantic segmentation in some rare scenarios, with red boxes marking areas where segmentation errors are more obvious. Figure 7 This shows the comparison results of open-world semantic segmentation based on rare object categories, with obvious segmentation error areas marked in red.
[0126] A comprehensive analysis of these results shows that, compared with the GroupViT model, AutoCC-Seg exhibits stronger segmentation capabilities in semantic segmentation tasks, especially when dealing with rare scenes and rare object categories.
[0127] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.
Claims
1. A pre-trained open-world semantic segmentation automatic concept completion method, characterized in that, Includes the following steps: The image coding features of the teacher model and the student model are calculated. Through image feature distillation, the knowledge of the teacher model is transferred to the student model, thereby enhancing the student model's grasp of global semantic information. Using a pre-trained visual language model, the similarity between the image coding features generated from large-scale image-text data and the image coding features generated by the teacher model is compared to generate subheadings. Automatic concept completion is then performed to obtain the completed title. The coding features of the title after automatic concept completion, the title coding features of the subtitle generated by the visual language pre-trained model, and the image coding features generated by the student model are compared using a contrast loss to promote the student model to achieve cross-modal semantic information alignment. The trained student model is used to effectively segment and recognize images; The automatic concept completion includes the following steps: A label set is constructed using large-scale image-text data, including a basic label set and an open label set; The label set is encoded using a visual language pre-trained model text encoder to obtain feature codes; Calculate the cosine similarity between the feature encoding and the image feature encoding generated by the teacher model, and extract the labels with high similarity as supplementary labels; Subheadings are generated by combining supplementary labels with prompting engineering in a visual language pre-trained model; The generated subtitles and the original title are merged into a single overall title, and this title is trained to align with the corresponding image semantic information. The image feature distillation includes the following steps: The image encoder, a pre-trained visual language model, is used as the teacher model to calculate image encoding features; the segment generator of GoupViT is used as the student model to segment the image into multiple segment tokens and calculate image encoding features. The image encoding features of the teacher model and the student model are normalized respectively, and their similarity is measured by calculating their total distillation contrast loss. The student model is improved by using total distillation contrast loss, so that the features it generates are closer to the features generated by the teacher model.
2. The method for automatic concept completion based on pre-training in open-world semantic segmentation according to claim 1, characterized in that, The preset visual language pre-trained model is the CLIP model.
3. The method for automatic concept completion based on pre-training in open-world semantic segmentation according to claim 1, characterized in that, The parameters of the teacher model are fixed.
4. The method for automatic concept completion based on pre-training in open-world semantic segmentation according to claim 1, characterized in that, The calculation of the image coding features of the student model includes the following steps: The image encoder of GoupViT is used as a segment generator to segment the image into multiple segment tokens; Perform an average pooling operation on the divided token segments; The pooling results are transformed using a multilayer perceptron to obtain encoded features.
5. The pre-trained open-world semantic segmentation automatic concept completion method according to claim 1, characterized in that, The formula for the total distillation comparative loss is as follows: Where L is the contrastive loss function used, and I is the image. and These represent the image feature embeddings encoded by the student model and the teacher model, respectively.
6. The method for automatic concept completion based on pre-training in open-world semantic segmentation according to claim 1, characterized in that, There is no overlap between the basic tag set and the open tag set, and the open tag set is not fixed.
7. The method for automatic concept completion based on pre-training in open-world semantic segmentation according to claim 1, characterized in that, The parameters of the text encoder in the visual language pre-trained model are fixed.
8. The method for automatic concept completion based on pre-training open-world semantic segmentation according to claim 1, characterized in that, The process of achieving cross-modal semantic information alignment includes the following steps: Calculate the encoding features of the title after automatic concept completion, the title encoding features of the subtitles generated by the visual language pre-trained model, and the image encoding features generated by the student model; The contrast loss between the encoding features of the title after automatic concept completion, the title encoding features of the subtitle generated by the visual language pre-trained model, and the image encoding features generated by the student model is calculated separately to obtain two cross-modal semantic alignment contrast losses. Calculate the total loss, which is the sum of the total distillation contrastive loss and the two cross-modal semantic alignment contrastive losses; The total loss is used to facilitate cross-modal semantic information alignment in student models.