Unsupervised visual positioning system based on pseudo labels
By using multimodal large language model and object detector to generate pseudo-labels of open vocabulary in unsupervised visual positioning, the problems of low quality of pseudo-labels and scarce data in the prior art are solved, and high accuracy and rich semantics are achieved Visual positioning effect.
Patent Information
- Application Number
- CN202411019001.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-29
- Publication Date
- 2024-11-22
AI Technical Summary
The existing unsupervised visual positioning technology relies on pseudo-labels generated by object detectors and attribute classifiers. Due to the vocabulary and category preferences, the pseudo-label quality is not high and the data scarcity cannot be effectively handled.
Through the area proposal generation module, the area description generation module and the query construction module, the pre-trained object detector and multimodal large language model are used to generate visual positioning pseudo-labels of open vocabulary, including category bias eliminaters, description correctors and Modified trees construct pseudo-queries, significantly improving the accuracy of unsupervised visual positioning.
It significantly improves the accuracy of unsupervised visual positioning, the generated pseudo-label semantics are rich, reduces the impact of category preferences, enhances the generalization ability of the model, and can achieve good performance in the case of scarcity of data.
Smart Images

Figure CN119006779A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of image processing, and in particular relates to an unsupervised visual positioning system based on pseudo labels. Background Art
[0002] Visual localization is an important task in the field of computer vision and natural language processing. Its main purpose is to find specific objects or regions in an image that correspond to a given text description. The core of this task lies in the alignment of cross-modal information, that is, establishing an accurate correspondence between vision (image) and language (text). Visual localization plays an important role in many practical applications, including but not limited to: human-computer interaction, directly manipulating image content through natural language descriptions, such as selecting and editing image areas with text descriptions in photo editing software; augmented reality (AR), in real scenes, enhancing the positioning and display of virtual information through language descriptions; assistive technology, helping the visually impaired to understand image content or scenes through descriptions; intelligent monitoring, in surveillance videos, quickly locating objects or regions of interest through language instructions. Unsupervised learning is a method for model training without explicitly annotated data. In unsupervised visual localization, there is no need to manually annotate a large number of images and corresponding text descriptions, but visual localization is achieved by exploring the intrinsic connection between images and text. Compared with supervised visual localization, unsupervised methods have the following advantages: reduce data annotation costs. Manually annotating data is a time-consuming and laborious task, especially accurately annotating specific objects or regions in images. Unsupervised methods significantly reduce the need for data annotation by utilizing unlabeled data, thereby reducing costs; enhance the generalization ability of the model. Unsupervised learning methods often improve the generalization ability of the model by learning the distribution characteristics of images and texts from a large amount of unlabeled data, making it more robust when processing new data; deal with data scarcity problems. In some specific fields, it may be difficult to obtain sufficient labeled data. Unsupervised methods can use unlabeled data to effectively alleviate the problem of data scarcity, so that the model can also achieve good performance in these fields; adaptive learning. Unsupervised learning methods can usually extract features from data adaptively, reducing dependence on predefined features and manually designed rules, so that the model can more flexibly adapt to different application scenarios and task requirements. Existing unsupervised visual localization technologies all rely on pseudo-labels generated by target detectors and attribute classifiers. The generated pseudo-labels are limited to the vocabulary of target detectors and attribute classifiers. Secondly, when selecting region proposals based on the confidence output by the target detector, it will be affected by the target detector's preference for analogies, and some categories will have a greater probability of being selected to generate pseudo labels. Summary of the invention
[0003] The purpose of the present invention is to provide a fast and highly accurate unsupervised visual positioning system based on pseudo labels, which can obtain a model with visual positioning capability by using a large number of unlabeled images.
[0004] The pseudo-label-based unsupervised visual positioning system provided by the present invention includes: a region proposal generation module, a region description generation module, and a query construction module; the region proposal generation module is composed of a region proposal generator and a category bias eliminator, and the module identifies the entity regions, categories and their confidences that may exist in the image through a pre-trained target detector, and solves the problem of uneven confidence distribution in the category through the category bias eliminator; the region description generation module is composed of a region descriptor and a description corrector; the module generates a description of the region proposal using the cross-modal knowledge contained in the multi-modal large language model, and corrects the wrong description through the description corrector; the query construction module constructs a query through a modification tree and a prompt template. The present invention constructs an open vocabulary visual positioning pseudo-label by using the pre-trained target detector and the cross-modal knowledge contained in the multi-modal large language model, and significantly improves the accuracy of unsupervised visual positioning without any manual annotation for the task.
[0005] (i) The region proposal generation module includes a region proposal generator and a category bias eliminator; the region proposal generation module generates region proposals that may contain entities using unlabeled images and pre-trained object detectors, and reduces the unevenness of the distribution of region proposals in categories through the category bias eliminator.
[0006] The region proposal generator is used to check the region where entities may exist in the image slice, the entity category and its confidence, which is expressed as:
[0007] b i ,c i ,s i =f detector (p)#(1)
[0008] Among them, p represents an image; f detector represents the pre-trained target detector; b i ,c i ,s i They represent the bounding box coordinates, entity category and confidence of the i-th possible entity area in the image respectively.
[0009] Since the object detector has a bias towards categories, specifically, influenced by the training data, the object detector tends to give higher scores to certain categories. Selecting region proposals directly based on confidence will result in an uneven distribution of the selected region categories, affecting the quality of the generated pseudo-labels, so:
[0010] The category bias eliminator is used to eliminate the bias of the target detector for the category, and is expressed as:
[0011]
[0012] Among them, s i represents the confidence of the ith region; μ c represents the mean confidence of the regions detected in the data set that belong to the same category as the i-th region; σ c It represents the standard deviation of the confidence of the regions detected in the dataset that belong to the same category as the i-th region.
[0013] Through the processing of the category bias eliminator, the confidence of entities of different categories conforms to the distribution with a mean of 0 and a standard deviation of 1. The k regions with the largest confidence in each image are selected as region proposals.
[0014] (ii) The region description generation module is composed of a region descriptor and a description corrector. This module generates descriptions of region proposals using cross-modal knowledge contained in a multimodal large language model, and corrects erroneous descriptions using the description corrector.
[0015] The region descriptor is used to generate a description of the region proposal to obtain rich attributes and category names of the open vocabulary. Through a specially designed prompt: "Please use a sentence or phrase to briefly describe the image, focusing on the most important object and not including any other objects. Your description can include attributes such as color, posture, clothing, etc., but it should not be too long", the multimodal large language model outputs a description of the main entity in the prompt region.
[0016] Taking into account the possibility that a large multimodal language model may misidentify the main entity in the wrong region, the present invention uses a description corrector to correct the incorrect description.
[0017] The description corrector uses WordNet to check whether the main entity of the generated description is consistent with the entity category obtained by the target detector. Specifically, when the main entity and the entity category are defined as synonyms or one is a hypernym of the other in WordNet, the subject described by the multimodal large language model is considered correct and the generated description is retained; otherwise, the entity category is used as the description for the region.
[0018] (III) The query construction module uses the modified tree to extract the main entities and their attributes in the region description, and then uses the prompt template "'Entity name' with 'attribute 1, attribute 2, attribute 3' attributes" to construct a pseudo query. Specifically:
[0019] The modified tree is constructed by using the syntax analysis of CoreNLP, and then by defining the priorities of different types of subtrees (for example, if the type of the node is a noun phrase, the noun type node in its subtree has a higher priority than the verb type, adjective type and adverb type node), the subtree with a high priority is used to replace the parent node to construct the modified tree; the root node of the modified tree is the central entity, and the subtree of the root node is the attribute of the central entity. Finally, the pseudo query is constructed using the prompt template "'Entity name' with 'attribute 1, attribute 2, attribute 3' attributes".
[0020] The pseudo-label-based unsupervised visual positioning system of the present invention has the following workflow:
[0021] (I) First, a large number of unlabeled images are input into the pre-trained object detector, and the region proposal generator generates the region that may contain the entity, the category of the entity, and the confidence. The confidence of each region proposal is recalculated using the category bias eliminator. For each image, the k regions with the highest confidence are selected as region proposals.
[0022] (ii) Secondly, the description of each region proposal is generated by the region description generator, and the similarity between the entity category output by the object detector and the central entity in the description is used to correct and delete the wrong description, and retain the correct description. The description corrector uses WordNet to check whether the main entity of the generated description is consistent with the entity category obtained by the object detector. Specifically, when the main entity and the entity category are defined as synonyms or one is the hypernym of the other in WordNet, the subject described by the multimodal large language model is considered correct and the generated description is retained; otherwise, the entity category is used as the description of the region. At the same time, the spatial attributes of the region proposal are generated based on the positional relationship of the region proposal in the image.
[0023] (iii) By defining the priorities of different types of nodes in the syntax tree, the syntax tree is reconstructed into a modification tree to extract the central entity and the attributes of the entity in the region description.
[0024] (iv) Using a specially designed prompt template, fill the generated central entity and attributes into the template to construct a pseudo label. Finally, use the constructed pseudo label to train the visual localization model.
[0025] The advantages of the present invention mainly include:
[0026] (1) The descriptions generated by the multimodal large language model contain rich attributes and entity categories of open vocabulary, and the generated pseudo-labels are more semantically rich and closer to the real annotations.
[0027] (2) The category bias eliminator reduces the impact of the target detector’s category preference on the generated pseudo-labels, so that entities of different categories have the same probability of generating pseudo-labels.
[0028] (3) The query construction module can not only be used to construct pseudo queries, but also to reconstruct real queries during reasoning, further narrowing the gap between the two and improving the effect of the final model.
[0029] (4) The present invention achieves leading unsupervised visual localization results on the public datasets RefCOCO, RefCOCO+, RefCOCOg, ReferItGame and Flickr30kEntities. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] Figure 1 It is a block diagram of the unsupervised visual positioning system based on pseudo labels of the present invention.
[0031] Figure 2 It is a structural diagram of the unsupervised visual positioning system based on pseudo labels of the present invention. DETAILED DESCRIPTION
[0032] The present invention is further described below by way of examples in conjunction with the accompanying drawings.
[0033] like Figure 1 As shown, the pseudo-label-based unsupervised visual positioning system of the present invention includes a region proposal generation module, a region description generation module, and a query construction module. Its workflow is as follows:
[0034] 1: The region proposal generation module uses EVA-02 pre-trained on the LVIS dataset as the target detector, iou_threshold is set to 0.6, max_dets_per_image is set to 1000, and uses unlabeled pictures to generate regions that may contain entities in the image, the category of the entity, and the confidence:
[0035] b i ,c i ,s i =f detector (p)#(1)
[0036] Considering that the confidence of the target detector is unevenly distributed across categories, the confidence is adjusted as follows:
[0037]
[0038] Through the processing of the category bias eliminator, the confidence of entities of different categories conforms to the distribution with a mean of 0 and a standard deviation of 1. The k regions with the largest adjusted confidence in each image are selected as region proposals.
[0039] 2: The region description generation module uses Minigpt4 or LLaVA-Plus to generate region descriptions. Taking LLaVA-Plus as an example, temperature is set to 0 and num_beams is set to 3. Specifically, the prompt is designed: "Please use a sentence or phrase to briefly describe the image, focusing on the most important objects and not including any other objects. Your description can include attributes such as color, posture, clothing, etc., but it should not be too long." The cropped region proposal and the prompt are input into the multimodal large language model together to obtain a description of the main entities in the prompt area. Then use the description corrector to correct the wrong description
[0040] 3: Use the modified tree to extract the central entity and the attributes of the entity in the description. Use the prompt template "'Entity name' with 'attribute 1, attribute 2, attribute 3' attributes" to construct a pseudo query (for example, a man with the attributes of tall, wearing white clothes, and standing). Use the obtained pseudo labels to train the visual localization model.
[0041] IV: We compare the results with existing unsupervised methods on the RefCOCOg, RefCOCO, RefCOCO+, ReferItGame, and Flickr30K Entities datasets. Consistent with previous studies, we choose top-1 accuracy (%) as the evaluation metric. The predicted bounding box is considered correct if its IoU with the true value exceeds 0.5. When selecting region proposals, for RefCOCOg, RefCOCO+, and RefCOCO, the top 2, top 3, and top 3 objects are selected, respectively. In ReferItGame and Flickr30K Entities, the top 6 and top 7 objects are selected, respectively.
[0042] The average length of queries in the RefCOCOg, RefCOCO+, and RefCOCO datasets is 8.43, 3.61, and 3.53, respectively. The query length of RefCOCOg is generally larger than that of RefCOCO and RefCOCO+, which means that these queries may have richer attributes and more complex sentence structures. The proposed method shows significant improvement on the RefCOCOg dataset. Specifically, compared with the previous SOTA method (CLIP-VG), the proposed method achieves significant improvements of 9.02% / 8.59% / 9.30% on val-g / val-u / test-u splits, respectively. In addition, when evaluated on RefCOCOg using the same model as Pseudo-Q, the proposed method also achieves improvements of 8.57% / 11.77% / 10.65%, respectively. It is worth noting that although less supervision signals are used, the performance of the proposed method on this dataset is significantly better than the existing SOTA weakly supervised methods, with improvements of 6.14% / 6.24% / 6.85%, showing its effectiveness in handling complex queries. Significant improvements were also achieved on the RefCOCO dataset, with 2.10% / 1.63% / 1.57% improvements observed in the val / testA / testB segmentation compared to CLIP-VG. In addition, significant improvements of 3.38% / 1.46% / 5.68% were obtained compared to Pseudo-Q. On the RefCOCO+testB segmentation, the proposed method also achieved the SOTA of unsupervised methods.
[0043] In addition, on the ReferItGame and Flickr30K Entities datasets, compared with CLIP-VG, the proposed method achieved an improvement of 3.14% and 5.56%, and compared with Pseudo-Q, it achieved an improvement of 6.28% and 3.02%.
[0044] In summary, the present invention is aimed at the problem of unsupervised visual positioning, and attempts to utilize the rich cross-modal knowledge contained in the multimodal large language model to construct a semantically rich pseudo-label training model. It can reduce the impact of the pseudo-labels generated by the target detector on the category bias, and at the same time make the generated pseudo-labels contain entities and attributes of the open vocabulary. This method makes the generated pseudo-labels refer to a wide range of entity types and rich attributes, narrowing the gap between unsupervised and fully supervised visual positioning.
Claims
1. An unsupervised visual positioning system based on pseudo labels, characterized in that: include: Region proposal generation module, region description generation module, query construction module; The region proposal generation module is composed of a region proposal generator and a category bias eliminator. The module uses a pre-trained object detector to identify possible entity regions, categories and their confidences in an image, and uses the category bias eliminator to solve the problem of uneven distribution of confidences in categories. The region description generation module is composed of a region descriptor and a description corrector. The module uses the cross-modal knowledge contained in the multimodal large language model to generate descriptions of region proposals, and uses the description corrector to correct erroneous descriptions. The query building module builds a query through a decoration tree and a prompt template.
2. The unsupervised visual positioning system according to claim 1, characterized in that In the region proposal generation module, the region proposal generator is used to check the region where an entity may exist in the image slice, the entity category and its confidence, which is expressed as: b i ,c i ,s i =f detector (p)#(1) Among them, p represents an image; f detector represents the pre-trained target detector; b i ,c i ,s i They represent the bounding box coordinates, entity category and confidence of the i-th possible entity area in the image respectively; The category bias eliminator is used to eliminate the bias of the target detector for the category, and is expressed as: Among them, s i represents the confidence of the ith region; μ c represents the mean confidence of the regions detected in the data set that belong to the same category as the i-th region; σ c Represents the standard deviation of the confidence of the regions detected in the dataset that belong to the same category as the i-th region; Through the processing of the category bias eliminator, the confidence of entities of different categories conforms to the distribution with a mean of 0 and a standard deviation of 1. The k regions with the largest confidence in each image are selected as region proposals.
3. The unsupervised visual positioning system according to claim 2, characterized in that In the region description generation module, the region descriptor is used to generate a description of the region proposal to obtain rich attributes and category names of the open vocabulary; through the designed prompt words, the multimodal large language model outputs a description of the main entity in the prompt region; The description corrector uses WordNet to check whether the main entity of the generated description is consistent with the entity category obtained by the target detector; specifically, when the main entity and the entity category are defined as synonyms or one of them is a hypernym of the other in WordNet, the subject described by the multimodal large language model is considered to be correct and the generated description is retained; otherwise, the entity category is used as the description for the region.
4. The unsupervised visual positioning system according to claim 3, characterized in that The query construction module uses a modified tree to extract the main entities and their attributes in the region description; the modified tree uses CoreNLP's syntactic analysis to construct a syntactic tree, and then defines the priorities of different types of subtrees and uses subtrees with high priorities to replace parent nodes to construct a modified tree; the root node of the modified tree is the central entity, and the subtrees of the root node are the attributes of the central entity; then, a pseudo query is constructed using the prompt template "'entity name' with 'attribute 1, attribute 2, attribute 3' attributes".
5. The unsupervised visual positioning system according to any one of claims 1 to 4, characterized in that: The workflow is: (i) Using an object detector, we obtain a large number of unlabeled images from the Internet, and generate regions in the image that may contain entities, entity categories, and confidences through a region proposal generator; we use a category bias eliminator to recalculate the confidence of each region proposal; and we select the k regions with the highest confidence as region proposals for each image; (ii) Generate a description for each region proposal through the region description generator, and delete the wrong description through the description corrector, and retain the correct description; the description corrector uses WordNet to check whether the main entity of the generated description is consistent with the entity category obtained by the object detector; specifically, when the main entity and the entity category are defined as synonyms or one is the hypernym of the other in WordNet, the subject described by the multimodal large language model is considered correct and the generated description is retained; otherwise, the entity category is used as the description of the region; (iii) The central entity and the attributes of the entity in the region description are extracted by modifying the tree, and a large number of pseudo-labels are constructed using the prompt templates; finally, the constructed pseudo-labels are used to train the visual localization model.
Citation Information
Cited By
Target detection method based on unsupervised feature clustering and multi-modal large model collaborative iteration
CN121883895A
