Open vocabulary target detection method and system based on visual retrieval enhancement prompt

Abstract

The invention relates to the technical field of computer vision, discloses a visual retrieval enhancement prompt-based open vocabulary target detection method and system, and aims to solve the problem of poor rare category detection performance caused by training data long tail distribution in open vocabulary target detection. The method comprises two stages of off-line construction of a visual concept knowledge base and on-line retrieval enhanced reasoning, wherein a large language model is utilized to generate discriminative visual descriptors for each category, and the discriminative visual descriptors are encoded by a CLIP text encoder and then stored in an FAISS vector database; during reasoning, global visual features of an input image are extracted through a CLIP image encoder, the most relevant descriptor in the knowledge base is retrieved and added to an original category name prompt, and an enhanced prompt is formed and input to an open vocabulary detector to complete detection. Experiments show that the method obviously improves the detection performance in COCO and LVIS benchmark tests, and especially obviously improves the detection effect on rare categories.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to an open vocabulary target detection method and system based on visual retrieval enhancement prompts. Background Technology

[0002] Object detection is one of the core tasks of computer vision. Traditional object detectors need to be trained on a fixed set of predefined categories. When adding new categories, data needs to be collected, labeled, and the model needs to be trained again, which is costly and inefficient. Open-vocabulary object detection utilizes the cross-modal alignment capability of visual language models to detect unseen categories, and has become a research hotspot.

[0003] Current mainstream open-vocabulary object detectors (such as Groundingdino) are based on the Transformer architecture and achieve detection through visual-text cross-modal fusion. However, due to the long-tail distribution of training data, their detection performance for rare categories is poor. This is because visual examples and textual descriptions of rare categories are scarce, making it difficult for the model to learn rich visual patterns and easily leading to confusion with visually similar categories.

[0004] In the field of natural language processing, Retrieval Augmented Generation (RAG) effectively alleviates the problem of model knowledge blind spots by dynamically retrieving information from external knowledge bases to assist in the generation task. Introducing the RAG concept into open-vocabulary object detection, constructing a visual concept knowledge base, and dynamically retrieving relevant descriptors to enhance prompts holds promise for solving the problem of poor performance in rare category detection. Summary of the Invention

[0005] The purpose of this invention is to provide an open vocabulary target detection method and system based on visual retrieval enhancement prompts, which solves the problem of poor performance of existing open vocabulary detectors in rare category detection scenarios with long-tail distribution.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] An open-vocabulary target detection method based on visual retrieval enhancement prompts includes the following steps:

[0008] S1: Offline construction of a visual concept knowledge base:

[0009] S11: Category Visual Descriptor Generation: For each target category in the open vocabulary target detection dataset, a structured prompt template is constructed and input into a large language model to generate a set of discriminative visual descriptors for that category; the visual descriptors contain visual attribute information such as shape, color, structure, and contextual patterns, and adopt a unified grammatical structure; the generated descriptors are filtered to remove duplicates, non-visual attributes, and grammatically incorrect descriptors;

[0010] S12: Descriptor Feature Encoding: For each category-descriptor pair, construct a combined text representation, input it into the pre-trained CLIP text encoder, and obtain a normalized descriptor feature embedding vector;

[0011] S13: Vector Database Construction: The descriptor feature embedding vectors are associated with the corresponding category metadata and descriptor text, and stored in the FAISS vector database to form a visual concept knowledge base;

[0012] S2: Online retrieval-enhanced reasoning:

[0013] S21: Global Image Feature Extraction: The input image to be detected is preprocessed and input into the pre-trained CLIP image encoder to extract the normalized global visual feature vector of the image.

[0014] S22: Relevant descriptor retrieval: Using the global visual feature vector of the image as the query vector, similarity retrieval is performed in the visual concept knowledge base using cosine similarity measurement to obtain the top k visual descriptors with similarity higher than a set threshold; for each category, a maximum of a set number of the highest similarity descriptors are retained.

[0015] S23: Enhanced cue construction: The retrieved visual descriptors are dynamically appended to the original category name cue, and the image conditional enhanced cue is formed using a period-separated format; if the length of the enhanced cue exceeds the token length limit of the detector, it is segmented into multiple coherent segments;

[0016] S24: Open vocabulary object detection: Input the enhanced cue and the image to be detected into the pre-trained open vocabulary object detector, and output the bounding box of the object detection and the corresponding confidence score.

[0017] Furthermore, the structured prompt template described in step S11 is used to constrain the grammatical structure of descriptors generated by large language models.

[0018] Furthermore, the combined text in step S12 is represented as a string concatenation of category name and descriptor.

[0019] Furthermore, the threshold value in step S22 is in the range of 0.2-0.3.

[0020] Furthermore, the original category name hint mentioned in step S23 is a text formed by connecting all category names in the dataset with periods as separators.

[0021] This invention also provides an open vocabulary target detection system based on visual retrieval enhancement prompts, comprising:

[0022] S31: Knowledge Base Construction Module: Used for offline construction of a visual concept knowledge base, including a descriptor generation unit, a feature encoding unit, and a vector storage unit; the descriptor generation unit generates and filters category visual descriptors through a large language model; the feature encoding unit generates descriptor feature embeddings through a CLIP text encoder; the vector storage unit stores the feature embeddings in the FAISS vector database;

[0023] S32: Retrieval Enhancement Inference Module: Used to perform target detection online, including an image feature extraction unit, a descriptor retrieval unit, a cue enhancement unit, and a detection unit; the image feature extraction unit extracts global visual features of the input image through a CLIP image encoder; the descriptor retrieval unit retrieves relevant descriptors in the knowledge base based on image features; the cue enhancement unit constructs image-conditional enhanced cue; the detection unit inputs the enhanced cue and the image to an open vocabulary detector and outputs the detection results.

[0024] Furthermore, the knowledge base construction module and the retrieval-enhanced reasoning module are deployed on GPU computing devices, and the visual encoder and text encoder of the CLIP model maintain weight freeze throughout the process.

[0025] The beneficial effects of this invention are as follows:

[0026] (1) The proposed visual retrieval enhancement cue framework guides the detector to focus on category attributes related to the image context by dynamically constructing image conditional enhancement cue, thereby improving the accuracy and generalization ability of detection.

[0027] (2) Visual descriptors generated by large language models effectively compensate for the lack of training data for rare categories and significantly improve the detection performance of rare categories in long-tailed distribution scenarios.

[0028] (3) Enhanced prompts use a period-separated format, which is compatible with the input requirements of existing open vocabulary detectors. No fine-tuning of the detector is required, and it has good compatibility and ease of use. Attached Figure Description

[0029] Figure 1 A schematic diagram of an open lexical object detection framework based on visual retrieval-enhanced prompts;

[0030] Figure 2 The following is a detailed architecture diagram of the VRAP framework, where (a) is the offline knowledge base construction module and (b) is the online retrieval enhanced reasoning module;

[0031] Figure 3 Example diagram of category visual descriptors;

[0032] Figure 4This is a comparison chart of the detection results of the baseline model and the method of this invention. Detailed Implementation

[0033] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the invention. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0034] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0035] Example 1

[0036] Offline construction of a visual concept knowledge base, as described in this embodiment. Figure 2 (a) The specific steps are as follows:

[0037] (1) Experimental environment preparation: Deploy two NVIDIA RTX 4090 GPUs computing devices, use the CLIP ViT-B / 32 model as the visual language model, keep the weights frozen; select DeepSeek-V3.1 as the large language model; build a structured prompt template, the template instruction is "generate about 10 discriminative visual descriptors for category {c}, including only shape, color, structure, and contextual features, format requirements: use 'is' to describe inherent attributes, 'has' to describe components, 'with' to describe features, and 'often' to describe contextual patterns.

[0038] (2) Category Visual Descriptor Generation: For the 80 target categories in the COCO 2017 dataset and the 1203 target categories in the LVIS dataset, structured hint templates were input into DeepSeek-V3.1 to generate descriptor sets. Filtering operations were performed on the generated descriptors: duplicate descriptors were removed, non-visual attribute descriptors such as "makes a sound" were eliminated, and descriptors with grammatical errors were discarded. The final result was 816 descriptors from the COCO dataset (average 10.2 per class) and 11410 descriptors from the LVIS dataset (average 9.5 per class). Some descriptor examples are provided. Figure 3 .

[0039] (3) Descriptor feature encoding: For each category-descriptor pair, construct a combined text of "category name + descriptor", such as "giraffe + has a long neck", input to CLIP text encoder, and output a 512-dimensional normalized feature embedding vector. This operation can solve the ambiguity problem of descriptors with the same name in different categories, such as distinguishing between "mouse's long tail" and "elephant's long tail".

[0040] (4) Construction of FAISS Vector Database: The feature embedding vectors are associated with the corresponding category metadata and descriptor text and stored in the FAISS vector database to form a visual concept knowledge base; the feature embedding vectors have a dimension of 512, and the database supports fast retrieval of cosine similarity.

[0041] Example 2

[0042] Online retrieval enhanced reasoning and object detection, this embodiment corresponds to the accompanying drawings. Figure 2 (b) The specific steps are as follows:

[0043] (1) Image preprocessing and feature extraction: Obtain the image to be detected and perform preprocessing operations: adjust the image size to 224×224 and normalize it using the mean and standard deviation of the ImageNet dataset; input the preprocessed image into the CLIP ViT-B / 32 image encoder to extract a 512-dimensional L2 normalized global visual feature vector.

[0044] (2) Relevant descriptor retrieval: Using the global visual feature vector of the image as the query vector, cosine similarity retrieval is performed in the FAISS knowledge base constructed in Example 1; the similarity threshold τ=0.3 is set, and the top-k highly relevant descriptors are retrieved; to prevent prompt overload, a maximum of 3 descriptors with the highest similarity scores are retained for each category.

[0045] (3) Enhanced hint construction: Construct the original category name hint by connecting all category names in the dataset with periods. For example, the original hint for the COCO dataset is "person.bicycle.car...". Dynamically append the retrieved descriptors to the corresponding category names to form the enhanced hint, in the format "category name.category name + descriptor 1.category name + descriptor 2...". If the length of the enhanced hint exceeds the 256 token limit of Groundingdino, it is split into multiple coherent segments.

[0046] (4) Open vocabulary target detection: IDEA-Research / Groundingdino-base was selected as the target detector, and the bounding box threshold was set to 0.3 and the text threshold was set to 0.2. The enhanced cue fragment and the image to be detected were input into the detector in sequence, the results of multiple detections were merged, and the bounding box of the target and the corresponding confidence score were output to complete the detection.

[0047] Experimental verification and result analysis

[0048] (1) Experimental datasets: The COCO 2017 validation set (5000 images, 80 categories) and the LVIS validation set (19800 images, 1203 categories) were selected; the evaluation metrics for the COCO dataset were AP and AP. 50 AP 75 AP S AP M AP L AR@1; The evaluation metrics for the LVIS dataset are AP and AP. r (Rare category), AP c (Common Categories), AP f (Frequency category).

[0049] (2) Experimental Results

[0050] COCO dataset validation: Experimental results are shown in Tables 1 and 2. The AP of the baseline model Groundingdino is 52.34. After adopting the method of this invention (τ=0.3), the AP is improved to 53.21, an improvement of 1.7. Among them, the APS of small targets is improved from 33.04 to 34.41, an improvement of 4.2%. Ablation experiments show that the prompt construction strategy of full-category period separation has the best performance, with an AP of 53.21, which is better than the strategy of only retrieving category connections and bracket format connections.

[0051] Validation on the LVIS dataset: Experimental results are shown in Table 3. The baseline model's AP is 25.1, and AP... r The AP was 16.3; after adopting the method of this invention (τ=0.3), the AP increased to 26.8. r Upgraded to 19.1, AP r The increase reached 2.8, with a relative improvement rate of 17.2%; the qualitative comparison results correspond to the attached figure. Figure 4 The baseline model missed detecting rare categories such as "surfboard", "sausage" and "street lamp". The method of this invention successfully locates the above targets by enhancing prompts.

[0052] Figure 1 This is a schematic diagram illustrating the overall process of the open vocabulary target detection framework based on visual retrieval enhancement prompts described in this invention. The diagram demonstrates the core logic of the VRAP framework and intuitively illustrates the working principle of the retrieval enhancement prompt mechanism.

[0053] Figure 2 This is a detailed architecture breakdown diagram of the VRAP framework of this invention, which includes two interdependent functional modules:

[0054] Figure 2(a) The offline visual descriptor generation and knowledge base construction module. The diagram illustrates the process steps: For each category in the target dataset, structured prompts are input into a large language model (LLM) to generate discriminative visual descriptors; after constructing combined text for the category-descriptor pairs, the text is input into the CLIP text encoder to complete feature embedding; finally, the feature vectors are associated with the corresponding metadata and stored in the FAISS vector database to form a visual concept knowledge base.

[0055] Figure 2 (b) is the online visual retrieval, enhancement, and inference module. The figure illustrates the workflow steps: After preprocessing the input image to be detected, it is input into the CLIP image encoder to extract the global visual feature vector; using this vector as the query, the top-k highly relevant descriptors are obtained by searching the FAISS database using cosine similarity; the retrieved descriptors are dynamically appended to the original category hints to generate image-conditional enhancement hints; finally, the enhancement hints and the input image are input together into an open vocabulary detector (such as Groundingdino) to output the target bounding box and confidence score.

[0056] Figure 3 This image shows an example of category visual descriptors generated using a large language model, as presented in this invention. The image selects four categories from the COCO dataset (person, zebra, dog, and umbrella) and two categories from the LVIS dataset (ottoman and paddle), listing 3-5 typical visual descriptors for each category. These descriptors cover attributes such as shape, color, structure, and context. For example, "person" corresponds to "has two legs for walking" and "often wears clothes," while "zebra" corresponds to "covered with black and white stripes," visually demonstrating the discriminative and structured features of the descriptors.

[0057] Figure 4 This is a qualitative comparison of the detection results of the VRAP framework described in this invention and the baseline Groundingdino model on the LVIS validation set. The figure presents the detection results of the same set of input images in a top-bottom column format: the top column shows the detection results of the baseline model, marking targets that the baseline model missed or falsely detected (such as rare categories like surfboard, sausage, and street lamp); the bottom column shows the detection results of the VRAP framework, marking targets successfully detected through augmented cues. The comparison clearly demonstrates the advantages of this invention in improving the accuracy of rare category detection.

[0058] Table 1. Performance comparison of baseline model and VRAP on the COCO validation set under different retrieval thresholds.

[0059] Model / Threshold AP AP50 AP75 APs APm APl AR@1 AR@10 AR@100 Baseline 52.34 68.28 57.42 33.04 56.45 68.13 37.70 61.10 64.90 VRAP (τ = 0.2) 51.01 66.03 55.89 32.09 54.44 68.52 37.60 58.60 60.70 VRAP (τ = 0.25) 53.03 68.75 58.20 34.34 56.95 69.45 38.90 60.80 62.90 VRAP (τ = 0.3) 53.21 69.28 58.38 34.41 57.09 69.23 38.70 61.50 64.60

[0060] The comparison results in Table 1 clearly demonstrate that VRAP outperforms the baseline detector. Without retrieval enhancement, Groundingdino achieves a localization accuracy of 52.34, with an AP50 of 68.28, exhibiting good localization accuracy at an IoU threshold of 0.5. However, the model has limitations in small object detection due to the semantic ambiguity of small-scale features (APsmall = 33.04). CLIP ViT-B / 32 extracts global image features and retrieves top-k descriptors from the knowledge base (k=3, up to 3 descriptors per category). Similarity thresholds τ are set to 0.3, 0.25, and 0.2. When VRAP with the optimal threshold τ = 0.3 is applied, we observe significant performance improvements: AP increases to 53.21 (+1.7 relative improvement), AP50 rises to 69.28, and AP75 reaches 58.38. Small object detection showed a significant enhancement (APsmall increased from 33.04 to 34.41, +4.2), indicating that the retrieved attribute descriptors provided supplementary visual cues, enhancing the perception of subtle features. Recall metrics (AR@1 improved from 37.70 to 38.70) suggest that the enhanced cues improved initial object recognition. Notably, the performance gain was threshold-dependent. When the threshold was relaxed to τ = 0.25, performance slightly decreased (AP = 53.03), but remained above baseline, suggesting that a lower threshold might introduce noisy descriptors, thus weakening the discriminability of the cues. Further lowering the threshold to τ = 0.2 resulted in an AP drop to 51.01, below baseline, confirming that only highly relevant descriptors effectively guided the detector.

[0061] Table 2. Impact of different prompting construction strategies on VRAP performance (τ=0.3, COCO validation set) leading to performance lower than baseline.

[0062] Strategy AP AP50 AP75 APs APm APl AR@1 AR@10 AR@100 All categories of splicing 53.21 69.28 58.38 34.41 57.09 69.23 38.70 61.50 64.60 Search only for splicing 50.87 66.20 55.86 34.91 54.23 66.97 38.40 59.80 61.70 bracket format 46.73 59.08 51.55 27.10 50.22 65.22 36.10 51.40 51.90

[0063] We further analyzed the impact of the cue construction strategy at a similarity threshold τ = 0.3. The experimental results in Table 2 show that full-class concatenation performs best, with an AP of 53.21. Conversely, retrieved-only concatenation is a cue construction method where only categories retrieved based on image content and their associated descriptors are dynamically incorporated into the cue. This method results in a significant drop in AP to 50.87. This degradation is primarily due to the limitations of the CLIP-based retrieval mechanism. Due to visual blurring or sparse representation, the retrieval process may fail to match all correct categories in the image. As shown in Table 2, the reduced recall indicates that true categories may be omitted. In contrast, full concatenation maintains the integrity of the cue by including all dataset categories. This approach preserves semantic relationships, avoids contextual breaks, and supports robust cross-category alignment. Furthermore, changing the dot-separated format to the bracketed format "category name (descriptor1, descriptor2, descriptor3)" causes a significant drop in AP to 46.73. This format may disrupt the detector's parsing of phrase structures, as Groundingdino relies on dot-separated category descriptor alignment.

[0064] Analysis. VRAP enhances open-vocabulary object detection performance through image-conditional semantic cues. The optimal retrieval threshold τ = 0.3 produces the most significant performance improvement while maintaining a balanced improvement across all object scales. VRAP exhibits particularly strong performance on small and medium-sized objects (AP). S Increased by 1.37, AP M The increase of 0.64 indicates that additional semantic cues help compensate for limited visual information in small object instances. Furthermore, preserving all categories in the cues, even without augmentation, is essential for maintaining the detector's overall recognition capability. Third, the syntactic structure of the augmented cues significantly impacts detection performance; the dot-separated format proves to be significantly superior to the bracket-annotated format. Although the improvements are modest, they are quite significant in open-vocabulary scenarios, as they validate that external knowledge bases can compensate for limitations in the model's internal representation. However, the performance improvements are limited by the relatively uniform category distribution in the COCO dataset, which consists almost entirely of frequent object categories.

[0065] Table 3. Performance comparison of the baseline model and VRAP on the LVIS validation set under different retrieval thresholds.

[0066] method AP APr APc APf Baseline 25.1 16.3 26.5 28.9 VRAP (τ = 0.2) 25.6 17.8 26.6 29.0 VRAP (τ = 0.25) 26.2 18.4 26.9 29.3 VRAP (τ = 0.3) 26.8 19.1 27.2 29.6

[0067] Table 3 summarizes the performance of our baseline model (IDEA-Research / grounded-dino-base) and VRAP for descriptor retrieval at different similarity thresholds τ. The baseline AP is 25.1, with APr, APc, and APf of 16.3, 26.5, and 28.9, respectively. VRAP with a similarity threshold τ = 0.2 improves the AP to 25.6 (+0.5). r The score improved to 17.8 (+1.5), indicating that even loosely correlated descriptors can provide useful semantic cues.

[0068] When τ = 0.25, performance further improves: AP reaches 26.2 (+1.1) and APr reaches 18.4 (+2.1), indicating that moderately selective retrieval improves suggestion quality. The best results are achieved at τ = 0.3, with VRAP at 26.8 (+1.7). r The value was 19.1 (+2.8). This indicates that a stricter retrieval threshold helps filter out noisy descriptors, thus providing more relevant visual cues. This represents the baseline AP. r This represents a 17.2% increase, highlighting VRAP's powerful ability to compensate for the scarcity of data in rare categories. It is worth noting that AP c and AP f It is also continuously being improved, which confirms that VRAP is beneficial for all category types without sacrificing the performance of frequently used categories.

Claims

1. An open-vocabulary target detection method based on visual retrieval enhancement prompts, characterized in that, Includes the following steps: S1: Offline construction of a visual concept knowledge base: S11: Category Visual Descriptor Generation: For each target category in the open vocabulary target detection dataset, a structured cue template is constructed, input into a large language model, and a set of discriminative visual descriptors for that category is generated. The visual descriptor contains visual attribute information such as shape, color, structure, and contextual pattern, and adopts a unified syntax structure; the generated descriptors are filtered to remove duplicates, descriptors with non-visual attributes, and descriptors with syntax errors. S12： Descriptor feature encoding: For each category-descriptor pair, construct a combined text representation, input it into the pre-trained CLIP text encoder, and obtain a normalized descriptor feature embedding vector; S13: Vector Database Construction: The descriptor feature embedding vectors are associated with the corresponding category metadata and descriptor text, and stored in the FAISS vector database to form a visual concept knowledge base; S2: Online retrieval-enhanced reasoning: S21: Global Image Feature Extraction: The input image to be detected is preprocessed and input into the pre-trained CLIP image encoder to extract the normalized global visual feature vector of the image. S22: Relevant descriptor retrieval: Using the global visual feature vector of the image as the query vector, similarity retrieval is performed in the visual concept knowledge base using cosine similarity measurement to obtain the top k visual descriptors with similarity higher than a set threshold; for each category, a maximum of a set number of the highest similarity descriptors are retained. S23: Enhanced cue construction: The retrieved visual descriptors are dynamically appended to the original category name cue, forming image-conditional enhanced cues using a period-separated format; If the length of the enhanced cue exceeds the token length limit of the detector, it is split into multiple coherent segments; S24: Open vocabulary object detection: Input the enhanced cue and the image to be detected into the pre-trained open vocabulary object detector, and output the bounding box of the object detection and the corresponding confidence score.

2. The method according to claim 1, characterized in that, The structured prompt template described in step S11 is used to constrain the grammatical structure of descriptors generated by large language models.

3. The method according to claim 1, characterized in that, The combined text in step S12 is represented as a string concatenation of the category name and the descriptor.

4. The method according to claim 1, characterized in that, The threshold value in step S22 is in the range of 0.2-0.

3.

5. The method according to claim 1, characterized in that, The original category name hint mentioned in step S23 is a text formed by connecting all category names in the dataset with periods as separators.

6. An open vocabulary target detection system based on visual retrieval enhancement prompts, characterized in that, include: S31: Knowledge Base Construction Module: Used for offline construction of a visual concept knowledge base, including a descriptor generation unit, a feature encoding unit, and a vector storage unit; the descriptor generation unit generates and filters category visual descriptors through a large language model; the feature encoding unit generates descriptor feature embeddings through a CLIP text encoder; the vector storage unit stores the feature embeddings in the FAISS vector database; S32: Retrieval Enhancement Inference Module: Used to perform target detection online, including an image feature extraction unit, a descriptor retrieval unit, a cue enhancement unit, and a detection unit; the image feature extraction unit extracts global visual features of the input image through a CLIP image encoder; The descriptor retrieval unit retrieves relevant descriptors from the knowledge base based on image features; the cue enhancement unit constructs enhanced cuees with image conditionalization; and the detection unit inputs the enhanced cuees and the image into the open vocabulary detector and outputs the detection results.

7. The system according to claim 6, characterized in that, The knowledge base construction module and the retrieval-enhanced reasoning module are deployed on GPU computing devices, and the visual encoder and text encoder of the CLIP model maintain weight freeze throughout the process.

Images