Open-vocabulary object detection method based on inference score adjustment
By generating prototype visual representations of basic categories in an open-vocabulary object detection model and adjusting for cosine similarity, the problem of low prediction scores for novel categories is solved, thus improving detection performance.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- INNER MONGOLIA UNIVERSITY
- Filing Date
- 2025-05-16
- Publication Date
- 2026-05-26
Smart Images
Figure CN120451678B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to an open-vocabulary object detection method based on inference score adjustment, belonging to the field of object detection technology in machine vision. Background Technology
[0002] Object detection is a core research area in computer vision, aiming to identify targets of interest from static images or dynamic videos and accurately label their spatial locations within the image. This location information is typically represented by a bounding box, a rectangular box defined by the coordinates of the top-left and bottom-right corners of the image, used to cover the target area. Furthermore, the detection process needs to determine the target's category; for example, in traffic scenarios, the category might include vehicles, pedestrians, or traffic lights. With continuous technological advancements, object detection has achieved efficient and accurate identification and localization, and is widely applied in various real-world fields. For instance, it can automatically identify abnormal behavior or suspicious objects in security monitoring, assist in product identification and inventory management in smart cities, and be used in autonomous driving to perceive key elements in the road environment such as pedestrians, vehicles, and traffic signs.
[0003] Most current mainstream object detection methods rely on training models using a closed set of categories; these methods are collectively known as Closed Set Object Detection (CSOD). In the CSOD paradigm, the model training process is based on a predefined, finite set of categories, and each category requires sufficient manually labeled samples with accurate bounding boxes and category labels. This method achieves excellent detection performance on standard datasets (such as PASCAL VOC and MS COCO). However, its fundamental assumption is that the categories involved in the training and testing phases are completely identical; in other words, the model can only detect objects of categories it has "seen" during training. However, the real world is open and dynamically changing, with a vast number of object categories exhibiting a long-tail distribution, and new object types constantly emerging, making it virtually impossible to pre-enumerate all possible detection categories. Traditional CSOD methods lack generalization ability when facing these "unseen / novel categories," often misidentifying them as background or other similar categories, severely limiting their applicability in real-world open environments. This closed-category assumption is particularly weak in application scenarios that heavily rely on the recognition of diverse objects, such as autonomous driving, telemedicine, and industrial inspection.
[0004] To address the limitations of traditional object detection methods under the category closure assumption—such as insufficient generalization ability, high data annotation costs, and difficulty adapting to long-tail and novel categories in real-world environments—researchers have recently proposed a new research paradigm: Open-Vocabulary Object Detection (OVOD). The core objective of OVOD is to enable detection models to accurately identify and locate "new categories" during testing, even when certain categories have not been explicitly encountered during training (i.e., trained on "base categories"). Specifically, OVOD maps category names to a shared semantic space by introducing linguistic representations of categories (such as text embeddings of category names) or utilizing semantic knowledge provided by large-scale pre-trained language models (such as BERT, GPT, CLIP, etc.). This allows the model to identify unseen categories based on semantic similarity.
[0005] Although existing models have achieved good results on open vocabulary object detection datasets, these models still have the following problems: during the inference process, the classifier usually tends to favor known categories, while novel categories tend to be assigned lower confidence scores (usually below 0.5), which usually means that the model is uncertain about the target, and the prediction may be judged as invalid or ignored, thus affecting the model's detection performance. Summary of the Invention
[0006] To address the problem of poor detection capability for novel categories in existing open-vocabulary object detection models, this invention provides an open-vocabulary object detection method based on inference score adjustment.
[0007] An open-vocabulary object detection method based on inference score adjustment according to the present invention includes:
[0008] Training phase: Obtain the region of interest candidate boxes and the encoded category labels of the sample input image. Pass the region of interest candidate boxes through the feature alignment layer to obtain the mapping features. Then perform binary matching between the mapping features and the encoded category labels to obtain the open vocabulary object detection pre-trained model.
[0009] Inference Phase: An open-vocabulary object detection pre-trained model is used to obtain candidate regions of interest (ROIs) and encoded class labels for the detected images. The ROIs are then processed through a feature alignment layer to obtain mapped features. The mapped features are then compared with the encoded class labels using cosine similarity calculation to obtain the original cosine similarity score. Finally, the original cosine similarity score is adjusted.
[0010] Visual features of all basic category targets are extracted from the training sample set and clustered according to the basic category. The average features of each cluster are used as the prototype visual representation of the basic category. At the same time, the encoded basic category labels are obtained. The encoded basic category labels are mapped to the prototype visual representation of the basic category to obtain the mapping matrix. Then, the encoded novel category labels of the novel category labels in the detection image are obtained. Combined with the mapping matrix, the prototype visual representation of the novel category is obtained.
[0011] The calculation results involving novel categories in the cosine similarity calculation are replaced with the corresponding mapped features and encoded category labels using the prototype visual representation of the novel category to calculate the cosine similarity, resulting in an adjusted cosine similarity score. The object detection result is obtained by combining the adjusted cosine similarity score with the remaining original cosine similarity scores.
[0012] According to the open vocabulary object detection method based on inference score adjustment of the present invention, in the training phase, an image encoder pre-trained in the CLIP multimodal language model is used to obtain sample image features from the sample input image; then, a region candidate network is used to obtain region of interest candidate boxes from the sample image features.
[0013] According to the open vocabulary object detection method based on inference score adjustment of the present invention, the feature alignment layer obtains the mapping features based on the sample image features and the candidate boxes of the region of interest.
[0014] According to the open vocabulary object detection method based on inference score adjustment of the present invention, during the training phase, a text encoder is used to encode the category labels of the sample input images to obtain the encoded category labels; the category labels of the sample input images are embedded in the natural language context based on the prompt template and input to the text encoder.
[0015] According to the open vocabulary object detection method based on inference score adjustment of the present invention, in the training phase, cross-entropy loss is used to supervise the matching results of binary matching to complete the training process of the open vocabulary object detection pre-training model.
[0016] According to the open vocabulary object detection method based on inference score adjustment of the present invention, in the inference stage, an inference score adjustment module is used to adjust the original cosine similarity score;
[0017] The inference score adjustment module uses a frozen image encoder to extract ground truth visual features of all basic category targets; the frozen image encoder is obtained based on the training process of the open vocabulary object detection pre-trained model.
[0018] According to the open vocabulary object detection method based on inference score adjustment of the present invention, the inference score adjustment module uses the K-means clustering algorithm to cluster the ground truth visual features of all basic category targets according to the basic category.
[0019] According to the open vocabulary object detection method based on inference score adjustment of the present invention, the inference score adjustment module uses a pre-trained text encoder to encode the basic category labels to obtain the encoded basic category labels; at the same time, it uses a pre-trained text encoder to encode the novel category labels in the detection image to obtain the encoded novel category labels.
[0020] According to the open vocabulary object detection method based on reasoning score adjustment of the present invention, the encoded basic category labels are mapped to the prototype visual representation of the basic category, and the resulting mapping matrix is optimized by the least squares method to obtain the optimized mapping matrix; the encoded novel category labels are mapped to the visual space through the optimized mapping matrix to obtain the prototype visual representation of the novel category.
[0021] According to the open-vocabulary object detection method based on inference score adjustment of the present invention, the training sample set is obtained from public channels through keyword retrieval.
[0022] The beneficial effects of this invention are as follows: The method of this invention is used for open-vocabulary object detection when the model lacks novel category data. It enables the classifier of the open-vocabulary object detection model to adjust its inference score for novel category targets, thereby improving the detection capability and application potential of OVOD in real-world scenarios. This invention's method can achieve higher inference scores for novel categories, overcoming the problem of existing open-vocabulary object detection technologies where the classifier is biased towards known categories. Attached Figure Description
[0023] Figure 1 This is an example image showing how a classifier biases towards known categories, resulting in lower prediction scores for novel categories; the umbrellas in the two red boxes have scores of 0.15 and 0.16, respectively.
[0024] Figure 2 This is a schematic diagram of the training process for an open-vocabulary object detection pre-trained model;
[0025] Figure 3 This is the inference flowchart of an open-vocabulary object detection pre-trained model;
[0026] Figure 4 This is a comparison chart of the predicted scores before and after cosine similarity score adjustment; in the chart, (a) the airplane scores are 0.87 and 0.65 respectively, and the umbrella scores are 0.56, 0.86, 0.30, 0.27 and 0.89 respectively; (b) the airplane scores are 0.99 and 0.74 respectively, and the umbrella scores are 0.94, 0.98, 0.81, 0.82 and 0.99 respectively.
[0027] Figure 5This is a flowchart of the open vocabulary object detection method based on inference score adjustment described in this invention;
[0028] Figure 6 This is the network schematic diagram of the inference score adjustment module; in the diagram, GT Boxes represent GT annotation boxes, and α represents the weight;
[0029] Figure 7 This is a comparison chart of the image's true annotation, the detection results of the baseline "open vocabulary object detector," and the detection results of the method of this invention in the verification experiment. Detailed Implementation
[0030] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0031] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.
[0032] The present invention will be further described below with reference to the accompanying drawings, but this should not be construed as limiting the invention.
[0033] Combination Figures 1 to 6 As shown, the present invention provides an open vocabulary object detection method based on inference score adjustment, including: training phase: obtaining candidate boxes of regions of interest of sample input image and encoded category labels of sample input image, obtaining mapping features by passing the candidate boxes of regions of interest through feature alignment layer, and then performing binary matching between the mapping features and the encoded category labels to obtain an open vocabulary object detection pre-trained model;
[0034] Inference Phase: An open-vocabulary object detection pre-trained model is used to obtain candidate regions of interest (ROIs) and encoded class labels for the detected images. The ROIs are then processed through a feature alignment layer to obtain mapped features. The mapped features are then compared with the encoded class labels using cosine similarity calculation to obtain the original cosine similarity score. Finally, the original cosine similarity score is adjusted.
[0035] Visual features of all basic category targets are extracted from the training sample set and clustered according to the basic category. The average features of each cluster are used as the prototype visual representation of the basic category. At the same time, the encoded basic category labels are obtained. The encoded basic category labels are mapped to the prototype visual representation of the basic category to obtain the mapping matrix. Then, the encoded novel category labels of the novel category labels in the detection image are obtained. Combined with the mapping matrix, the prototype visual representation of the novel category is obtained.
[0036] The calculation results involving novel categories in the cosine similarity calculation are replaced with the corresponding mapped features and encoded category labels using the prototype visual representation of the novel category to calculate the cosine similarity, resulting in an adjusted cosine similarity score. The object detection result is obtained by combining the adjusted cosine similarity score with the remaining original cosine similarity scores.
[0037] Furthermore, during the training phase, an image encoder pre-trained on the CLIP multimodal language model is used to obtain sample image features from the sample input images; then, a region candidate network is used to obtain candidate boxes for regions of interest from the sample image features.
[0038] In this embodiment, the feature alignment layer obtains the mapped features based on the sample image features and the candidate bounding boxes of the region of interest.
[0039] Furthermore, during the training phase, a text encoder is used to encode the category labels of the sample input images to obtain encoded category labels; the category labels of the sample input images are embedded in the natural language context based on the cue template and input to the text encoder.
[0040] During the training phase, cross-entropy loss is used to supervise the matching results of binary matching, thus completing the training process of the open vocabulary object detection pre-training model.
[0041] Furthermore, during the reasoning stage, a reasoning score adjustment module is used to adjust the original cosine similarity score;
[0042] The inference score adjustment module uses a frozen image encoder to extract ground truth visual features of all basic category targets; the frozen image encoder is obtained based on the training process of the open vocabulary object detection pre-trained model.
[0043] The reasoning score adjustment module uses the K-means clustering algorithm to cluster the ground truth visual features of all base category targets according to the base category.
[0044] The inference score adjustment module uses a pre-trained text encoder to encode the basic category labels, resulting in encoded basic category labels; at the same time, it uses a pre-trained text encoder to encode the novel category labels in the detection image, resulting in encoded novel category labels.
[0045] The encoded base category labels are mapped to the prototype visual representations of the base categories. The resulting mapping matrix is then optimized using the least squares method to obtain the optimized mapping matrix. The encoded novel category labels are then mapped to the visual space through the optimized mapping matrix to obtain the prototype visual representations of the novel categories.
[0046] As an example, the training sample set was obtained from public channels through keyword retrieval.
[0047] This implementation method uses images or video frames from real-world scenes as the research object, and the specific detection categories can be flexibly set according to actual application needs. With the rapid development of Internet technology, image and video acquisition devices are becoming increasingly widespread. According to relevant statistics, the image and video content on the YouTube platform is currently growing at a rate of approximately 58 images and 3.6 videos per second. Users only need to focus on the novel target category to be detected and obtain relevant image data from public channels through keyword search to quickly build a dataset that fits the actual problem. The basic category data can be existing public databases (PASCAL VOC, Microsoft COCO, etc., see Table 1 for details). Since this implementation method does not rely on manually labeled information of novel categories during the training process, it reduces the dependence on large-scale labeling work.
[0048] Table 1 Summary of commonly used databases for object detection
[0049]
[0050] After establishing the training database, an open-vocabulary object detection detector can be trained using existing open-vocabulary object detection techniques. The flowchart for training an open-vocabulary object detection model is shown below. Figure 2 As shown, in the first step, the input image (training data) first passes through a convolutional neural network (such as ResNet50) pre-trained in the CLIP (Contrastive Language Vision Pre-trained Model) multimodal language model to extract features, resulting in input image features. These features are then fed into a Region Proposal Network (RPN) to extract regions of interest, i.e., candidate boxes. Finally, the input image features and the candidate boxes generated by the RPN are simultaneously input into a feature alignment layer (ROI Align layer) to obtain the mapping features of the candidate boxes on the image, i.e., candidate box features. In the second step, the category label of the target in the image is input as text information into the CLIP text encoder for encoding. To better model category semantics, this process uses a prompt template, such as the text format "A photo of a [CATEGORY] in the scene," embedding the category name into the natural language context. This prompt template helps activate prior knowledge in the language model, enhancing the alignment ability of visual and text features in the shared semantic space, thereby improving the model's understanding and transfer of category semantics. Through the two steps described above, the visual features of the candidate bounding boxes and the semantic features of the category labels can be obtained. These two types of features are then input into the binary matching module to achieve a one-to-one correspondence between labels and candidate regions. Finally, cross-entropy loss is used to supervise the matching results, completing the training process of the open-vocabulary object detection model.
[0051] After the model training is completed, an open-vocabulary object detection model trained on basic category data is obtained. Then, its inference process is executed, and the inference process is as follows: Figure 3 As shown. The inference process is largely consistent with the training process, with two main differences: 1) the label input consists of all category labels in the dataset; 2) the candidate box features and label embeddings are compared using cosine similarity instead of binary matching, and the prediction result is generated based on the similarity. However, during the inference process, the model's classifier often tends to favor known categories that appear in the training set, resulting in lower prediction scores for novel categories, such as... Figure 1 As shown in the figure, the inference scores for the umbrellas are all below 0.5. This usually means that the model is uncertain about the target, and the prediction may be judged as invalid or ignored, thus affecting the model's detection performance.
[0052] This invention studies classifiers in open-vocabulary object detection models and finds that models trained on known category data often tend to favor known categories in the training set during inference, resulting in lower prediction scores for novel categories and thus affecting the model's detection performance for novel categories. To address this problem and analysis, this invention proposes an Inference Score Adjusting (ISA) module, which utilizes the mapping between the visual space and linguistic space of basic categories to construct visual features for novel categories. These visual features are then used to adjust the score of the classifier in the open-vocabulary object detection model. (Based on the inference score before adjustment...) Figure 4 (a) and reasoning score after adjustment Figure 4 (b) The comparison chart of predicted scores is shown below. Figure 4 As shown.
[0053] In summary, this invention effectively solves the problem in existing open-vocabulary object detection methods where classifiers tend to favor basic categories and it is difficult to assign high inference scores to novel categories. The open-vocabulary object detection method based on inference score adjustment is a novel approach for object detection in real-world scenarios.
[0054] The specific implementation process of this invention will be further explained below with reference to the accompanying drawings.
[0055] The implementation of this invention is based on the two-stage detection framework of Faster R-CNN. In the first stage, following the standard Faster R-CNN method, target candidate boxes are generated by the region candidate box network. The second stage is modified to use an open vocabulary setting, replacing class-specific localizers with class-independent localizers. Furthermore, to achieve open vocabulary target classification, the trainable classifier weights are replaced with fixed language embeddings generated by CLIP, enabling the detector to operate in an open vocabulary environment. To more clearly illustrate the technical flow of this invention, the overall method will be explained below with reference to a flowchart, and the specific training steps are as follows: Figure 5 As shown, the black flowchart represents the training phase, and the red flowchart represents the testing phase, which is the core of this invention.
[0056] First, training samples are prepared according to the needs of actual application scenarios to provide a data foundation for subsequent model training. Second, in the candidate box generation stage, a Region Proposal Network (RPN) is used to generate regions of interest (ROIs) in the input image, i.e., candidate boxes. Then, the RPN-generated candidate boxes undergo RoI Align (RoI Align) operation on the feature maps to extract corresponding region features. Simultaneously, image labels combined with cue templates are input into a text encoder to generate label embeddings containing semantic information, which are also input into a binary matching module to achieve semantic alignment between candidate regions and category labels. Finally, the matching pairs obtained from binary matching are used to train the open-vocabulary object detector. During inference, the model first calculates the similarity between candidate box features and all category label embeddings to obtain preliminary prediction scores. Subsequently, these scores are corrected by the inference score adjustment module proposed in this invention to alleviate the classifier's bias towards basic categories, thereby improving the ability to recognize novel categories.
[0057] The first step is the preparation of training samples. Depending on the specific object detection requirements, suitable sample data can be obtained in various ways: for routine object detection tasks (such as pedestrian detection, vehicle detection, etc.), search engines can be used to automatically collect relevant images through keyword retrieval (such as "pedestrian," "vehicle," etc.); for general object detection, publicly available datasets such as PASCAL VOC and MS COCO can be used; while for certain specific category detection tasks (such as face recognition), dedicated datasets such as WIDER FACE and FDDB can be used. To balance generality and practical feasibility, this invention uses the MS COCO 2017 dataset as the experimental basis. The "train" subset is used to train the weakly supervised object detector, while the "val" subset is used to evaluate its performance. It is important to note that this invention only uses image and label information of basic categories during the training phase, without using any standard information or label information of novel categories. To further improve the model's performance and robustness, this invention also introduces data augmentation and multi-scale training strategies. For data augmentation, all training images are flipped horizontally, and the flipped samples are added to the training set to expand the data scale and improve the model's generalization ability. For multi-scale training, considering the significant differences in target size in real-world scenes, the model randomly selects a scale from {480, 576, 688, 864, 1200} as the shortest side size of the image while maintaining the image aspect ratio; simultaneously, to prevent excessively large images from causing GPU memory overflow, the longest side size is limited to no more than 2000 pixels. Through these training sample preparation and augmentation strategies, the model's adaptability to diverse scenes can be improved while ensuring training efficiency, providing better support for the practical application of open-vocabulary object detection tasks.
[0058] RPN generates candidate bounding boxes. This module obtains feature maps from a feature extraction network (such as ResNet50). Then, at each spatial location in this feature map, RPN places a predefined set of anchor boxes with various scales and aspect ratios, generating nine anchors at each location. These anchors are mapped back into the original image, representing candidate regions where objects may exist. Next, RPN uses a small convolutional sub-network to predict for each anchor. This sub-network typically consists of a 3×3 convolutional layer extracting local contextual features, followed by two 1×1 convolutional heads that output two results: a foreground / background binary classification score to determine whether the anchor contains an object; and a location regression offset to correct the anchor's coordinates to be closer to the ground truth. After NMS (Non-Maximum Suppression), RPN outputs a set of candidate boxes with high scores and low overlap for further processing. Finally, these candidate boxes and the feature map are input into RoIAlign to obtain candidate box features.
[0059] Bipartite matching. After the RPN generates candidate boxes, the corresponding candidate region feature representations are obtained, containing multiple region embeddings, each corresponding to a candidate box. For the descriptive text of the image, the CLIP text encoder is used to extract image-level label (i.e., word) embeddings, resulting in a set of word embeddings. After obtaining the region embeddings and word embeddings, the region-word alignment process aims to find the best-matching image region for each word. In this invention, this alignment is modeled as a bipartite graph matching problem, the objective of which is to minimize the matching cost matrix. This cost matrix C... ij The optimal matching process is obtained by calculating the inner product between the region embedding and the word embedding, where each element represents the similarity between the i-th region and the j-th word. This optimal matching process is formalized as an optimization problem aimed at minimizing the total cost of all matching pairs. After bipartite graph matching, a corresponding candidate region embedding can be assigned to each word.
[0060] An open-vocabulary object detector is trained. After binary matching, each label of the input image receives a corresponding candidate embedding. This invention employs a loss function based on binary cross-entropy for training, which considers both the matching scores of positive sample pairs and negative sample pairs from other categories. The specific training settings are as follows: total iterations: 90,000; learning rate strategy: during the warm-up phase, the learning rate linearly increases from 0 to 0.002 in the first 1,000 iterations. The model is trained for 90,000 iterations using an SGD optimizer with a batch size of 8, and the learning rate is reduced by a factor of 10 at iterations 60,000 and 80,000 to ensure gradual convergence and avoid overfitting.
[0061] The open-vocabulary object detection network trained through the above steps can achieve object detection without novel category annotations and labels. This invention primarily functions in the inference phase of the open-vocabulary object detection method, aiming to adjust the prediction scores of the model classifier. Specifically, after completing model training, an open-vocabulary object detection model trained based on basic category data is obtained, followed by the inference phase. The overall process is as follows: Figure 5 The red flowchart illustrates the process. The inference process of this invention remains consistent with the training phase in most steps, with the main differences being: 1) the input labels are the set of labels for all categories in the dataset; 2) the matching method between candidate box features and category label embeddings is changed from bipartite graph matching to cosine similarity calculation, and the final prediction result is generated based on the similarity score. The following will elaborate on the implementation details of each part of the testing phase:
[0062] Cosine similarity calculation. After the candidate boxes are generated by the RPN module, the model extracts the visual representations of these regions on the feature map. Simultaneously, a pre-trained text encoder (such as the text branch of CLIP) encodes all target category labels to obtain corresponding semantic vector representations. Next, the model evaluates the relevance of the candidate box features and category semantic vectors in the semantic space by calculating the cosine similarity. Cosine similarity measures the proximity of two vectors in a high-dimensional semantic space; the higher the similarity, the more likely the candidate region is to match the corresponding category semantics, thus achieving category discrimination and recognition.
[0063] Inference score adjustment. During cosine similarity calculation, the model's classifier often exhibits bias towards known categories, leading to relatively low prediction scores for novel categories and thus affecting the model's detection performance on unseen categories. To address this issue, this invention proposes an inference score adjustment method aimed at mitigating category bias and improving the model's ability to identify novel categories. For example... Figure 6 As shown, this invention first utilizes a frozen visual encoder to extract Ground Truth (GT) visual features for all basic categories from the training set. This visual encoder follows the open-vocabulary object detection training process described in this invention. For the GT region embedding of each basic category, this invention employs a K-means clustering algorithm to divide it into Z clusters and calculates the average feature of each cluster, thereby generating a prototype visual representation for each basic category, i.e., selecting a typical visual sample as the representative of that category. Next, a pre-trained text encoder is used to encode all basic category labels to obtain their corresponding semantic embedding representations. Based on the above processing, the visual prototype embeddings (corresponding to...) of each basic category can be obtained. Figure 6 The basic category prototype features), basic category text semantic embedding (corresponding to) Figure 6 (Based category label features) and novel category text semantic embeddings (corresponding to) Figure 6 The present invention aims to learn a mapping matrix W to map the text embeddings of base categories to their corresponding visual prototype embeddings. To this end, the present invention employs the least squares method to optimize W. After obtaining the mapping matrix W, the text embeddings of novel categories can be further mapped to the visual space through W, thereby obtaining the prototype visual embeddings of novel categories (corresponding to...). Figure 6 (Novel category prototype features). Therefore, for candidate box features processed by RPN and RoI Align, the classifier's score for the novel category can be recalculated using the prototype visual embedding of the novel category. Finally, the score of the novel category will replace the classifier's original score for the novel category. Note that this invention only re-scores the novel category and does not change the scores of the base categories.
[0064] Experiments demonstrate the effectiveness of the "open vocabulary object detection method based on inference score adjustment" of this invention. Table 2 shows the comparative data of the experimental results:
[0065] Table 2 Comparison of Experimental Results
[0066]
[0067] Novel, Base, and Overall represent metrics for novelty, base, and overall categories, respectively, used to evaluate detection accuracy on test samples and reflect the model's performance on different target categories (novel and base). 50 indicates an IoU (Intersection over Union) threshold of 0.5; mAP is mean average precision. Novel mAP is the primary metric for evaluating the performance of open-vocabulary object detection models. Comparative data shows that the proposed "open-vocabulary object detector + inference score adjustment" framework demonstrates a significant improvement in Novel mAP compared to the traditional "open-vocabulary object detector." Figure 7 The figure shows the experimental results, with all detected targets belonging to novel categories. The first row contains ground truth labels, the second row shows the detection results of the baseline "open-vocabulary object detector," and the third row shows the detection results of the "open-vocabulary object detection method based on inference score adjustment" proposed in this invention. It can be seen that this invention has significant advantages: the detection scores obtained for objects are much higher than those of detectors without inference score adjustment.
[0068] While the invention has been described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the invention. Therefore, it should be understood that many modifications can be made to the exemplary embodiments, and other arrangements can be designed without departing from the spirit and scope of the invention as defined by the appended claims. It should be understood that different dependent claims and features described herein can be combined in ways different from those described in the original claims. It is also understood that features described in conjunction with individual embodiments can be used in other described embodiments.
Claims
1. An open-vocabulary object detection method based on inference score adjustment, characterized in that... include: Training phase: Obtain the region of interest candidate boxes and the encoded category labels of the sample input image. Pass the region of interest candidate boxes through the feature alignment layer to obtain the mapping features. Then perform binary matching between the mapping features and the encoded category labels to obtain the open vocabulary object detection pre-trained model. Inference stage: An open-vocabulary object detection pre-trained model is used to obtain candidate boxes of regions of interest and encoded category labels of the detected image. The candidate boxes of regions of interest are passed through a feature alignment layer to obtain mapped features. Then, the cosine similarity between the mapped features and the encoded category labels is calculated to obtain the original cosine similarity score. Then, the original cosine similarity score is adjusted: Visual features of all basic category targets are extracted from the training sample set and clustered according to the basic category. The average features of each cluster are used as the prototype visual representation of the basic category. Simultaneously obtain the encoded basic category label; The encoded base category labels are mapped to the prototype visual representations of the base categories to obtain the mapping matrix; After obtaining the encoded novel category label in the detected image, the novel category label is combined with the mapping matrix to obtain the prototype visual representation of the novel category; The calculation results involving novel categories in the cosine similarity calculation are replaced with the corresponding mapping features and encoded category labels using the prototype visual representation of the novel category to calculate the cosine similarity, resulting in the adjusted cosine similarity score. The object detection result is obtained by combining the adjusted cosine similarity score and the original cosine similarity score.
2. The open-vocabulary object detection method based on inference score adjustment according to claim 1, characterized in that, During the training phase, the image encoder pre-trained in the CLIP multimodal language model is used to obtain sample image features from the sample input images; Then, a region candidate network is used to obtain candidate boxes for regions of interest from the features of the sample images.
3. The open-vocabulary object detection method based on inference score adjustment according to claim 2, characterized in that, The feature alignment layer obtains mapped features based on sample image features and region of interest candidate boxes.
4. The open-vocabulary object detection method based on inference score adjustment according to claim 1, characterized in that, During the training phase, a text encoder is used to encode the category labels of the sample input images to obtain the encoded category labels. The category labels of the sample input images are embedded in the natural language context based on the cue template and input to the text encoder.
5. The open-vocabulary object detection method based on inference score adjustment according to claim 1, characterized in that, During the training phase, cross-entropy loss is used to supervise the matching results of binary matching, thus completing the training process of the open vocabulary object detection pre-training model.
6. The open-vocabulary object detection method based on inference score adjustment according to claim 1, characterized in that, During the reasoning phase, the original cosine similarity score is adjusted using the reasoning score adjustment module. The inference score adjustment module uses a frozen image encoder to extract ground truth visual features of all basic category targets; the frozen image encoder is obtained based on the training process of the open vocabulary object detection pre-trained model.
7. The open-vocabulary object detection method based on inference score adjustment according to claim 6, characterized in that, The reasoning score adjustment module uses the K-means clustering algorithm to cluster the ground truth visual features of all base category targets according to the base category.
8. The open-vocabulary object detection method based on inference score adjustment according to claim 7, characterized in that, The inference score adjustment module uses a pre-trained text encoder to encode the basic category labels, resulting in encoded basic category labels; at the same time, it uses a pre-trained text encoder to encode the novel category labels in the detection image, resulting in encoded novel category labels.
9. The open-vocabulary object detection method based on inference score adjustment according to claim 8, characterized in that, The encoded base category labels are mapped to the prototype visual representations of the base categories. The resulting mapping matrix is then optimized using the least squares method to obtain the optimized mapping matrix. The encoded novel category labels are then mapped to the visual space through the optimized mapping matrix to obtain the prototype visual representations of the novel categories.
10. The open-vocabulary object detection method based on inference score adjustment according to claim 1, characterized in that, The training sample set was obtained from public channels through keyword retrieval.