Open vocabulary object detection method based on reasoning score adjustment

Abstract

The invention discloses an open vocabulary object detection method based on reasoning score adjustment, and belongs to the technical field of object detection in machine vision. The method aims at solving the problem that in existing open vocabulary object detection, a model is poor in novel category detection capacity. Comprising the steps of obtaining an open vocabulary object detection pre-training model in a training stage; in the reasoning stage, an open vocabulary object detection pre-training model is adopted to obtain mapping features of a detection image, and cosine similarity calculation is carried out on the mapping features and coded category labels to obtain an original cosine similarity score; obtaining prototype visual representation of a novel category, replacing the corresponding mapping features with the prototype visual representation of the novel category according to a calculation result related to the novel category in cosine similarity calculation, and performing cosine similarity calculation with the encoded category label by adopting the prototype visual representation of the novel category to obtain an adjusted cosine similarity score; and combining the adjusted cosine similarity score with other original cosine similarity scores to obtain an object detection result. The method is used for open vocabulary object detection.

Description

Technical Field

[0001] The present invention relates to an open vocabulary object detection method based on inference score adjustment, and belongs to the technical field of object detection in machine vision. Background Art

[0002] Object detection is one of the core research directions in computer vision, aiming to identify objects of interest from static images or dynamic videos and accurately mark their spatial locations in the image. This location information is usually represented by a bounding box, which is a rectangular box defined by the coordinates of the upper left and lower right corners of the image to cover the target area. In addition, the detection process also needs to determine the category to which the target belongs. For example, in traffic scenarios, the categories may include vehicles, pedestrians, or traffic lights. With the continuous evolution of technology, object detection has been able to achieve efficient and accurate identification and positioning, and is widely used in many fields in the real world. For example, in security monitoring, it can automatically identify abnormal behavior or suspicious objects, assist in product identification and inventory management in smart cities, and be used in autonomous driving to perceive key elements such as pedestrians, vehicles, and traffic signs in the road environment.

[0003] Most current mainstream object detection methods rely on a closed set of categories for model training, collectively referred to as closed set object detection (CSOD). Within the CSOD paradigm, model training is based on a predefined, limited set of categories, each of which requires sufficient, manually annotated examples with precise bounding boxes and class labels. This approach achieves excellent detection performance on standard datasets such as PASCAL VOC and MS COCO. However, its fundamental assumption is that the categories used during training and testing are identical. In other words, the model can only detect objects from categories it has "seen" during training. However, the real world is open and dynamic, with a large number of object categories and a long-tail distribution. New object types are constantly emerging, making it nearly impossible to pre-emptively enumerate all possible detection categories. Traditional CSOD methods lack generalization ability when faced with these "unseen" (novel) categories, often misclassifying them as background or other similar categories, severely limiting their applicability in real-world open environments. This closed category assumption is particularly weak in application scenarios such as autonomous driving, telemedicine, and industrial inspection that rely heavily on the recognition of diverse objects.

[0004] To address the challenges faced by traditional object detection methods under the closed-category assumption, such as insufficient generalization, high data annotation costs, and difficulty adapting to long-tail and novel categories in real-world environments, researchers have recently proposed a new research paradigm: open-vocabulary object detection (OVOD). The core goal of open-vocabulary object detection is to enable the detection model to accurately identify and localize "new categories" during testing, even when the model has not explicitly seen certain categories during training (i.e., trained on "base categories"). Specifically, OVOD methods map category names into a shared semantic space by introducing linguistic representations of categories (such as text embeddings of category names) or leveraging semantic knowledge provided by large-scale pre-trained language models (such as BERT, GPT, and CLIP). This allows the model to recognize unseen categories based on semantic similarity.

[0005] Although existing models have achieved good results on open vocabulary object detection datasets, these models still have the following problems: during the inference process, the classifier usually tends to favor seen categories, while novel categories are often assigned lower confidence scores (usually below 0.5), which usually means that the model is uncertain about the target and the prediction may be judged as invalid or ignored, thus affecting the detection performance of the model. Summary of the Invention

[0006] To address the problem that existing open vocabulary object detection models have poor detection capabilities for novel categories, the present invention provides an open vocabulary object detection method based on inference score adjustment.

[0007] The present invention provides an open vocabulary object detection method based on inference score adjustment, comprising:

[0008] Training phase: Obtain the candidate bounding boxes of the region of interest of the sample input image and the encoded category labels of the sample input image. Pass the candidate bounding boxes of the region of interest through the feature alignment layer to obtain mapping features. Then, perform binary matching on the mapping features and the encoded category labels to obtain an open vocabulary object detection pre-training model.

[0009] Inference stage: Use the open vocabulary object detection pre-trained model to obtain the candidate bounding boxes of the region of interest of the detection image and the encoded category labels of the detection image. The candidate bounding boxes of the region of interest are passed through the feature alignment layer to obtain the mapping features. The cosine similarity between the mapping features and the encoded category labels is then calculated to obtain the original cosine similarity score. The original cosine similarity score is then adjusted:

[0010] The visual features of all base category targets are extracted from the training sample set and clustered according to the base category. The average feature of each cluster is used as the prototype visual representation of the base category. At the same time, the encoded base category label of the base category label is obtained. The encoded base category label is mapped to the prototype visual representation of the base category to obtain a mapping matrix. The encoded novel category label of the novel category label in the detection image is then obtained, and the prototype visual representation of the novel category is obtained by combining the mapping matrix.

[0011] The calculation results of the novel categories involved in the cosine similarity calculation are replaced by the prototype visual representation of the novel category and the corresponding mapping features and the encoded category label to calculate the cosine similarity to obtain the adjusted cosine similarity score; the object detection result is obtained by combining the adjusted cosine similarity score with the remaining original cosine similarity scores.

[0012] According to the open vocabulary object detection method based on inference score adjustment of the present invention, during the training phase, an image encoder pre-trained in the CLIP multimodal language model is used to obtain sample image features from a sample input image; and then a region proposal network is used to obtain a candidate frame of a region of interest from the sample image features.

[0013] According to the open vocabulary object detection method based on inference score adjustment of the present invention, the feature alignment layer obtains mapping features based on sample image features and region of interest candidate boxes.

[0014] According to the open vocabulary object detection method based on inference score adjustment of the present invention, during the training phase, a text encoder is used to encode the category label of a sample input image to obtain an encoded category label; the category label of the sample input image is embedded in a natural language context based on a prompt template and input into the text encoder.

[0015] According to the open vocabulary object detection method based on inference score adjustment of the present invention, during the training phase, the cross entropy loss is used to supervise the matching results of the binary matching to complete the training process of the open vocabulary object detection pre-training model.

[0016] According to the open vocabulary object detection method based on inference score adjustment of the present invention, in the inference stage, an inference score adjustment module is used to adjust the original cosine similarity score;

[0017] Among them, the inference score adjustment module uses a frozen image encoder to extract GT visual features of all base category targets; the frozen image encoder is obtained based on the training process of the open vocabulary object detection pre-trained model.

[0018] According to the open vocabulary object detection method based on inference score adjustment of the present invention, the inference score adjustment module uses the K-means clustering algorithm to cluster the GT visual features of all basic category targets according to the basic category.

[0019] According to the open vocabulary object detection method based on inference score adjustment of the present invention, the inference score adjustment module uses a pre-trained text encoder to encode the basic category label to obtain the encoded basic category label; at the same time, the pre-trained text encoder is used to encode the novel category label in the detection image to obtain the encoded novel category label.

[0020] According to the open vocabulary object detection method based on inference score adjustment of the present invention, the encoded basic category labels are mapped to the prototype visual representation of the basic category, and the obtained mapping matrix is optimized and solved using the least squares method to obtain an optimized mapping matrix; the encoded novel category labels are mapped to the visual space through the optimized mapping matrix to obtain the prototype visual representation of the novel category.

[0021] According to the open vocabulary object detection method based on inference score adjustment of the present invention, the training sample set is obtained from a public channel by keyword retrieval.

[0022] The present invention's beneficial effects include: This method is used for open-vocabulary object detection in environments where the model lacks data from novel categories. It enables the classifier of the open-vocabulary object detection model to adjust its inference score for novel categories, thereby improving OVOD's detection capabilities and potential applications in real-world scenarios. This method achieves higher inference scores for novel categories, overcoming the problem of existing open-vocabulary object detection techniques where the classifier is biased towards previously seen categories. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is an example diagram where the classifier is biased towards known categories, resulting in low prediction scores for novel categories. The two red-boxed umbrellas in the figure have scores of 0.15 and 0.16 respectively.

[0024] Figure 2 This is a diagram of the training process for the open vocabulary object detection pre-trained model;

[0025] Figure 3 This is the inference flowchart of the open vocabulary object detection pre-trained model;

[0026] Figure 4 This is a comparison of the predicted scores before and after the cosine similarity score adjustment; in the figure (a), the corresponding scores of the aircraft are 0.87 and 0.65, and the scores of the umbrella are 0.56, 0.86, 0.30, 0.27 and 0.89 respectively; in the figure (b), the corresponding scores of the aircraft are 0.99 and 0.74, and the scores of the umbrella are 0.94, 0.98, 0.81, 0.82 and 0.99 respectively;

[0027] Figure 5is a flow chart of the open vocabulary object detection method based on inference score adjustment according to the present invention;

[0028] Figure 6 This is the network principle diagram of the inference score adjustment module; in the figure, GT Boxes represents the GT annotation box, and α represents the weight;

[0029] Figure 7 This is a comparison chart of the real image annotation, the detection results of the baseline "open vocabulary object detector", and the detection results of the method of the present invention in the verification experiment. DETAILED DESCRIPTION

[0030] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0031] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0032] The present invention will be further described below with reference to the accompanying drawings, but is not intended to limit the present invention.

[0033] Combine Figures 1 to 6 As shown, the present invention provides an open vocabulary object detection method based on inference score adjustment, comprising: a training phase: obtaining a region of interest candidate box of a sample input image and an encoded category label of the sample input image, subjecting the region of interest candidate box to a feature alignment layer to obtain a mapping feature, and then performing a binary matching between the mapping feature and the encoded category label to obtain an open vocabulary object detection pre-training model;

[0034] Inference stage: Use the open vocabulary object detection pre-trained model to obtain the candidate bounding boxes of the region of interest of the detection image and the encoded category labels of the detection image. The candidate bounding boxes of the region of interest are passed through the feature alignment layer to obtain the mapping features. The cosine similarity between the mapping features and the encoded category labels is then calculated to obtain the original cosine similarity score. The original cosine similarity score is then adjusted:

[0035] The visual features of all base category targets are extracted from the training sample set and clustered according to the base category. The average feature of each cluster is used as the prototype visual representation of the base category. At the same time, the encoded base category label of the base category label is obtained. The encoded base category label is mapped to the prototype visual representation of the base category to obtain a mapping matrix. The encoded novel category label of the novel category label in the detection image is then obtained, and the prototype visual representation of the novel category is obtained by combining the mapping matrix.

[0036] The calculation results of the novel categories involved in the cosine similarity calculation are replaced by the prototype visual representation of the novel category and the corresponding mapping features and the encoded category label to calculate the cosine similarity to obtain the adjusted cosine similarity score; the object detection result is obtained by combining the adjusted cosine similarity score with the remaining original cosine similarity scores.

[0037] Furthermore, during the training phase, an image encoder pre-trained in the CLIP multimodal language model is used to obtain sample image features from the sample input image; and then a region proposal network is used to obtain candidate frames of the region of interest from the sample image features.

[0038] In this embodiment, the feature alignment layer obtains mapping features based on sample image features and ROI candidate boxes.

[0039] Furthermore, during the training phase, a text encoder is used to encode the category labels of the sample input images to obtain encoded category labels; the category labels of the sample input images are embedded in the natural language context based on the prompt template and input into the text encoder.

[0040] During the training phase, cross entropy loss is used to supervise the matching results of binary matching to complete the training process of the open vocabulary object detection pre-training model.

[0041] Furthermore, in the inference stage, the inference score adjustment module is used to adjust the original cosine similarity score;

[0042] Among them, the inference score adjustment module uses a frozen image encoder to extract GT visual features of all base category targets; the frozen image encoder is obtained based on the training process of the open vocabulary object detection pre-trained model.

[0043] The inference score adjustment module uses the K-means clustering algorithm to cluster the GT visual features of all basic category targets according to the basic category.

[0044] The inference score adjustment module uses a pre-trained text encoder to encode the basic category labels to obtain the encoded basic category labels; at the same time, it uses a pre-trained text encoder to encode the novel category labels in the detection image to obtain the encoded novel category labels.

[0045] The encoded basic category labels are mapped to the prototype visual representation of the basic category, and the obtained mapping matrix is optimized and solved using the least squares method to obtain the optimized mapping matrix; the encoded novel category labels are mapped to the visual space through the optimized mapping matrix to obtain the prototype visual representation of the novel category.

[0046] As an example, the training sample set is obtained from a public channel by keyword retrieval.

[0047] This embodiment takes images or video frames in real scenes as the research objects, and the specific detection categories can be flexibly set according to the actual application needs. With the rapid development of Internet technology, image and video acquisition equipment is becoming increasingly popular. According to relevant statistics, the image and video content on the YouTube platform is currently growing at a rate of about 58 pictures and 3.6 videos per second. Users only need to obtain relevant image data from public channels through keyword retrieval around the novel target category to be detected, and they can quickly build a data set that fits the actual problem. The basic category data can be an existing public database (PASCAL VOC, Microsoft COCO, etc., see Table 1 for details). Since this embodiment does not rely on manually labeled information of novel categories during the training process, it reduces the dependence on large-scale labeling work.

[0048] Table 1 Summary of commonly used databases for object detection

[0049]

[0050] After the training database is established, an open vocabulary object detector can be trained using existing open vocabulary object detection technology. The open vocabulary object detection model training flow chart is as follows: Figure 2 As shown in the figure, in the first step, the input image (training data) is first extracted using a convolutional neural network (e.g., ResNet50) pre-trained on the CLIP (Contrastive Language Visual Pretraining Model) multimodal language model to obtain input image features. The image features are then fed into a Region Proposal Network (RPN) to extract regions of interest (or candidate boxes). The input image features and the candidate boxes generated by the RPN are then simultaneously fed into a feature alignment layer (ROI Align layer) to obtain the mapped features of the candidate boxes on the image, i.e., the candidate box features. In the second step, the category labels of the objects in the image are input as text information into the CLIP text encoder for encoding. To better model the category semantics, this process employs a prompt template approach, such as using the text format "A photo of a [CATEGORY] in the scene" to embed the category name in the natural language context. This prompt template helps activate prior knowledge in the language model and enhances the alignment of visual and textual features in a shared semantic space, thereby improving the model's understanding and transfer of category semantics. Through these two steps, we obtain the visual features of the candidate boxes and the semantic features of the class labels. These two features are then fed into a bipartite matching module to achieve a one-to-one match between the labels and the candidate regions. Finally, a cross-entropy loss is used to supervise the matching results, completing the training process of the open-vocabulary object detection model.

[0051] After the model training is completed, an open vocabulary object detection model trained on the basic category data is obtained, and then its reasoning process is performed. The reasoning process is as follows Figure 3 The inference process is consistent with the training process in most steps, with two main differences: 1) the label input is all the category labels in the dataset; 2) the cosine similarity between the candidate box features and the label embedding is calculated instead of the bipartite matching, and the prediction result is finally generated based on the similarity. However, during the inference process, the model's classifier usually tends to favor the known categories that appear in the training set, resulting in low prediction scores for novel categories, such as Figure 1 As shown in the figure, the umbrella reasoning scores are all below 0.5. This usually means that the model is uncertain about the target and the prediction may be judged as invalid or ignored, thus affecting the detection performance of the model.

[0052] The present invention studies the classifier in the open vocabulary object detection model and finds that the model obtained by training with known category data usually tends to favor the known categories that appear in the training set during the inference process, resulting in lower prediction scores for novel categories, thereby affecting the model's detection performance for novel categories. In response to the above problems and analysis, the present invention proposes an inference score adjustment (ISA) module, which uses the mapping between the visual space and language space of the basic category to construct the visual features of the novel category. Then, the visual features of the novel category are used to adjust the score of the open vocabulary object detection model classifier. Based on the inference score adjustment Figure 4 (a) and reasoning score after adjustment Figure 4 (b) The prediction score comparison chart is as follows Figure 4 shown.

[0053] In summary, the present invention effectively solves the problem in existing open vocabulary object detection methods that the classifier tends to favor basic categories and has difficulty assigning high inference scores to novel categories. The open vocabulary object detection method based on inference score adjustment is a new method for object detection in real scenarios.

[0054] The specific implementation process of the present invention is further described below with reference to the accompanying drawings.

[0055] The implementation of the method of the present invention is based on the two-stage detection framework of Faster R-CNN. In the first stage, the target candidate boxes are generated by the region candidate box network following the standard method of Faster R-CNN. The second stage is modified to an open vocabulary setting, and the class-specific localization head is replaced with a class-independent localization head. In addition, in order to achieve open vocabulary target classification, the trainable classifier weights are replaced with fixed language embeddings generated by CLIP, so that the detector can work in an open vocabulary environment. In order to more clearly explain the technical process of the present invention, the overall method will be described in conjunction with the flowchart below. The specific training steps are as follows Figure 5 As shown, the black flowchart represents the training phase, and the red flowchart represents the testing phase, which are the core of the present invention.

[0056] First, training samples are prepared according to the needs of the actual application scenario to provide a data basis for subsequent model training. Secondly, in the candidate box generation stage, the regional candidate network RPN is used to generate the region of interest of the input image, that is, the candidate box. Subsequently, the RoI Align operation is performed on the candidate box generated by RPN on the feature map to extract the corresponding regional features. At the same time, the image label is combined with the prompt template and input into the text encoder to generate the label embedding containing semantic information, which is also input into the bipartite matching module to achieve semantic alignment between the candidate region and the category label. Finally, the matching pairs obtained by bipartite matching are used to train the open vocabulary object detector. During the reasoning process, the model first calculates the similarity between the candidate box features and the embeddings of all category labels to obtain a preliminary prediction score. Subsequently, these scores will be corrected by the reasoning score adjustment module proposed in the present invention to alleviate the bias of the classifier to the basic category, thereby improving the recognition ability of novel categories.

[0057] The first step is to prepare the training samples. According to the specific target detection requirements, suitable sample data can be obtained in a variety of ways: for conventional target detection tasks (such as pedestrian detection, vehicle detection, etc.), you can use search engines to automatically collect relevant images through keyword retrieval (such as "pedestrian", "vehicle", etc.); if general target detection is performed, you can choose to use public data sets, such as PASCAL VOC and MS COCO; and for certain specific categories of detection tasks (such as face recognition), you can use dedicated data sets, such as WIDER FACE and FDDB. In order to take into account both versatility and practical feasibility, the present invention selected the MS COCO 2017 data set as the experimental basis. Among them, the "train" subset is used to train the weakly supervised target detector, and the "val" subset is used to evaluate its performance. It should be noted that the present invention only uses the images and label information of the basic categories in the training stage, and does not use any standard information and label information of any novel categories. In order to further improve the performance and robustness of the model, the present invention also introduces data enhancement and multi-scale training strategies. For data augmentation, all training images are flipped left-right and added to the training set to expand the data size and improve the model's generalization. For multi-scale training, considering the large variation in object sizes in real-world scenarios, the model randomly selects a scale from {480, 576, 688, 864, 1200} as the shortest side of the image while maintaining the image aspect ratio. Furthermore, to prevent GPU memory overflow due to excessive image size, the longest side is limited to no more than 2000 pixels. This training sample preparation and augmentation strategy ensures efficient model training while improving its adaptability to diverse scenarios, providing better support for practical applications of open-vocabulary object detection tasks.

[0058] The RPN generates candidate boxes. This module obtains a feature map from a feature extraction network (such as ResNet50). At each spatial location in the feature map, the RPN places a predefined set of anchor boxes (anchor boxes) with various scales and aspect ratios, generating nine anchor boxes per location. These anchor boxes are mapped back to the original image, representing candidate regions where objects may be located. Next, the RPN uses a small convolutional subnetwork to predict each anchor. This subnetwork typically consists of a 3×3 convolutional layer to extract local contextual features. This subnetwork then passes through two 1×1 convolutional heads, each of which outputs two results: a foreground / background binary classification score, used to determine whether the anchor contains an object; and a position regression offset, used to correct the anchor coordinates to more closely match the ground truth object box. After NMS (non-maximum suppression), the RPN outputs a set of candidate boxes with high scores and low overlap for further processing. Finally, these candidate boxes and the feature map are input into RoIAlign to obtain the candidate box features.

[0059] Bipartite matching. After RPN generates a candidate box, the corresponding candidate region feature representation can be obtained, which includes multiple region embeddings, each of which corresponds to a candidate box. For the descriptive text of the image, the CLIP text encoder is used to extract the image-level label (i.e., word) embedding to obtain a set of word embeddings. After obtaining the region embedding and word embedding, the region-word alignment process aims to find the best matching image region for each word. In the present invention, the alignment is modeled as a bipartite graph matching problem, the goal of which is to minimize the matching cost matrix. The cost matrix C ij This is obtained by computing the inner product between the region embedding and the word embedding, where each element represents the similarity between the i-th region and the j-th word. This optimal matching process is formulated as an optimization problem, aiming to minimize the total cost of all matching pairs. After bipartite graph matching, each word is assigned a corresponding candidate region embedding.

[0060] Training an open vocabulary object detector. After bipartite matching, each label in the input image is assigned a candidate embedding. Training is performed using a binary cross-entropy-based loss function that considers the matching scores of positive pairs as well as negative pairs from other classes. The training settings are as follows: total number of iterations: 90,000; learning rate strategy: during the warm-up phase, the learning rate is linearly increased from 0 to 0.002 over the first 1,000 iterations. The model is trained for 90,000 iterations using the SGD optimizer with a batch size of 8. The learning rate is reduced by a factor of 10 at iterations 60,000 and 80,000 to ensure gradual convergence and avoid overfitting.

[0061] The open vocabulary object detection network trained by the above steps can realize the object detection function without the annotation information and label information of novel categories. The present invention mainly plays a role in the inference stage of the open vocabulary object detection method, aiming to adjust the prediction score of the model classifier. Specifically, after completing the model training, an open vocabulary object detection model trained based on the basic category data is obtained, and then enters the inference stage. The overall process is as follows: Figure 5 The red flow chart shows this. The inference process of this invention is consistent with the training phase in most steps, with two key differences: 1) The input labels are the set of labels for all categories in the dataset; 2) The matching method between the candidate box features and the category label embeddings is replaced by cosine similarity calculation instead of the original bipartite graph matching, and the final prediction result is generated based on the similarity score. The following will elaborate on the implementation details of each part of the testing phase:

[0062] Cosine similarity calculation. After the candidate boxes are generated by the RPN module, the model can extract the visual representation of these areas on the feature map. At the same time, all target category labels are encoded using a pre-trained text encoder (such as the text branch of CLIP) to obtain the corresponding semantic vector representation. Next, the model evaluates their correlation in the semantic space by calculating the cosine similarity between the candidate box features and the category semantic vectors. Cosine similarity is used to measure the proximity of two vectors in a high-dimensional semantic space. The higher the similarity, the more likely the candidate area is to match the corresponding category semantics, thereby achieving category discrimination and recognition.

[0063] Inference score adjustment. During the cosine similarity calculation process, the model's classifier is often biased towards the seen categories, resulting in relatively low prediction scores for novel categories, which affects the model's detection performance on unseen categories. To solve this problem, the present invention proposes an inference score adjustment method to alleviate category bias and improve the model's ability to recognize novel categories. Figure 6 As shown, first, the present invention uses a frozen visual encoder to extract the Ground Truth (GT) visual features of all basic categories from the training set. The visual encoder is obtained by following the open vocabulary object detection training process described in the present invention. For the GT region embedding of each basic category, the present invention uses the K-means clustering algorithm to divide it into Z clusters, and calculates the average features of each cluster, so as to generate a prototype visual representation for each basic category, that is, select a typical visual sample as the representative of the category. Then, use the pre-trained text encoder to encode all basic category labels to obtain their corresponding semantic embedding representations. Based on the above processing, the visual prototype embeddings of the basic categories (corresponding to Figure 6 Basic category prototype features in the basic category text semantic embedding (corresponding to Figure 6 The basic category label features in ) and the novel category text semantic embedding (corresponding to Figure 6 The present invention aims to learn a mapping matrix W for mapping the text embedding of the base category to its corresponding visual prototype embedding. To this end, the present invention uses the least squares method to optimize and solve W. After obtaining the mapping matrix W, the text embedding of the novel category can be further mapped to the visual space through W, thereby obtaining the prototype visual embedding of the novel category (corresponding to Figure 6 Therefore, for the candidate box features that have passed RPN and RoI Align, the prototype visual embedding of the novel category can be used to recalculate the classifier's score for the novel category. Finally, the score of the novel category will replace the classifier's original score for the novel category. Please note that the present invention only re-scores the novel category and does not change the score of the base category.

[0064] Experiments have proven the effectiveness of the "open vocabulary object detection method based on inference score adjustment" of the present invention. Table 2 shows the comparative data of the experimental results:

[0065] Table 2 Experimental results comparison data

[0066]

[0067] Novel, Base, and Overall represent metrics for novel, basic, and all categories, respectively, and are used to evaluate detection accuracy in test samples, reflecting the model's detection performance for different target categories (novel and basic). 50 represents the IoU (Intersection over Union) threshold of 0.5. mAP stands for Mean Average Precision (MAP). Novel mAP is the primary metric for evaluating the performance of open vocabulary object detection models. The comparative data shows that the "open vocabulary object detector + inference score adjustment" framework proposed in this paper significantly improves the Novel mAP compared to the "open vocabulary object detector" approach. Figure 7 The following figure shows experimental results. All detected objects are novel categories. The first row shows the ground truth annotations, the second row shows the detection results of the baseline "open vocabulary object detector", and the third row shows the detection results of the "open vocabulary object detection method based on inference score adjustment" proposed in this invention. As can be seen, the proposed method has a clear advantage: the detection scores of objects are much higher than those of the detector without inference score adjustment.

[0068] Although the present invention is described herein with reference to specific embodiments, it should be understood that these embodiments are merely illustrative of the principles and applications of the invention. It should be understood that many modifications may be made to the illustrative embodiments, and that other arrangements may be devised, without departing from the spirit and scope of the invention as defined by the appended claims. It should be understood that the various dependent claims and features described herein may be combined in ways other than those described in the original claims. It should also be understood that features described in conjunction with individual embodiments may be used in conjunction with other described embodiments.

Claims

1. An open vocabulary object detection method based on inference score adjustment, characterized by include: Training phase: Obtain the candidate bounding boxes of the region of interest of the sample input image and the encoded category labels of the sample input image. Pass the candidate bounding boxes of the region of interest through the feature alignment layer to obtain mapping features. Then, perform binary matching on the mapping features and the encoded category labels to obtain an open vocabulary object detection pre-training model. Inference stage: Use the open vocabulary object detection pre-trained model to obtain the candidate bounding boxes of the region of interest of the detection image and the encoded category labels of the detection image. The candidate bounding boxes of the region of interest are passed through the feature alignment layer to obtain mapping features. The cosine similarity between the mapping features and the encoded category labels is then calculated to obtain the original cosine similarity score. Then adjust the original cosine similarity score: The visual features of all basic category targets are extracted from the training sample set, and clustered according to the basic category. The average feature of each cluster is used as the prototype visual representation of the basic category. At the same time, the encoded basic category label of the basic category label is obtained; Map the encoded basic category label to the prototype visual representation of the basic category to obtain a mapping matrix; Then, the encoded novel category labels of the novel category labels in the detection image are obtained, and the prototype visual representation of the novel category is obtained by combining the mapping matrix; For the calculation results of novel categories involved in the cosine similarity calculation, the prototype visual representation of the novel category is used to replace the corresponding mapping features and the encoded category label to calculate the cosine similarity, and the adjusted cosine similarity score is obtained; The object detection result is obtained by combining the adjusted cosine similarity score and the rest of the original cosine similarity scores.

2. The open vocabulary object detection method based on inference score adjustment according to claim 1, characterized in that In the training phase, the image encoder pre-trained in the CLIP multimodal language model is used to obtain sample image features from the sample input image; Then, the region proposal network is used to obtain the candidate frame of the region of interest from the sample image features.

3. The open vocabulary object detection method based on inference score adjustment according to claim 2, characterized in that The feature alignment layer obtains mapping features based on sample image features and region of interest candidate boxes.

4. The open vocabulary object detection method based on inference score adjustment according to claim 1, characterized in that During the training phase, a text encoder is used to encode the category labels of the sample input images to obtain the encoded category labels; the category labels of the sample input images are embedded in the natural language context based on the prompt template and input into the text encoder.

5. The open vocabulary object detection method based on inference score adjustment according to claim 1, characterized in that During the training phase, cross entropy loss is used to supervise the matching results of binary matching to complete the training process of the open vocabulary object detection pre-training model.

6. The open vocabulary object detection method based on inference score adjustment according to claim 1, characterized in that In the inference stage, the inference score adjustment module is used to adjust the original cosine similarity score; Among them, the inference score adjustment module uses a frozen image encoder to extract GT visual features of all base category targets; the frozen image encoder is obtained based on the training process of the open vocabulary object detection pre-trained model.

7. The open vocabulary object detection method based on inference score adjustment according to claim 6, characterized in that The inference score adjustment module uses the K-means clustering algorithm to cluster the GT visual features of all basic category targets according to the basic category.

8. The open vocabulary object detection method based on inference score adjustment according to claim 7, characterized in that The inference score adjustment module uses a pre-trained text encoder to encode the basic category labels to obtain the encoded basic category labels; at the same time, it uses a pre-trained text encoder to encode the novel category labels in the detection image to obtain the encoded novel category labels.

9. The open vocabulary object detection method based on inference score adjustment according to claim 8, characterized in that The encoded basic category labels are mapped to the prototype visual representation of the basic category, and the obtained mapping matrix is optimized and solved using the least squares method to obtain the optimized mapping matrix; the encoded novel category labels are mapped to the visual space through the optimized mapping matrix to obtain the prototype visual representation of the novel category.

10. The open vocabulary object detection method based on inference score adjustment according to claim 1, characterized in that The training sample set is obtained from public channels by keyword retrieval.

Images