Open vocabulary object detection method based on SAM candidate box generation and candidate region-word clustering

Abstract

The invention discloses an open vocabulary object detection method based on SAM candidate box generation and candidate region-word clustering, and belongs to the technical field of object detection in machine vision. The invention aims to solve the problems of insufficient candidate frames of novel categories and missing matching caused by binary matching in the existing open vocabulary object detection technology. Comprising the following steps: acquiring an SAM-module-based candidate frame and a region-of-interest candidate frame of an input image, splicing the candidate frames, and obtaining candidate frame mapping features through a feature alignment layer; obtaining coding features of the label of the input image, and carrying out binary matching on the coding features and the candidate box mapping features; adding the coding feature and the candidate frame mapping feature corresponding to each pair of bisection matching results by adopting a candidate region-word clustering module to obtain the candidate frame mapping feature of the configured tag; and calculating the cosine similarity between the other candidate frame mapping features and the candidate frame mapping features configured with the labels, and matching label categories for the other candidate frame mapping features to realize one-to-many matching. The method is used for open vocabulary object detection.

Description

Technical Field

[0001] The present invention relates to an open vocabulary object detection method based on SAM candidate frame generation and candidate region-word clustering, and belongs to the technical field of object detection in machine vision. Background Art

[0002] Object detection is one of the core research directions in computer vision, aiming to identify objects of interest from static images or dynamic videos and accurately mark their spatial locations in the image. This location information is usually represented by a bounding box, which is a rectangular box defined by the coordinates of the upper left and lower right corners of the image to cover the target area. In addition, the detection process also needs to determine the category to which the target belongs. For example, in traffic scenarios, the categories may include vehicles, pedestrians, or traffic lights. With the continuous evolution of technology, object detection has been able to achieve efficient and accurate identification and positioning, and is widely used in many fields in the real world. For example, in security monitoring, it can automatically identify abnormal behavior or suspicious objects, assist in product identification and inventory management in smart cities, and be used in autonomous driving to perceive key elements such as pedestrians, vehicles, and traffic signs in the road environment.

[0003] Most current mainstream object detection methods rely on a closed set of categories for model training, collectively referred to as closed set object detection (CSOD). Within the CSOD paradigm, model training is based on a predefined, limited set of categories, each of which requires sufficient, manually annotated examples with precise bounding boxes and class labels. This approach achieves excellent detection performance on standard datasets such as PASCAL VOC and MS COCO. However, its fundamental assumption is that the categories used during training and testing are identical. In other words, the model can only detect objects from categories it has "seen" during training. However, the real world is open and dynamic, with a large number of object categories and a long-tail distribution. New object types are constantly emerging, making it nearly impossible to pre-emptively enumerate all possible detection categories. Traditional CSOD methods lack generalization ability when faced with these "unseen" (novel) categories, often misclassifying them as background or other similar categories, severely limiting their applicability in real-world open environments. This closed category assumption is particularly weak in application scenarios such as autonomous driving, telemedicine, and industrial inspection that rely heavily on the recognition of diverse objects.

[0004] To address the challenges faced by traditional object detection methods under the closed-category assumption, such as insufficient generalization, high data annotation costs, and difficulty adapting to long-tail and novel categories in real-world environments, researchers have recently proposed a new research paradigm: open-vocabulary object detection (OVOD). The core goal of open-vocabulary object detection is to enable the detection model to accurately identify and localize "new categories" during testing, even when the model has not explicitly seen certain categories during training (i.e., trained on "base categories"). Specifically, OVOD methods map category names into a shared semantic space by introducing linguistic representations of categories (such as text embeddings of category names) or leveraging semantic knowledge provided by large-scale pre-trained language models (such as BERT, GPT, and CLIP). This allows the model to recognize unseen categories based on semantic similarity.

[0005] However, most existing open vocabulary object detection methods typically rely on Region Proposal Networks (RPNs) to extract regions of interest on the underlying categories, and then use bipartite matching to align the candidate regions with words. Although these methods have achieved good results on open vocabulary object detection datasets, these models still face the following challenges:

[0006] (1) RPN generates high-confidence regions of interest for basic categories, but generates low-confidence candidate boxes for novel categories. This results in a lack of candidate boxes for novel categories in the candidate region-word matching process. (2) Figure 1 As shown in Figure 2, when multiple instances of the same category appear in the same image, using bipartite matching will result in missing matches. Summary of the Invention

[0007] Aiming at the problems of insufficient candidate boxes of novel categories and missing matches caused by bipartite matching in existing open vocabulary object detection technologies, the present invention provides an open vocabulary object detection method based on SAM candidate box generation and candidate region-word clustering.

[0008] The present invention provides an open vocabulary object detection method based on SAM candidate box generation and candidate region-word clustering, comprising:

[0009] The SAM module is used to obtain candidate frames of the input image. An image encoder is used to obtain image features of the input image. A region candidate network is then used to obtain candidate frames of the region of interest based on the image features. The candidate frames based on the SAM module and the region of interest are concatenated to obtain a candidate frame set. The candidate frame mapping features are then obtained through a feature alignment layer.

[0010] Use a text encoder to obtain the encoded features of the input image label and perform binary matching with the candidate box mapping features;

[0011] The candidate region-word clustering module is used to add the encoding features and candidate box mapping features corresponding to each pair of binary matching results to obtain the candidate box mapping features of the configured label; the cosine similarity of the remaining candidate box mapping features and the candidate box mapping features of the configured label is calculated, and the remaining candidate box mapping features with a cosine similarity greater than a set threshold are matched to the corresponding label category to achieve one-to-many matching between labels and candidate box mapping features; during the training process, the network parameters of the image encoder are adjusted based on the binary cross entropy loss function.

[0012] According to the open vocabulary object detection method based on SAM candidate box generation and candidate region-word clustering of the present invention, the input image is configured with multiple evenly distributed grid points, and the grid points are used as prompts of the SAM module to obtain candidate boxes based on the SAM module.

[0013] According to the open vocabulary object detection method based on SAM candidate box generation and candidate region-word clustering of the present invention, the input image is configured with 25 evenly distributed grid points.

[0014] According to the open vocabulary object detection method based on SAM candidate box generation and candidate region-word clustering of the present invention, the label of the input image is embedded in the natural language context based on the prompt template and input into the text encoder.

[0015] According to the open vocabulary object detection method based on SAM candidate box generation and candidate region-word clustering of the present invention, encoding features and candidate box mapping features are weighted fused with an adjustable fusion ratio to obtain candidate box mapping features with configured labels.

[0016] According to the open vocabulary object detection method based on SAM candidate box generation and candidate region-word clustering of the present invention, the image encoder is a convolutional neural network pre-trained in the CLIP multimodal language model.

[0017] According to the open vocabulary object detection method based on SAM candidate box generation and candidate region-word clustering of the present invention, the image encoder is implemented based on the ResNet50 network.

[0018] According to the open vocabulary object detection method based on SAM candidate box generation and candidate region-word clustering of the present invention, sample data in the image encoder training process is obtained from public channels through keyword retrieval.

[0019] According to the open vocabulary object detection method based on SAM candidate box generation and candidate region-word clustering of the present invention, the candidate box set is a candidate box covering basic categories and novel categories.

[0020] According to the open vocabulary object detection method based on SAM candidate box generation and candidate region-word clustering of the present invention, the set threshold of cosine similarity is the number of clusters.

[0021] The present invention has the following beneficial effects: The method is applicable to object detection in the field of machine vision, particularly when the model lacks data on novel categories. This method enables open-vocabulary object detection models to generate more comprehensive candidate boxes covering both novel and basic categories, and to mine more matching examples for model training, thus promoting the application of OVOD in object detection in real-world scenarios.

[0022] The method of the present invention is based on open vocabulary object detection based on SAM (Segment Everything Model) candidate box generation and candidate region-word clustering, which solves the problems of insufficient novel category candidate boxes and missing matches caused by bipartite matching in object detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is an example of a missing match caused by bipartite matching; n is the nth detection category;

[0024] Figure 2 This is a flow chart of the open vocabulary object detection model training in the open vocabulary object detection method based on SAM candidate box generation and candidate region-word clustering described in the present invention;

[0025] Figure 3 This is a comparison chart of the original input image, the input image with 25 evenly distributed grid points, and the candidate boxes generated based on the SAM module;

[0026] Figure 4 It is a comparison diagram of the results of bipartite matching and candidate region-word cluster matching in the method of the present invention;

[0027] Figure 5 is a flow chart of the method of the present invention;

[0028] Figure 6 This is a schematic diagram of the principle of SAM candidate box generation and candidate region-word clustering method; u and r p Mapping features for candidate boxes, w u and w p is the corresponding coding feature; C represents splicing;

[0029] Figure 7 This is a comparison chart of the real image annotation, the detection results of the baseline "open vocabulary object detector", and the detection results of the method of the present invention in the verification experiment. DETAILED DESCRIPTION

[0030] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0031] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0032] The present invention will be further described below with reference to the accompanying drawings, but is not intended to limit the present invention.

[0033] Combine Figures 1 to 6 As shown, the present invention provides an open vocabulary object detection method based on SAM candidate box generation and candidate region-word clustering, comprising:

[0034] The SAM module is used to obtain candidate frames of the input image. An image encoder is used to obtain image features of the input image. A region candidate network is then used to obtain candidate frames of the region of interest based on the image features. The candidate frames based on the SAM module and the region of interest are concatenated to obtain a candidate frame set. The candidate frame mapping features are then obtained through a feature alignment layer.

[0035] Use a text encoder to obtain the encoded features of the input image label and perform binary matching with the candidate box mapping features;

[0036] The candidate region-word clustering module is used to add the encoding features and candidate box mapping features corresponding to each pair of binary matching results to obtain the candidate box mapping features of the configured label; the cosine similarity of the remaining candidate box mapping features and the candidate box mapping features of the configured label is calculated, and the remaining candidate box mapping features with a cosine similarity greater than a set threshold are matched to the corresponding label category to achieve one-to-many matching between labels and candidate box mapping features; during the training process, the network parameters of the image encoder are adjusted based on the binary cross entropy loss function.

[0037] Furthermore, the input image is configured with a plurality of evenly distributed grid points, and the grid points are used as prompts of the SAM module to obtain candidate boxes based on the SAM module.

[0038] As an example, the input image is configured with 25 evenly distributed grid points.

[0039] In this embodiment, the label of the input image is embedded in a natural language context based on a prompt template and input to a text encoder.

[0040] The encoding features and the candidate box mapping features are weighted fused with an adjustable fusion ratio to obtain the candidate box mapping features of the configuration label.

[0041] As an example, the image encoder is a convolutional neural network pre-trained on the CLIP multimodal language model.

[0042] As an example, the image encoder is implemented based on the ResNet50 network.

[0043] The sample data in the image encoder training process is obtained from public channels through keyword retrieval.

[0044] This embodiment can use images or video frames in real scenes as research objects, and the specific detection categories can be flexibly set according to actual application requirements. With the rapid development of Internet technology, image and video acquisition equipment is becoming increasingly popular. According to relevant statistics, the image and video content on the YouTube platform is currently growing at a rate of about 58 pictures and 3.6 videos per second. Users only need to obtain relevant image data from public channels through keyword retrieval around the novel target category to be detected, and they can quickly build a data set that fits the actual problem. The basic category data can be an existing public database (PASCAL VOC, Microsoft COCO, etc., see Table 1 for details). Since this embodiment does not rely on manually labeled information of novel categories during the training process, it reduces the dependence on large-scale labeling work.

[0045] Table 1 Summary of commonly used databases for object detection

[0046]

[0047] After the training database is established, an open vocabulary object detector can be trained using existing open vocabulary object detection technology. The open vocabulary object detection model training flow chart is as follows: Figure 2As shown in the figure, in the first step, the input image (training data) is first extracted using a convolutional neural network (e.g., ResNet50) pre-trained on the CLIP (Contrastive Language Visual Pretraining Model) multimodal language model to obtain input image features. The image features are then fed into a Region Proposal Network (RPN) to extract regions of interest (or candidate boxes). The input image features and the candidate boxes generated by the RPN are then simultaneously fed into a feature alignment layer (ROI Align layer) to obtain the mapped features of the candidate boxes on the image, i.e., the candidate box features. In the second step, the category labels of the objects in the image are input as text information into the CLIP text encoder for encoding. To better model the category semantics, this process employs a prompt template approach, such as using the text format "A photo of a [CATEGORY] in the scene" to embed the category name in the natural language context. This prompt template helps activate prior knowledge in the language model and enhances the alignment of visual and textual features in a shared semantic space, thereby improving the model's understanding and transfer of category semantics. Through the above two steps, the visual features of the candidate box and the semantic features of the category label can be obtained. Subsequently, these two types of features are input into the binary matching module to achieve a one-to-one correspondence between the label and the candidate region. Finally, the cross-entropy loss is used to supervise the matching results to complete the training process of the open vocabulary object detection model. However, when generating candidate boxes, RPN usually only has a high confidence in the basic categories, which makes it difficult to provide a sufficient number and quality of candidate regions when facing novel categories, resulting in insufficient samples in the subsequent region and word embedding alignment stage. In addition, the adopted binary matching strategy only assigns one candidate region to each word embedding at a time during the assignment process, which limits the flexibility of many-to-many matching, resulting in the lack of valid matching pairs, thereby affecting the detection performance.

[0048] This embodiment studies the candidate box generation and candidate region-word matching module in the open vocabulary object detection model, and finds that the RPN can only access the training data of the basic category and calculate the regression loss of the bounding box, thereby generating high-confidence regions of interest for the basic category and low-confidence candidate boxes for the novel category. The high-confidence region of interest will be used as the subsequent candidate region and word alignment stage, while the low-confidence region cannot be assigned a corresponding word embedding. In addition, when multiple instances of the same category appear in an image, the use of the binary matching algorithm will result in missing matches. Because a word embedding will only be assigned one region of interest by binary matching, this results in other candidate boxes being unable to be assigned word embeddings, even though the candidate box belongs to a certain category, thereby affecting the detection performance.

[0049] In response to the above problems, this implementation method performs open vocabulary object detection based on SAM candidate box generation and candidate region-word clustering. In response to the first problem and analysis, in order to ensure that the model has a sufficient number of candidate boxes in the candidate region-word matching stage, a SAM-based candidate box generation module is proposed to obtain a candidate box set. It uses an unsupervised positioning method to generate candidate boxes covering basic categories and novel categories for the input image. 25 grid points are evenly distributed in the original input image, and these points are used as prompts for SAM to generate different target candidate boxes. Figure 3 Figures (a), (b), and (c) show comparisons under three different scenarios. Then, RoIAlign is performed on the SAM-generated candidate boxes and the feature maps to obtain the SAM-generated candidate box features. By integrating the SAM-generated candidate boxes, this implementation effectively enhances the diversity of target candidate boxes and overcomes the limitations of RPN in detecting new categories.

[0050] In this embodiment, the threshold value of the cosine similarity is set to the number of clusters.

[0051] In response to the second question and analysis, the "candidate region-word clustering method" of this embodiment can find similar candidate boxes in the embedding space for the matched regions from the candidate region-word pairs obtained by bipartite matching, and assign the same pseudo-label to them. Assuming that a given input image contains multiple identical "umbrellas" as the target to be detected, then after bipartite matching, only one candidate box of "umbrella" is assigned the corresponding word embedding. In order to mine more matching samples, this embodiment uses cosine similarity to measure the similarity between the remaining region embeddings and the currently matched region embeddings. That is, for the "umbrella" word embedding, the cosine similarity between the candidate box corresponding to the current "umbrella" and the remaining candidate boxes is calculated, and the "umbrella" label is assigned to the top k most similar candidate region embeddings. Then, for the "umbrella" word embedding, k+1 matching pairs can be obtained, thereby using the binary cross entropy classification loss to learn the pairing relationship between regions and words. The previous model assigned 1 matching pair to the "umbrella" word embedding, while the present invention has k+1 matching pairs for training, Figure 4 Comparison diagram of bipartite matching (a) and candidate region-word cluster matching (b).

[0052] In summary, the present invention solves the problems that the existing open vocabulary object detection method relies on the RPN network and cannot generate high-confidence candidate boxes for novel categories, and the bipartite matching algorithm leads to the lack of matching between candidate regions and words. The open vocabulary object detection method based on SAM candidate box generation and candidate region-word clustering is a new method for object detection in real scenarios.

[0053] The specific implementation process of the present invention is further described below with reference to the accompanying drawings.

[0054] The method implemented in this paper is built on the two-stage detection framework of Faster R-CNN. In the first stage, the target proposal boxes are generated by the region proposal network following the standard method of Faster R-CNN. The second stage is modified to an open vocabulary setting, replacing the class-specific localization head with a class-independent localization head. In addition, to achieve open vocabulary object classification, the trainable classifier weights are replaced with fixed language embeddings generated by CLIP, so that the detector can work in an open vocabulary environment. In order to explain the technical process more clearly, the following will be combined with Figure 5 To explain. First, prepare training samples according to the needs of the actual application scenario to provide a data basis for subsequent model training. Secondly, in the candidate box generation stage, an unsupervised method based on SAM is adopted to generate a variety of candidate boxes covering basic categories and novel categories. Subsequently, the RoI Align operation is performed on the candidate box generated by SAM on the feature map, the corresponding region features are extracted, and they are fused with the candidate box features generated by the RPN module. The fused candidate box features are input into the candidate region-category clustering module. At the same time, the image label is combined with the prompt template and input into the text encoder to generate a label embedding containing semantic information, which is also input into the candidate region-word clustering module to achieve semantic alignment between the candidate region and the category label. Finally, the matching pairs obtained by the candidate region-word clustering are used to train the open vocabulary object detector. The following will elaborate on the implementation details of each part:

[0055] The first step is to prepare training samples. According to the specific target detection requirements, suitable sample data can be obtained in a variety of ways: for conventional target detection tasks (such as pedestrian detection, vehicle detection, etc.), you can use search engines to automatically collect relevant images through keyword retrieval (such as "pedestrian", "vehicle", etc.); if general target detection is performed, you can choose to use public datasets, such as PASCAL VOC and MS COCO; and for certain specific categories of detection tasks (such as face recognition), you can use dedicated datasets, such as WIDER FACE and FDDB. In order to take into account both versatility and practical feasibility, the present invention can use the MS COCO 2017 dataset as the experimental basis. Among them, the "train" subset is used to train the weakly supervised target detector, while the "val" subset is used to evaluate its performance. It should be noted that only the images and label information of the basic categories are used in the training stage, and no standard information and label information of any novel categories are used. In order to further improve the performance and robustness of the model, data enhancement and multi-scale training strategies are also introduced. For data augmentation, all training images are flipped left-right and added to the training set to expand the data size and improve the model's generalization. For multi-scale training, considering the large variation in object sizes in real-world scenarios, the model randomly selects a scale from {480, 576, 688, 864, 1200} as the shortest side of the image while maintaining the image aspect ratio. Furthermore, to prevent GPU memory overflow due to excessive image size, the longest side is limited to no more than 2000 pixels. This training sample preparation and augmentation strategy ensures efficient model training while improving its adaptability to diverse scenarios, providing better support for practical applications of open-vocabulary object detection tasks.

[0056] SAM-based proposal box generation. Existing open vocabulary object detectors rely on the RPN to generate high-confidence regions of interest for base categories, but generate low-confidence proposal boxes for novel categories, resulting in a lack of proposal boxes for novel categories during the region-word matching process. The specific reason is that the RPN can only access the training set of the base categories and calculate the bounding box regression loss. To ensure that the model has a sufficient number of proposal boxes during the proposal region-word matching stage, a SAM-based proposal box generation method is adopted. 25 grid points in the input image are uniformly sampled and used as cues for the SAM to generate diverse proposal boxes. The proposal boxes generated by the SAM are then merged with the proposal boxes from the region proposal network (RPN) and fed into RoIAlign to obtain proposal box features. The proposal box features are then embedded with the label. By integrating the proposal boxes generated by the SAM, the diversity of the proposal boxes is increased, alleviating the limitations of the RPN in generating proposal boxes for novel categories.

[0057] Candidate region-word clustering. Through the above steps, we can obtain the candidate box features and label features. The candidate region-word clustering module aims to determine multiple matches between image regions and words, that is, to find the corresponding region embedding for each word (label feature). Figure 6 As shown, bipartite matching will assign the best candidate regions to labels such as "umbrella" and "person". Once these matching pairs are obtained, traditional methods learn the alignment relationship between image regions and words by minimizing the classification loss. However, as analyzed above, when there are multiple instances of the same category in the image, most of the relevant candidate regions will be discarded during the matching process, resulting in an insufficient number of samples for the new category, thereby limiting the open vocabulary object detection model's ability to recognize new categories. Therefore, these models find it difficult to learn the alignment relationship between image regions and words. In order to solve this problem, the present invention proposes a region-word clustering method. As Figure 6 As shown in the figure, after completing the initial bipartite matching, the labels "umbrella" and "person" each obtain their corresponding image region embeddings. To discover more potential candidate regions corresponding to these labels, a similarity calculation mechanism is introduced. Specifically, for the region embedding corresponding to "umbrella," its similarity score with all other candidate regions is calculated. Subsequently, several regions that are most similar to it are selected and assigned the same category as the original label. This expands the original situation where each label can only be paired with one candidate region, thereby obtaining multiple high-quality matching pairs, facilitating subsequent training using a classification loss function and further improving the matching between regions and words. However, standard similarity calculation methods, such as cosine similarity, may not accurately capture the semantic relationship between region embeddings, as different instances of the same category can have significantly different appearances. To overcome this problem, "word-guided similarity calculation" is further introduced in the region-word clustering module, making the similarity assessment label-aware. Specifically, given the word embedding of a target category and its corresponding candidate region embedding, a prototype region feature is constructed. This feature is formed by a weighted fusion of word embeddings and region embeddings, with an adjustable fusion ratio to balance the contributions of visual and semantic features. Next, the similarity scores between this prototype region feature and other candidate regions are recalculated. Based on these scores, a group of regions with similarities above a preset threshold are selected as clustering results. These regions share the same category label, greatly increasing the number of training samples corresponding to each label and effectively alleviating the problem of scarce samples for new categories. Ultimately, each label will correspond to multiple candidate regions, which are collectively used for supervised training of region-word alignment.

[0058] Training an open vocabulary object detector. After candidate region-word clustering, each label in the input image is given multiple candidate embeddings. This paper uses a binary cross-entropy-based loss function for training, which considers the matching scores of positive pairs as well as negative pairs from other categories. The specific training settings are as follows: total number of iterations: 90,000; learning rate strategy: during the warm-up phase, the learning rate is linearly increased from 0 to 0.002 over the first 1,000 iterations. The model is trained for 90,000 iterations using the SGD optimizer with a batch size of 8, and the learning rate is reduced by a factor of 10 at 60,000 and 80,000 iterations to ensure gradual convergence and avoid overfitting.

[0059] The open-vocabulary object detection network trained through the above steps can perform object detection without the need for annotation and label information for novel categories. Experiments demonstrate that the "open-vocabulary object detection method based on SAM candidate box generation and candidate region-word clustering" proposed in this invention achieves accurate positioning accuracy. Table 2 presents comparative experimental results. Novel, Base, and Overall represent metrics for measuring novel, basic, and all categories, respectively, used to evaluate detection accuracy in test samples and reflect the model's detection performance for different target categories (novel and basic). 50 represents the IoU (Intersection over Union) threshold of 0.5. mAP stands for Mean Average Precision (MAP). Novel mAP is a key metric for evaluating the performance of open-vocabulary object detection models. The comparative data demonstrates that the proposed "open-vocabulary object detector + SAM-based candidate box generation + candidate region-word clustering" framework achieves a significant improvement in Novel mAP over the "open-vocabulary object detector." The "candidate region-word clustering" approach also significantly improves detection results compared to the approach without "candidate region-word clustering." Figure 7 The following figure shows experimental results. All detected objects are novel categories. The first row shows the ground truth annotations, the second row shows the detection results of the baseline "open vocabulary object detector", and the third row shows the detection results of the proposed method. As can be seen, the proposed method has a clear advantage: its detection results cover more novel objects.

[0060] Table 2 Experimental results comparison data

[0061]

[0062] Although the present invention is described herein with reference to specific embodiments, it should be understood that these embodiments are merely illustrative of the principles and applications of the invention. It should be understood that many modifications may be made to the illustrative embodiments, and that other arrangements may be devised, without departing from the spirit and scope of the invention as defined by the appended claims. It should be understood that the various dependent claims and features described herein may be combined in ways other than those described in the original claims. It should also be understood that features described in conjunction with individual embodiments may be used in conjunction with other described embodiments.

Claims

1. An open vocabulary object detection method based on SAM candidate box generation and candidate region-word clustering, characterized by include: Use the SAM module to obtain the candidate box of the input image; At the same time, an image encoder is used to obtain the image features of the input image, and then a region candidate network is used to obtain a candidate frame of the region of interest based on the image features; the candidate frame based on the SAM module and the candidate frame of the region of interest are spliced to obtain a candidate frame set; and then The candidate box mapping features are obtained through the feature alignment layer; Use a text encoder to obtain the encoded features of the input image label and perform binary matching with the candidate box mapping features; The candidate region-word clustering module is used to add the encoding features and candidate box mapping features corresponding to each pair of binary matching results to obtain the candidate box mapping features of the configuration label; The cosine similarity between the remaining candidate box mapping features and the candidate box mapping features of the configured label is calculated, and the remaining candidate box mapping features whose cosine similarity is greater than the set threshold are matched to the corresponding label category to achieve one-to-many matching between labels and candidate box mapping features; during the training process, the network parameters of the image encoder are adjusted based on the binary cross entropy loss function.

2. The open vocabulary object detection method based on SAM candidate box generation and candidate region-word clustering according to claim 1, characterized in that: The input image is configured with a plurality of evenly distributed grid points, and the grid points are used as prompts of the SAM module to obtain candidate boxes based on the SAM module.

3. The open vocabulary object detection method based on SAM candidate box generation and candidate region-word clustering according to claim 2, characterized in that: The input image is configured with 25 evenly distributed grid points.

4. The open vocabulary object detection method based on SAM candidate box generation and candidate region-word clustering according to claim 1, characterized in that: The label of the input image is embedded in the natural language context based on the prompt template and input to the text encoder.

5. The open vocabulary object detection method based on SAM candidate box generation and candidate region-word clustering according to claim 1, characterized in that: The encoding features and the candidate box mapping features are weighted fused with an adjustable fusion ratio to obtain the candidate box mapping features of the configuration label.

6. The open vocabulary object detection method based on SAM candidate box generation and candidate region-word clustering according to claim 1, characterized in that: The image encoder is a convolutional neural network pre-trained on the CLIP multimodal language model.

7. The open vocabulary object detection method based on SAM candidate box generation and candidate region-word clustering according to claim 6, characterized in that: The image encoder is implemented based on the ResNet50 network.

8. The open vocabulary object detection method based on SAM candidate box generation and candidate region-word clustering according to claim 1, characterized in that: The sample data in the image encoder training process is obtained from public channels through keyword retrieval.

9. The open vocabulary object detection method based on SAM candidate box generation and candidate region-word clustering according to claim 1, characterized in that: The candidate box set includes candidate boxes covering basic categories and novel categories.

10. The open vocabulary object detection method based on SAM candidate box generation and candidate region-word clustering according to claim 1, characterized in that: The threshold of cosine similarity is set to the number of clusters.

Images