Open vocabulary object detection method based on prototype library feature mining, electronic device and storage medium

Abstract

This invention discloses an open vocabulary target detection method, electronic device, and storage medium based on prototype library feature mining, which improves the detection performance of new target categories. It includes a category-generalized Region Proposal Network (CG-RPN) and a category-specific knowledge distillation method (CSKD), and comprises: 1. Preprocessing the image to be detected, and then feeding the preprocessed image and its corresponding image-level label into a Visual Language Model (VLM); 2. The VLM performs feature extraction and feature fusion on the image to be detected, and feeds the resulting feature map into the CG-RPN; the CG-RPN selects sample boxes based on features, assigns labels to the sample boxes based on classification scores, and feeds them into CSKD; 3. CSKD processes the sample boxes into two sets of features using the RoI head and the VLM image encoder respectively, and then uses the category-specific knowledge distillation method to transfer rich visual semantic knowledge from the VLM; 4. The classification head and regression head obtain the final prediction result based on the features generated by the RoI head, the ground truth calculates the relevant loss, and backpropagates it to the VLM to adjust the parameters.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, and specifically refers to an open vocabulary object detection method, electronic device, and storage medium based on prototype library feature mining. Background Technology

[0002] In recent years, with the improvement of computer performance and the development of big data, visual information data has increased rapidly. Multimedia data, including still images, moving images, video files, and audio files, are spreading quickly on various social media platforms. Object detection, as one of the most fundamental problems in computer vision, is widely used in object tracking, behavior understanding, human-computer interaction, facial recognition, and many other fields, attracting widespread attention and research from scholars since the early 21st century. Humans primarily receive external information through vision, so visual information-based application technologies will be a forward-looking research area in artificial intelligence. Among these, facial recognition, video surveillance, object detection, internet image content review, and biometric recognition technologies have become current research hotspots. These technologies are also widely used in fields such as healthcare, elderly care, transportation, urban operations, and security, for example: medical image diagnosis, pose estimation, station security checks, autonomous driving, vehicle speed detection, and video surveillance behavior analysis.

[0003] Open-vocabulary object detection is an emerging research field in recent years, aiming to break the closed-set assumption upon which traditional object detection is based. Its goal is to identify objects with open categories, namely a base category and a new category, and annotations during training apply only to the base category. Thanks to its ability to access large amounts of weakly supervised data during training, open-vocabulary object detection has the ability to recognize and locate arbitrary textual concepts, demonstrating outstanding performance in fields such as autonomous driving, medical image processing, and interactive detection.

[0004] Object detection technology is becoming increasingly mature, and general-purpose object detectors are widely used in various real-world scenarios such as daily life, urban management, and industrial manufacturing. However, most detectors still fail to achieve satisfactory results in open-vocabulary object detection tasks targeting new categories. In real-world applications, enabling general-purpose detectors to detect various object categories, especially new ones, remains a challenging problem. Therefore, open-vocabulary object detection tasks have recently attracted increasing attention.

[0005] Existing open object detection methods based on pre-trained visual language models (VLMs) outperform earlier zero-shot object detection methods. They extract features from images using convolutional neural networks (CNNs) and feed them into a detector for classification and regression. Then, the VLM is used as a teacher model for knowledge distillation, which can detect new categories of objects more accurately. Although VLM-based methods have made great breakthroughs in detection performance, there are still three main challenging problems due to the limitations of the granularity of features for new categories of objects: (1) The training data for open vocabulary object detection is generally a large number of weakly supervised image-text pairs and a small number of region-level annotations. The image-text pairs are responsible for enabling the detector to acquire open vocabulary capabilities, while the region-level annotations are responsible for enabling the detector to acquire general object localization capabilities. The two different granularities of data will produce domain differences, making it impossible for the detector to learn the knowledge of both types of data. (2) VLM-based methods will perform scaling and cropping operations on images during the image preprocessing stage. In the process, key information may be lost. Furthermore, the VLM trained on image-text pairs is not a region-centric detector. Therefore, how to better utilize the knowledge in VLM has become an increasingly important focus. (3) How open vocabulary detectors can effectively generalize their localization capabilities to new categories of targets helps improve their ability to identify new categories of targets.

[0006] To address the aforementioned issues, numerous methods for open-vocabulary object detection have been proposed. Based on their strategies, these methods can be categorized into knowledge distillation-based, transfer learning-based, feature fusion-based, visual-text spatial alignment-based, pseudo-label-based, and other methods. While these methods improve the performance of open-vocabulary object detection from different perspectives, they typically require numerous complex operations, incurring additional computational costs. Furthermore, some methods are difficult to design end-to-end, failing to meet the needs of real-world scenarios. Some methods even overemphasize novel object categories, hindering the detection of basic object categories.

[0007] Recent work has noted the impact of VLM on detector performance. For example, Li et al. (Liangqi Li, Jiaxu Miao, Dahu Shi, Wenming Tan, Ye Ren, Yi Yang, and Shiliang Pu. 2023. Distilling detail with visual-linguistic knowledge for open-vocabulary object detection. In Proceedings of the IEEE / CVF International Conference on Computer Vision. 6501–6510.) modeled object relationships by aligning the feature relationship matrices between student and teacher models; Wang et al. (Luting Wang, Yi Liu, Penghui Du, Zihan Ding, Yue Liao, Qiaosong Qi, Biaolong Chen, and Si Liu. 2023. Object-aware distillation pyramid for open-vocabulary object detection. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 11186–11196.) aligned image patch embeddings with the embeddings of the teacher model; Ma et al. (Zongyang Ma, Guan Luo, Jin Gao, Liang Li, Yuxin Chen, Shaoru) Wang, Congxuan Zhang, and Weiming Hu. 2022. Open-vocabulary one-stage detection with hierarchical visual-language knowledgedistillation. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 14074–14083.Extracting multi-scale global features from images to align captions; Han et al. (Kunyang Han, Yong Liu, Jun Hao Liew, Henghui Ding, Jiajun Liu, Yitong Wang, Yansong Tang, Yujiu Yang, Jiashi Feng, Yao Zhao, et al. 2023. Globalknowledge calibration for fast open-vocabulary segmentation. In Proceedings of the IEEE / CVF International Conference on Computer Vision. 797–807.) avoid overfitting by introducing additional category synonyms. Most of these methods explore better knowledge distillation architectures to more efficiently transfer knowledge from VLMs. Summary of the Invention

[0008] The main objective of this invention is to provide an open vocabulary target detection method, electronic device, and storage medium based on prototype library feature mining, which solves the problems existing in the prior art, improves the monitor's perception of categories, and thus improves the detection performance of new category targets.

[0009] To achieve the above objectives, one solution of the present invention is:

[0010] An open-vocabulary object detection method based on prototype library feature mining includes a category-generalized region proposal network (CG-RPN) and a category-specific knowledge distillation method (CSKD), and includes the following steps:

[0011] Step 1: First, preprocess the image to be detected, and then feed the preprocessed image to be detected and its corresponding image level label into the Visual Language Model (VLM).

[0012] Step 2: The VLM performs feature extraction and feature fusion on the image to be detected, and sends the resulting feature map into the CG-RPN; the CG-RPN selects sample boxes based on the features, assigns labels to the sample boxes based on the classification score, and sends them into the CSKD.

[0013] Step 3: CSKD uses the RoI head and VLM image encoder to process the sample boxes into two sets of features, and then uses a category-specific knowledge distillation method to transfer rich visual semantic knowledge from the VLM.

[0014] Step 4: The classification head and regression head obtain the final prediction results based on the features generated by the RoI head, and the GT calculates the relevant loss, which is then backpropagated to the VLM to adjust the parameters.

[0015] The preprocessing in step 1 involves first standardizing the image to be detected, then scaling the image to 256×256, and finally randomly cropping it to 224×224.

[0016] In step 2, feature extraction is performed using the backbone feature extraction networks ResNet50 and ResNet101, feature fusion is performed using the feature pyramid network FPN, and sample box generation and classification scores are generated by the category generalization region proposal network CG-RPN, and include the following steps:

[0017] Step 2.1: For each sample box and its corresponding image features, use a linear layer to map them to the feature dimension of the prototype library;

[0018] Step 2.2: Extract image features from the sample boxes using the detector, and calculate the cosine similarity score between the extracted image features and the features stored in the prototype library according to the following formula:

[0019] ;

[0020] in, Represents the features of the sample bounding box; Representing sample box features The components in the space formed by the prototype feature vectors, where It is obtained by processing the prototype vectors into a matrix based on singular value decomposition.

[0021] Step 3 of the CSKD includes the following steps:

[0022] Step 3.1: For each sample box, use the RoI head to obtain the feature set. The feature set is obtained using a VLM image encoder. ; for feature set and For each feature in the dataset, calculate the cluster center. and ;

[0023] Step 3.2: For the feature set and Each feature in the cluster, and the cluster center and The loss is calculated using the following formula:

[0024] ;

[0025] ;

[0026] in, and These represent the training losses of the relevant modules, respectively. Indicates the number of samples; Indicates a sample; This represents the feature set samples extracted by the detector; This represents the feature set samples extracted by VLM; Indicates the number of clusters; This represents the feature set sample score extracted by the detector; This represents the feature set sample score extracted by VLM; Indicates a sample; This represents the feature set sample score extracted by VLM.

[0027] The second solution of the present invention is:

[0028] An electronic device is characterized by comprising a processor, a memory, and an application program; the application program is stored in the memory and configured to be executed by the processor using the open vocabulary target detection method based on prototype library feature mining.

[0029] The third solution of the present invention is:

[0030] A computer-readable storage medium, characterized in that the storage medium stores a computer program; when the computer program is executed in the computer, it causes the computer to perform the open vocabulary target detection method based on prototype library feature mining.

[0031] After adopting the above technical solution, the present invention has the following technical effects:

[0032] (1) This invention has a unique perspective and re-examines the role of VLM in feature mining. It is the first method to use feature mining based on prototype library and category-related knowledge distillation in open vocabulary target detection. It aims to mine potential new category targets in the background. At the same time, it takes into account the semantic advantages of VLM, so that the detector can make full use of the powerful semantic space of VLM, help the detector improve the credibility of sample proposals, and make the features of different categories show significant differences after clustering.

[0033] (2) The feature of this invention is that the principle is simple and easy to implement, it is plug-and-play, it only uses bilinear layer to map background vectors and cosine similarity to calculate loss, and it can be easily transferred to other models and methods;

[0034] (3) The effects of this invention are obvious, specifically: ① After the region proposal network provides the sample box and the corresponding category score, the output classification confidence is adjusted by simple singular value extraction and similarity calculation between the candidate box and the existing high-quality foreground features; ② The RoIs of the same candidate box are respectively input into the student and teacher models to calculate features, the features of the same object are brought closer, and the cluster centers of the same category are brought closer, so as to make full use of the knowledge of VLM; ③ The calculated loss is backpropagated to the VLM detector, so that the detector can obtain the target localization ability and the new category perception ability. Attached Figure Description

[0035] Figure 1 This is a schematic diagram of the neural network structure according to a specific embodiment of the present invention;

[0036] Figure 2 This is a schematic diagram of the processing procedure in a specific embodiment of the present invention;

[0037] Figure 3 This is a performance comparison table of the present invention with existing technologies on the COCO dataset;

[0038] Figure 4 This is a performance comparison table of the present invention with existing technologies on the LVIS dataset;

[0039] Figure 5 This is a table of functional ablation experimental data for the present invention on the COCO dataset and in the prior art. Detailed Implementation

[0040] To further explain the technical solution of the present invention, the present invention will be described in detail below through specific embodiments.

[0041] refer to Figure 1-2 As shown, this invention discloses an open vocabulary target detection method based on prototype library feature mining, abbreviated as CAKE, which includes a category-generalized region proposal network CG-RPN and a category-specific knowledge distillation method CSKD, and includes the following steps:

[0042] Step 1: First, preprocess the image to be detected, and then feed the preprocessed image to be detected and its corresponding image level label into the Visual Language Model (VLM).

[0043] Step 2: The VLM performs feature extraction and feature fusion on the image to be detected, and sends the resulting feature map into the CG-RPN; the CG-RPN selects sample boxes based on the features, assigns labels to the sample boxes based on the classification score, and sends them into the CSKD.

[0044] Step 3: CSKD uses the RoI head and VLM image encoder to process the sample boxes into two sets of features, and then uses a category-specific knowledge distillation method to transfer rich visual semantic knowledge from the VLM.

[0045] Step 4: The classification head and regression head obtain the final prediction results based on the features generated by the RoI head, and the GT calculates the relevant loss, which is then backpropagated to the VLM to adjust the parameters.

[0046] Through the above scheme, this invention has a unique perspective, re-examining the role of VLM in feature mining. It is the first method to use feature mining based on prototype library and category-related knowledge distillation in open vocabulary target detection. It aims to mine potential new category targets in the background, while also noting the semantic advantages of VLM, enabling the detector to make full use of the powerful semantic space of VLM, helping the detector to improve the credibility of sample proposals, and making the features of different categories intuitively show significant differences after clustering.

[0047] The invention is characterized by its simple principle, ease of implementation, plug-and-play nature, and ability to easily transfer to other models and methods by using only a bilinear layer to map the background vector and cosine similarity to calculate the loss.

[0048] The present invention has significant effects, specifically:

[0049] (1) After the region proposal network provides the sample boxes and corresponding category scores, the output classification confidence is adjusted by simply extracting singular values ​​and calculating the score between the candidate boxes and the existing high-quality foreground features.

[0050] (2) RoIs with the same candidate boxes are input into the student and teacher models respectively to calculate features. Features of the same objects are brought closer together, and cluster centers of the same category are brought closer together, so as to make full use of VLM knowledge.

[0051] (3) The calculated loss is backpropagated to the VLM detector, thereby enabling the detector to gain target localization capability and new category perception capability.

[0052] The following illustrates specific embodiments of the present invention.

[0053] The preprocessing in step 1 above involves first standardizing the image to be detected (including scaling, cropping, color normalization, etc.), then scaling the image to 256×256, and finally randomly cropping it to 224×224.

[0054] In step 2 above, feature extraction is performed using the backbone feature extraction networks ResNet50 and ResNet101, feature fusion is performed using the feature pyramid network FPN, and sample box generation and classification scores are generated by the category generalization region proposal network CG-RPN, and include the following steps:

[0055] Step 2.1: For each sample box and its corresponding image features, use a linear layer to map them to the feature dimension of the prototype library;

[0056] Step 2.2: Extract image features from the sample boxes using the detector, and calculate the cosine similarity score (CSS) between the extracted image features and the features stored in the prototype library according to the following formula:

[0057] ;

[0058] in, Represents the features of the sample bounding box; Representing sample box features The components in the space formed by the prototype feature vectors (composed of the feature cluster centers of VLM), where The matrix is ​​obtained by processing the prototype vectors using singular value decomposition (SVD).

[0059] In step 3 above, CSKD includes the following steps:

[0060] Step 3.1: For each sample box, use the RoI head to obtain the feature set. The feature set is obtained using a VLM image encoder. ; for feature set and For each feature in the dataset, calculate the cluster center. and ;

[0061] Step 3.2: For the feature set and Each feature in the cluster, and the cluster center and The loss is calculated using the following formula:

[0062] ;

[0063] ;

[0064] in, and These represent the training losses of the relevant modules, respectively. Indicates the number of samples; Indicates a sample; This represents the feature set samples extracted by the detector; This represents the feature set samples extracted by VLM; Indicates the number of clusters; This represents the feature set sample score extracted by the detector; This represents the feature set sample score extracted by VLM; Indicates a sample; This represents the feature set sample score extracted by VLM. This function aims to bring cluster features of the same category closer together and push cluster features of different categories further apart, allowing the supervised detector to learn semantic knowledge of strong category correlations.

[0065] The effects of this invention are further illustrated by the following simulation experiments.

[0066] (1) Simulation conditions

[0067] This invention was developed on the Ubuntu platform, and the deep learning framework developed is based on PyTorch. The primary language used in this invention is Python.

[0068] (2) Simulation content

[0069] We selected two commonly used object detection datasets, COCO and LVIS, and applied common settings for open lexicons. Following the steps outlined above, we trained the network and tested it using the test set. Figure 3 and Figure 4 The detection results of the present invention and other methods on two datasets show that the present invention consistently improves the performance of the baseline algorithm, and achieves the best results compared to other methods. Figure 5 Functional ablation experiments of relevant modules of this method show that the present invention greatly improves the performance of the detector on new categories, among which... The model represents the new category. result, Represents the model in basic categories result.

[0070] Extensive experiments on the COCO and LVIS datasets demonstrate that, compared to other competing methods, the proposed CAKE not only achieves significant and consistent performance improvements over baseline methods but also yields the best detection results.

[0071] The present invention also discloses an electronic device, including a processor, a memory, and an application program; the application program is stored in the memory and configured to be executed by the processor using the methods described above.

[0072] The present invention also discloses a computer-readable storage medium storing a computer program; when the computer program is executed in a computer, it causes the computer to perform the above-described method. The storage medium may include various media capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0073] The above embodiments and figures are not intended to limit the product form and style of the present invention. Any appropriate changes or modifications made by those skilled in the art should be considered as not departing from the patent scope of the present invention.

Claims

1. An open vocabulary object detection method based on prototype library feature mining, characterized in that, It includes a category-generalized region proposal network (CG-RPN) and a category-specific knowledge distillation method (CSKD), and includes the following steps: Step 1: First, preprocess the image to be detected, and then feed the preprocessed image to be detected and its corresponding image level label into the Visual Language Model (VLM). Step 2: The VLM performs feature extraction and feature fusion on the image to be detected, and sends the resulting feature map into the CG-RPN; the CG-RPN selects sample boxes based on the features, assigns labels to the sample boxes based on the classification score, and sends them into the CSKD. The methods by which CG-RPN achieves category generalization include: Step 2.1: For each sample box and its corresponding image features, use a linear layer to map them to the feature dimension of the prototype library; Step 2.2: Extract image features from the sample boxes using the detector. Calculate the cosine similarity score between the extracted image features and the features stored in the prototype library according to the following formula. Adjust the classification score of the sample boxes using the cosine similarity score: ； wherein, represents a sample frame feature; represents a sample frame feature in the space of the prototype feature vectors, wherein obtained from singular value decomposition of a matrix of the prototype vectors; Step 3: CSKD uses the RoI head and VLM image encoder to process the sample boxes into two sets of features, and then uses a category-specific knowledge distillation method to transfer the visual semantic knowledge of VLM to the RoI head. The category-specific knowledge distillation methods include: Step 3.1, for each sample box, get a feature set using RoI head , get a feature set using VLM image encoder ; for each feature in the feature set and , compute cluster centers and ; Step 3.2, for each feature in the feature set and , and the cluster centers and , the loss is computed as follows, which pulls features of the same class closer and pushes features of different classes further apart: ； ； in, and These represent the target distillation loss and the category distillation loss, respectively. Indicates the number of samples; Indicates a sample; This represents the feature set samples extracted by the detector; This represents the feature set samples extracted by VLM; Indicates the number of clusters; This represents the feature set sample score extracted by the detector; This represents the feature set sample score extracted by VLM; Indicates a sample; This represents the feature set sample score extracted by VLM; Step 4: The classification head and regression head obtain the final prediction results based on the features generated by the RoI head, and the GT calculates the relevant loss. Then, the losses from Step 3 and Step 4 are backpropagated to VLM to adjust the parameters.

2. The open vocabulary target detection method based on prototype library feature mining as described in claim 1, characterized in that: The preprocessing in step 1 involves first standardizing the image to be detected, then scaling the image to 256×256, and finally randomly cropping it to 224×224.

3. The open vocabulary target detection method based on prototype library feature mining as described in claim 1, characterized in that: In step 2, feature extraction is performed using the backbone feature extraction networks ResNet50 and ResNet101, feature fusion is performed using the feature pyramid network FPN, and sample box generation and classification score are generated by the category generalization region proposal network CG-RPN.

4. An electronic device, characterized in that... It includes a processor, memory, and an application program; the application program is stored in the memory and configured to be executed by the processor as described in any one of claims 1 to 3, the open vocabulary target detection method based on prototype library feature mining.

5. A computer readable storage medium characterized by The storage medium stores a computer program; when the computer program is executed in the computer, it causes the computer to perform the open vocabulary target detection method based on prototype library feature mining as described in any one of claims 1 to 3.

Images