Open vocabulary target detection method based on prototype library feature mining, electronic equipment and storage medium

Abstract

The invention discloses an open vocabulary target detection method based on prototype library feature mining, electronic equipment and a storage medium, which improve the detection performance of new category targets, and comprises a category generalization regional proposal network CG-RPN and a category specific knowledge distillation method CSKD, and comprises the following steps: 1, preprocessing a to-be-detected picture, sending the preprocessed to-be-detected picture and the corresponding picture level label into a visual language model (VLM); 2, performing feature extraction and feature fusion on a to-be-detected picture by the VLM, and sending an obtained feature map into the CG-RPN; the CG-RPN screens sample frames according to the features, allocates labels to the sample frames according to the classification scores and sends the labels to CSKD; 3, the CSKD respectively uses RoI head and VLM image encoder to process the sample frame into two groups of features, and then uses a category-specific knowledge distillation method to transfer rich visual semantic knowledge from the VLM; and 4, the classification head and the regression head obtain a final prediction result according to features generated by the RoI head, and correlation loss is calculated by GT and is reversely propagated to VLM to adjust parameters.

Description

Technical Field

[0001] The invention belongs to the technical field of computer vision, and in particular refers to an open vocabulary target detection method based on prototype library feature mining, an electronic device and a storage medium. Background Art

[0002] In recent years, with the improvement of computer performance and the development of big data, visual information data has increased rapidly, including multimedia data such as static images, dynamic images, video files, audio files, etc., which are spread at a very fast speed on various social media. As one of the most basic problems in the field of computer vision, target detection is widely used in many fields such as target tracking, behavior understanding, human-computer interaction, face recognition, etc., and has attracted extensive attention and research from many scholars in the early 21st century. Humans mainly receive external information through vision, so application technology based on visual information will be a forward-looking research point of artificial intelligence. Among them, face recognition, video surveillance, target detection, Internet image content review, biometric recognition and other technologies have become research hotspots today. These technologies are also widely used in medical care, elderly care, transportation, urban operation, security and other fields, such as: medical image diagnosis, posture estimation, station security, automatic driving, vehicle speed detection, video surveillance behavior analysis, etc.

[0003] Open vocabulary object detection is an emerging research field in recent years, which aims to break the closed set assumption based on traditional object detection. It aims to identify objects of open categories, i.e., basic categories and new categories, and annotations only apply to basic categories during training. Thanks to the fact that open vocabulary object detection can access a large amount of weakly supervised data during training, this enables open vocabulary detectors to recognize and locate arbitrary text concepts, and has shown excellent performance in fields such as autonomous driving, medical image processing, and interactive detection.

[0004] At present, object detection technology is becoming more and more mature, and general object detectors are also widely used in various real-life scenarios such as daily life, urban management, and industrial manufacturing. However, most detectors still cannot achieve satisfactory results in open vocabulary detection tasks for new categories. In real application scenarios, it is still a challenging problem for general detectors to have the ability to detect targets of various categories, especially new categories. Therefore, the open vocabulary object detection task has attracted more and more attention recently.

[0005] The performance of existing open object detection methods based on pre-trained visual language models (VLMs) is far superior to that of early zero-shot object detection methods. Convolutional neural networks (CNNs) are used to extract features from images and feed them into the detection head for classification and regression. VLM is then used as a teacher model for knowledge distillation, which can more accurately detect new categories of objects. Although VLM-based methods have made great breakthroughs in detection performance, due to the limitations of the granularity of new category object features, there are still three major challenging issues: (1) The training data for open vocabulary object detection is generally a large number of weakly supervised image-text pairs and a small number of region-level annotations. The image-text pairs are responsible for enabling the detector to acquire open vocabulary capabilities, while the region-level annotations are responsible for enabling the detector to acquire general object localization capabilities. The data at two different granularities will produce domain differences, making it impossible for the detector to comprehensively learn the knowledge of the two data. (2) VLM-based methods will scale and crop the images during the image preprocessing stage, which may lose key information in the process. In addition, the VLM trained on image-text pairs is not a region-centric detector. Therefore, how to better utilize the knowledge in VLM has become an increasing focus of attention. (3) How the open vocabulary detector can generalize the localization capability to new categories of targets helps improve the recognition capability of new categories of targets.

[0006] In order to solve the above problems, many methods for open vocabulary object detection have been proposed. According to the strategies of different methods, they can be divided into knowledge distillation-based methods, transfer learning-based methods, feature fusion-based methods, visual-text space alignment-based methods, pseudo-label-based methods and other methods. Although these methods have improved the performance of open vocabulary object detection from different perspectives, they usually require a large number of complex operations, resulting in additional computational costs, and some methods are difficult to design in an end-to-end manner, making it difficult to meet the needs of real-world scenarios. Some methods even focus too much on new categories of objects, which hinders the ability to detect basic categories of objects.

[0007] There are many recent works that have noticed the impact of VLM on detector performance. For example, Li et al. (Liangqi Li, Jiaxu Miao, Dahu Shi, Wenming Tan, Ye Ren, Yi Yang, and Shiliang Pu. 2023. Distilling detr with visual-linguistic knowledge for open-vocabulary object detection. In Proceedings of the IEEE / CVF International Conference on Computer Vision. 6501–6510.) model object relations by aligning the feature relationship matrices between the student and teacher models; Wang et al. (Luting Wang, Yi Liu, Penghui Du, Zihan Ding, Yue Liao, Qiaosong Qi, Biaolong Chen, and Si Liu. 2023. Object-aware distillation pyramid for open-vocabulary object detection. In Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition. 11186–11196.) align image patch embeddings with the embeddings of the teacher model; Ma et al. (Zongyang Ma, Guan Luo, Jin Gao, Liang Li, Yuxin Chen, Shaoru Wang, Congxuan Zhang, and Weiming Hu. 2022. Open-vocabulary one-stage detection with hierarchical visual-language knowledgedistillation. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 14074–14083.) extracts multi-scale global features of images to align titles; Han et al. (Kunyang Han, Yong Liu, Jun Hao Liew, Henghui Ding, Jiajun Liu, YitongWang, Yansong Tang, Yujiu Yang, Jiashi Feng, Yao Zhao, et al. 2023. Globalknowledge calibration for fast open-vocabulary segmentation. In Proceedingsof the IEEE / CVF International Conference on Computer Vision. 797–807.) avoid overfitting by introducing additional category synonyms. Most of these methods explore better knowledge distillation architectures to more efficiently transfer knowledge from VLM. . Summary of the invention

[0008] The main purpose of the present invention is to provide an open vocabulary target detection method, electronic device and storage medium based on prototype library feature mining, which can solve the problems existing in the prior art and improve the monitor's perception of categories, thereby improving the detection performance of new category targets.

[0009] In order to achieve the above object, one of the solutions of the present invention is: An open vocabulary object detection method based on prototype library feature mining includes a category generalization region proposal network CG-RPN and a category specific knowledge distillation method CSKD, and includes the following steps: Step 1: First, preprocess the image to be detected, and then send the preprocessed image to be detected and its corresponding image-level label to the visual language model VLM; Step 2: VLM extracts and fuses features of the image to be detected, and sends the obtained feature map to CG-RPN; CG-RPN selects sample frames according to the features, assigns labels to the sample frames according to the classification scores, and sends them to CSKD; Step 3: CSKD uses RoI head and VLM image encoder to process the sample frame into two sets of features, and then uses category-specific knowledge distillation method to transfer rich visual semantic knowledge from VLM. Step 4: The classification head and regression head obtain the final prediction results based on the features generated by the RoI head, and the GT calculates the relevant losses, which are then back-propagated to the VLM to adjust the parameters.

[0010] The preprocessing in step 1 first standardizes the image to be detected, then scales the image to a size of 256×256, and finally randomly crops it to a size of 224×224.

[0011] In step 2, feature extraction is performed using backbone feature extraction networks ResNet50 and ResNet101, feature fusion is performed using feature pyramid network FPN, sample frame generation and classification scores are generated by category generalization region proposal network CG-RPN, and the following steps are included: Step 2.1: For each sample frame and corresponding image feature, use a linear layer to map it to the prototype library feature dimension; Step 2.2: Use the detector to extract image features from the sample frame, and calculate the cosine similarity score between the extracted image features and the features stored in the prototype library according to the following formula: ; in, Represents the sample frame characteristics; Represents the sample frame characteristics The component in the space formed by the prototype eigenvectors, where It is obtained by processing the matrix composed of the prototype vectors according to the singular value decomposition.

[0012] In step 3, CSKD includes the following steps: Step 3.1: For each sample frame, use RoI head to get the feature set , use VLM image encoder to get the feature set ; For feature set and For each feature in , calculate the cluster center and ; Step 3.2: Feature set and Each feature in, and the cluster center and , the loss is calculated according to the following formula: ; ; in, and Respectively represent the training losses of related modules; represents the number of samples; Indicates a sample; Represents the feature set samples extracted by the detector; represents the feature set samples extracted by VLM; Indicates the number of clusters; Represents the sample fraction of the feature set extracted by the detector; represents the sample fraction of feature set extracted by VLM; Indicates a sample; Represents the sample fraction of feature set extracted by VLM.

[0013] The second solution of the present invention is: An electronic device, characterized by comprising a processor, a memory and an application; the application is stored in the memory and is configured so that the processor executes the open vocabulary target detection method based on prototype library feature mining.

[0014] The third solution of the present invention is: A computer-readable storage medium, characterized in that the storage medium stores a computer program; when the computer program is executed in the computer, the computer executes the open vocabulary target detection method based on prototype library feature mining.

[0015] After adopting the above technical solution, the present invention has the following technical effects: (1) This invention has a unique perspective and re-examines the role of VLM in feature mining. It is the first method to use prototype-based feature mining and category-strong correlation knowledge distillation in open vocabulary object detection, aiming to mine potential new category targets in the background. At the same time, it pays attention to the semantic advantages of VLM, allowing the detector to make full use of the powerful semantic space of VLM, helping the detector to improve the credibility of sample proposals, so that features of different categories can intuitively show large differences after clustering. (2) The present invention is characterized by its simple principle, easy implementation, plug-and-play, and only uses a bilinear layer to map the background vector and cosine similarity to calculate the loss, which can be easily migrated to other models and methods; (3) The present invention has obvious effects, specifically: ① After the region proposal network gives the sample box and the corresponding category score, the score between the candidate box and the existing high-quality foreground feature is calculated by simple singular value extraction and similarity to adjust the output classification confidence; ② The RoI of the same candidate box is input into the student and teacher models respectively to calculate the features, the features of the same object are brought closer, and the cluster centers of the same category are brought closer, so as to make full use of the knowledge of VLM; ③ The loss generated by the calculation is back-propagated to the detector of VLM, so that the detector can obtain the target positioning ability and the new category perception ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A schematic diagram of a neural network structure of a specific embodiment of the present invention; Figure 2 It is a schematic diagram of the processing process of a specific embodiment of the present invention; Figure 3 This is a data table comparing the performance of the present invention with that of the prior art on the COCO dataset; Figure 4 This is a data table comparing the performance of the present invention with that of the prior art on the LVIS data set; Figure 5 This is a functional ablation experiment data table of the present invention and the prior art on the COCO dataset. DETAILED DESCRIPTION

[0017] In order to further explain the technical solution of the present invention, the present invention is described in detail below through specific embodiments.

[0018] refer to Figure 1-2 As shown, the present invention discloses an open vocabulary object detection method based on prototype library feature mining, referred to as CAKE, including a category-generalized region proposal network CG-RPN and a category-specific knowledge distillation method CSKD, and includes the following steps: Step 1: First, preprocess the image to be detected, and then send the preprocessed image to be detected and its corresponding image-level label to the visual language model VLM; Step 2: VLM extracts and fuses features of the image to be detected, and sends the obtained feature map to CG-RPN; CG-RPN selects sample frames according to the features, assigns labels to the sample frames according to the classification scores, and sends them to CSKD; Step 3: CSKD uses RoI head and VLM image encoder to process the sample frame into two sets of features, and then uses category-specific knowledge distillation method to transfer rich visual semantic knowledge from VLM. Step 4: The classification head and regression head obtain the final prediction results based on the features generated by the RoI head, and the GT calculates the relevant losses, which are then back-propagated to the VLM to adjust the parameters.

[0019] Through the above scheme, the present invention has a unique perspective and re-examines the role of VLM in feature mining. It is the first method to use prototype library-based feature mining and category-strong correlation knowledge distillation in open vocabulary object detection, aiming to mine potential new category targets in the background. At the same time, it pays attention to the semantic advantages of VLM, so that the detector can make full use of the powerful semantic space of VLM, helping the detector to improve the credibility of sample proposals, so that features of different categories can intuitively show large differences after clustering. The invention is characterized in that the principle is simple and easy to implement, plug and play, and only uses a bilinear layer to map the background vector and cosine similarity to calculate the loss, and can be easily migrated to other models and methods; The effects of the present invention are obvious, specifically: ① After the region proposal network gives the sample frame and the corresponding category score, the output classification confidence is adjusted by simple singular value extraction and similarity calculation between the candidate frame and the existing high-quality foreground features; ② The RoIs of the same candidate boxes are input into the student and teacher models respectively to calculate features. The features of the same objects are brought closer together, and the cluster centers of the same categories are brought closer together, making full use of the knowledge of VLM. ③ The calculated loss is back-propagated to the VLM detector, so that the detector can acquire the ability to locate targets and perceive new categories.

[0020] Specific embodiments of the present invention are shown below.

[0021] In the preprocessing of step 1 above, the image to be detected is first standardized (including scaling, cropping, color normalization, etc.), then the image is scaled to 256×256, and finally randomly cropped to 224×224.

[0022] In the above step 2, feature extraction is completed using the backbone feature extraction network ResNet50 and ResNet101, feature fusion is completed using the feature pyramid network FPN, sample frame generation and classification scores are generated by the category generalization region proposal network CG-RPN, and the following steps are included: Step 2.1: For each sample frame and corresponding image feature, use a linear layer to map it to the prototype library feature dimension; Step 2.2: Use the detector to extract image features from the sample frame, and calculate the cosine similarity score (CSS) between the extracted image features and the features stored in the prototype library according to the following formula: ; in, Represents the sample frame characteristics; Represents the sample frame characteristics The component in the space formed by the prototype feature vector (composed of the feature clustering centers of the VLM), where The matrix composed of prototype vectors is obtained by processing the matrix with singular value decomposition (SVD).

[0023] In step 3 above, CSKD includes the following steps: Step 3.1: For each sample frame, use RoI head to get the feature set , use VLM image encoder to get the feature set ; For feature set and For each feature in , calculate the cluster center and ; Step 3.2: Feature set and Each feature in, and the cluster center and , the loss is calculated according to the following formula: ; ; in, and Respectively represent the training losses of related modules; represents the number of samples; Indicates a sample; Represents the feature set samples extracted by the detector; represents the feature set samples extracted by VLM; Indicates the number of clusters; Represents the sample fraction of the feature set extracted by the detector; represents the sample fraction of feature set extracted by VLM; Indicates a sample; Represents the sample score of the feature set extracted by VLM. This function aims to bring the clustering features of the same category closer and push the clustering features of different categories farther, supervising the detector to learn the semantic knowledge of strong correlation between categories.

[0024] The effect of the present invention is further illustrated by the following simulation experiment.

[0025] (1) Simulation conditions The present invention is developed on the Ubuntu platform, and the developed deep learning framework is based on Pytorch. The main language used in the present invention is Python.

[0026] (2) Simulation content Take two commonly used object detection datasets COCO and LVIS in the field of object detection, and apply the common settings of open vocabulary. Train the network according to the above steps and test it with the test set. Figure 3 and Figure 4 The detection results of the present invention and other methods on two data sets are shown respectively. It can be found that the present invention consistently improves the performance of the baseline algorithm, and compared with other methods, the present invention achieves the best result. Figure 5 The functional ablation experiment of the modules related to this method shows that the present invention has a great help to the performance of the detector on the new category. Represents the model on the new category result, Indicates the model's basic category result.

[0027] Extensive experiments on COCO and LVIS datasets demonstrate that the proposed CAKE not only achieves significant and consistent performance improvements over baseline methods, but also achieves state-of-the-art detection results compared to other competitive methods.

[0028] The present invention also discloses an electronic device, comprising a processor, a memory and an application program; the application program is stored in the memory and is configured so that the processor executes the above method.

[0029] The present invention also discloses a computer-readable storage medium, which stores a computer program; when the computer program is executed in a computer, the computer executes the above method. The storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0030] The above embodiments and drawings do not limit the product form and style of the present invention. Any appropriate changes or modifications made by ordinary technicians in the relevant technical field should be deemed to be within the patent scope of the present invention.

Claims

1. An open vocabulary object detection method based on prototype library feature mining, characterized in that It includes a category-generalized region proposal network CG-RPN and a category-specific knowledge distillation method CSKD, and includes the following steps: Step 1: First, preprocess the image to be detected, and then send the preprocessed image to be detected and its corresponding image-level label to the visual language model VLM; Step 2: VLM extracts and fuses features of the image to be detected, and sends the obtained feature map to CG-RPN; CG-RPN selects sample frames according to the features, assigns labels to the sample frames according to the classification scores, and sends them to CSKD; Step 3: CSKD uses RoI head and VLM image encoder to process the sample frame into two sets of features, and then uses category-specific knowledge distillation method to transfer rich visual semantic knowledge from VLM. Step 4: The classification head and regression head obtain the final prediction results based on the features generated by the RoI head, and the GT calculates the relevant losses, which are then back-propagated to the VLM to adjust the parameters.

2. The open vocabulary object detection method based on prototype library feature mining as claimed in claim 1, characterized in that: The preprocessing in step 1 first standardizes the image to be detected, then scales the image to a size of 256×256, and finally randomly crops it to a size of 224×224.

3. The open vocabulary object detection method based on prototype library feature mining as claimed in claim 1, characterized in that In step 2, feature extraction is performed using backbone feature extraction networks ResNet50 and ResNet101, feature fusion is performed using feature pyramid network FPN, sample frame generation and classification scores are generated by category generalization region proposal network CG-RPN, and the following steps are included: Step 2.1: For each sample frame and corresponding image feature, use a linear layer to map it to the prototype library feature dimension; Step 2.2: Use the detector to extract image features from the sample frame, and calculate the cosine similarity score between the extracted image features and the features stored in the prototype library according to the following formula: ； in, Represents the sample frame characteristics; Represents the sample frame characteristics The component in the space formed by the prototype eigenvectors, where It is obtained by processing the matrix composed of the prototype vectors according to the singular value decomposition.

4. The open vocabulary object detection method based on prototype library feature mining as claimed in claim 1, characterized in that In step 3, CSKD includes the following steps: Step 3.1: For each sample frame, use RoI head to get the feature set , use VLM image encoder to get the feature set ; For feature set and For each feature in , calculate the cluster center and ; Step 3.2: Feature set and Each feature in, and the cluster center and , the loss is calculated according to the following formula: ； ； in, and Respectively represent the training losses of related modules; represents the number of samples; Indicates a sample; Represents the feature set samples extracted by the detector; represents the feature set samples extracted by VLM; Indicates the number of clusters; Represents the sample fraction of the feature set extracted by the detector; represents the sample fraction of feature set extracted by VLM; Indicates a sample; Represents the sample fraction of feature set extracted by VLM.

5. An electronic device, characterized in that It comprises a processor, a memory and an application program; the application program is stored in the memory and is configured so that the processor executes the open vocabulary target detection method based on prototype library feature mining as described in any one of claims 1 to 5.

6. A computer-readable storage medium, characterized in that The storage medium stores a computer program; when the computer program is executed in the computer, the computer is caused to execute the open vocabulary object detection method based on prototype library feature mining as described in any one of claims 1 to 5.

Images