A zero-shot object detection method based on deep learning

Abstract

The application discloses a zero-shot target detection method based on deep learning, and relates to the fields of computer vision and machine learning.The zero-shot target detection method based on deep learning can improve the detection accuracy of base classes and the generalization ability to new classes on the basis of an existing open-vocabulary target detection model by adding a simple module without increasing training costs.The accuracy and generalization ability of zero-shot target detection are improved by encoding and feature extraction of images and text descriptions, and refining the classification probability of each proposal in the image by using a global classification scoring module.The accuracy of detection is improved by refining the classification probability of each region candidate obtained by a zero-shot target detection classifier by using the CLIP model for image classification probability.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and machine learning, and in particular to a zero-shot object detection system based on deep learning. Background Technology

[0002] With the continuous development of deep learning technology, object detection has made significant progress. However, traditional object detection methods often require a large amount of labeled data for training when handling new object categories, which limits their flexibility in practical applications. To address this issue, Zero-Shot Object Detection (ZSD) has emerged, aiming to detect new object categories that do not appear in the training set.

[0003] Existing zero-shot object detection methods are mainly divided into embedding-based learning methods and generative adversarial methods. Zhou X, Girdhar R, Joulin A, et al. Detecting twenty-thousand classes using image-level supervision[C] / / European Conference on Computer Vision.Cham:Springer Nature Switzerland,2022:350-368 applies image-level labels to the largest region in the region proposal. This method does not require a complex prediction-based label assignment mechanism, but directly uses image-level annotations to constrain the object detection classifier. Under the constraint of image-level annotations, the detector trained by Detic can generalize to new datasets without fine-tuning. However, this method relies on the largest region in the region proposal, which may lead to a decrease in accuracy when dealing with smaller targets. Lin J, Shen Y, Wang B, et al. Weakly supervised open-vocabulary object detection[C] / / Proceedings of the AAAI Conference on Artificial Intelligence.2024,38(4):3404-3412. This paper achieves the detection and localization of open-vocabulary concepts on diverse datasets with only image-level annotations through three key strategies: dataset-level feature adaptation, image-level salient object localization, and region-level visual-language alignment. However, when facing large-scale datasets, the feature adaptation and alignment processes may become performance bottlenecks, affecting detection speed. Lin C, Jiang Y, Qu L, et al. Generative Region-Language Pretraining for Open-Ended Object Detection[C] / / Proceedings of the IEEE / CVF Conference on ComputerVision and Pattern Recognition.2024:13958-13968. This framework redefines the object detection problem as a generative problem and proposes an end-to-end learning framework consisting of a visual object detector and a language model. It directly generates object names in free form and enriches label diversity through a pseudo-label method, realizing open-vocabulary object detection without the need for predefined categories in the inference stage.While pseudo-labels increase label diversity in this method, their quality is difficult to guarantee, affecting detection performance. Pham C, Vu T, Nguyen K. Lp-ovod: Open-vocabulary object detection by linear probing[C] / / Proceedings of the IEEE / CVF Winter Conferenceon Applications of Computer Vision.2024:779-788. This method improves detection performance for new categories by training a sigmoid linear classifier on pseudo-labels retrieved from new relevant region proposals to discard low-quality boxes. Although discarding low-quality boxes improves detection performance, the detection accuracy for new categories still needs improvement. Jeong J, Park G, Yoo J, et al. ProxyDet: Synthesizing Proxy Novel Classes via ClasswiseMixup for Open-Vocabulary Object Detection[C] / / Proceedings of the AAAIConference on Artificial Intelligence.2024,38(3):2462-2470. This paper proposes to synthesize Proxy Novel Classes that approximate the new classes through a linear mixture between a pair of base classes. Our detector is trained using these synthesized approximate new classes, thereby preserving the position of the invisible classes in the feature space.

[0004] The present invention can improve the detection accuracy of the base class and the generalization ability to new classes on the basis of existing open vocabulary object detection models by simply adding a simple module without increasing training costs. Summary of the Invention

[0005] This invention addresses the problem of insufficient generalization ability of classification heads for new classes in existing zero-shot object detection methods when handling zero-shot object detection tasks. By encoding and extracting features from image and text descriptions, and refining the classification probability of each proposal in the image using a global classification scoring module, the accuracy and generalization ability of zero-shot object detection are improved. Furthermore, by utilizing the CLIP model to refine the classification probabilities of each region candidate obtained by the zero-shot object detection classifier, the detection accuracy is further enhanced.

[0006] The technical solution of this invention is: a zero-shot target detection method based on deep learning, the method comprising:

[0007] Step 1: Obtain sample data, which includes images and corresponding labels;

[0008] Step 2: Use the CLIP text encoder to extract features from the tags, obtaining text features t. i ′;

[0009] Step 3: Use the CLIP image encoder to extract features from the image to obtain visual features v′;

[0010] Step 4: Calculate text features t i The similarity between v′ and visual feature v′;

[0011] Step 5: Establish a zero-shot object detection model, and use the established zero-shot object detection model to initially obtain the classification results and detection boxes of the image;

[0012] Step 6: Combine the classification results from Step 5 with the similarity results from Step 4, and obtain the final detection result based on the combination result.

[0013] Furthermore, the data requirements for obtaining the data in step 1 are as follows: Let the set of all category names in the dataset be... The total number of elements is N, and the set of class names is . Each collection of invisible classes is Visible classes and invisible classes have no intersection. All categories are precisely divided into visible and invisible categories. The set of text descriptions for the target category is {d} ij} i＝1,2,…N；j＝1,2,…m , where d ij This represents the j-th description of the i-th category.

[0014] Furthermore, the specific method for step 2 is as follows:

[0015] Step 2.1: Tag multiple text descriptions of different categories to obtain a tag sequence;

[0016] Step 2.2: Encode the labeled sequence using the CLIP text encoder, obtaining multiple different text features for each category; assume the text encoder is a function E. t , then d ij The text feature vector is: e ij =E t (d ij );

[0017] Step 2.3: Normalize the obtained text feature vectors to ensure that the norm of each vector is 1. The normalized feature vectors are as follows:

[0018] Step 2.4: For the text features obtained from multiple descriptions of the i-th category, take the average value as the text feature t of that category. i The average eigenvector is Finally, the text features t of this category are obtained. i After normalization, its norm is ensured to be 1:

[0019] Furthermore, the specific method for step 3 is as follows:

[0020] Image I is preprocessed to resize it to the input size required by the CLIP model; the preprocessed image is represented as I′. The preprocessed image is then encoded using the CLIP image encoder to obtain a high-dimensional visual feature vector. The image encoder is assumed to be a function E. v Then the visual feature vector of the image is: v = E v (I′); The obtained visual feature vectors are normalized to ensure that the norm of each vector is 1; the normalized visual feature vectors are:

[0021] Furthermore, the similarity in step 4 is S. i ;

[0022] Where d is the dimension of the feature vector, v j ' represents the image, t' ij Let j represent the j-th component of the text feature vector of the i-th category.

[0023] Furthermore, the specific method for step 5 is as follows:

[0024] Step 5.1: Establish a zero-shot target detection model;

[0025] Step 5.2: Preprocess the image; the preprocessed image is represented as X′.

[0026] Step 5.3: Extract the feature map F = E of the image. d (X′); A set of candidate boxes is generated on the feature map using a region proposal network; the set of candidate boxes is represented as Among them B i This represents the i-th candidate bounding box in the image, and there are a total of n candidate bounding boxes;

[0027] Step 5.4: Input the features of the candidate boxes into the classification layer and calculate the preliminary classification probability of each candidate box belonging to each category; the classification head is function C, then the preliminary classification probability of the i-th candidate box in the image is: p i =C(F,B) i Among them, p iIt is a vector representing the probability that the candidate box belongs to each category;

[0028] Step 5.5: Input the features of the candidate boxes into the regression head and calculate the detection box coordinates of each candidate box; the regression head is a function R, then the detection box coordinates of the i-th candidate box in the image are: b i =R(F,B i ); where b i It is a vector representing the coordinates of the top-left and bottom-right corners of the candidate box.

[0029] Furthermore, the specific method for step 6 is as follows:

[0030] Step 6.1: First, perform a logit transformation on the classification probability of the zero-shot object detection model.

[0031] Step 6.2: Perform exponentiation and logit transformation on the similarity obtained in Step 4: Where α is an adjustment factor, which is set according to the actual situation;

[0032] Step 6.3: Add the results of Step 6.1 and Step 6.2 to obtain the fused classification score. Finally, the refined classification probability is obtained by using the Sigmoid function:

[0033] Compared with the prior art, the present invention has the following significant advantages and positive effects: by using the CLIP model to extract global features from the image and combining it with the existing zero-shot target detection model, the classification probability of each proposal is refined, which can effectively handle new categories that do not appear in the training set and improve the detection performance of new categories. Attached Figure Description

[0034] Figure 1 This is a diagram illustrating the overall strategic framework of the present invention. Detailed Implementation

[0035] This invention proposes a zero-shot object detection method based on the CLIP model and existing zero-shot object detection models. By encoding and extracting features from image and text descriptions, and refining the classification probability of each proposal in the image using a global classification scoring module, the accuracy and generalization ability of zero-shot object detection are improved. Furthermore, by utilizing the classification probability of the image using the CLIP model, the classification probability of each region candidate obtained by the zero-shot object detection classifier is refined, thereby improving detection accuracy. The specific implementation is as follows:

[0036] Step 1: Extract text features for each category using CLIP's text encoder. The set of all category names in the dataset is as follows: The total number of elements is N, and the set of class names is . Each collection of invisible classes is Visible classes and invisible classes have no intersection. All categories are precisely divided into visible and invisible categories. Input the text descriptions of the target category into the system. The text descriptions can include m different descriptive forms, such as "a photo of a{object}", "a close-up photo of a{object}", etc. Tokenize the multiple text descriptions of different categories, converting the natural language text into a tokenized sequence that the model can process. Assume the set of text descriptions for the target category is...

[0037] {d ij} i＝1,2,…N；j＝1,2,…m , where d ij Let represent the j-th description of the i-th category. The labeled sequence is encoded using the CLIP text encoder, yielding multiple distinct text features for each category. Assume the text encoder is a function E. t , then d ij The text feature vector is: e ij =E t (d ij The obtained text feature vectors are then normalized to ensure that the norm of each vector is 1. The normalized feature vectors are as follows: For the text features obtained from multiple descriptions of the i-th category, the average value is taken as the text feature t of that category. i Then the average eigenvector is Finally, the text features t of this category are obtained. i After normalization, its norm is ensured to be 1:

[0038] Step 2: Extract global features of the image using CLIP's image encoder. (This involves extracting the global features of the image to be detected.) Input System. The image is preprocessed, resized to the CLIP model's required input size of 224*224, and the preprocessed image is represented as I′. The preprocessed image is then encoded using the CLIP image encoder to obtain a high-dimensional visual feature vector. Assume the image encoder is a function E. v Then the visual feature vector of the image is: v = E v (I′). The obtained visual feature vectors are normalized to ensure that the norm of each vector is 1. The normalized visual feature vectors are:

[0039] Step 3: Calculate the cosine similarity between visual features and text features. The normalized text feature vector t for each category has already been obtained using the CLIP text encoder. i '', where i represents the category number. The normalized visual feature vector v' for each image has been obtained using the CLIP visual encoder. The cosine similarity S between the input image and the i-th category is calculated. i . Where d is the dimension of the feature vector v′ j and t′ ij Let represent the j-th component of the feature vector of the image and the text of the i-th category, respectively.

[0040] Step 4: Obtain preliminary classification results and detection boxes for candidate regions using existing zero-shot object detection models. Input the image into an existing zero-shot object detection model (using the Detic model as an example here) to obtain a preprocessed image, represented as X′. Input the preprocessed image into the backbone network E of the Detic model. d In the process, the feature map F = E of the image is extracted. d (X′). A set of candidate boxes is generated on the feature map using a region proposal network. The set of candidate boxes is represented as... Among them B i Let represent the i-th candidate bounding box in the image, and there are a total of n candidate bounding boxes. The features of the candidate bounding boxes are input into the classification layer to calculate the preliminary classification probability of each candidate bounding box belonging to each category. Assuming the classification head is function C, the preliminary classification probability of the i-th candidate bounding box in the image is: p i =C(F,B) i Among them, p i This is a vector representing the probability that the candidate box belongs to each category. The features of the candidate boxes are input into the regression head to calculate the detection box coordinates for each candidate box. Assuming the regression head is a function R, the detection box coordinates of the i-th candidate box in the image are: b i =R(F,B i ), where b i It is a vector representing the coordinates of the top-left and bottom-right corners of the candidate box.

[0041] Step 5: Fuse the classification results from the Detic model and the CLIP model to improve the classification performance of the candidate boxes. To fuse the classification results from the Detic and CLIP models, the classification probabilities of the Detic model are first transformed using a logit transformation. To prevent the CLIP model from being biased towards the primary objective, the classification scores of the CLIP model are subjected to exponentiation and logit transformation: Where α = 0.5. The two logits are added together to obtain the fused classification score. Finally, the refined classification probability is obtained by using the Sigmoid function:

[0042] The system of this invention mainly includes the following two modules:

[0043] 1. Global Classification Scoring Module:

[0044] CLIP Image Encoder: The CLIP model uses an image encoder to encode the input image, generating global features. The image encoder employs a pre-trained convolutional neural network (CNN), such as ResNet or VisionTransformer, to convert the image into high-dimensional feature vectors. These feature vectors represent the visual information of the image and are used for subsequent similarity calculations.

[0045] CLIP Text Encoder: This text encoder uses the CLIP model to encode category labels and obtain text features for each category. The specific process is as follows: For each category label, such as "airplane," "bicycle," "chair," and "sheep," it is converted into a natural language description, such as "a photo of an object" or "a close-up photo of an object." The CLIP text encoder encodes these descriptions, resulting in multiple different text features for each category. For multiple descriptions of the same category, the average value is taken as the text feature for that category. Multiple text descriptions improve the model's robustness and reduce the impact of uncertainty or bias caused by a single description. For example, while both "a photo of a bicycle" and "a close-up photo of a bicycle" describe bicycles, they have different focuses. By combining these different descriptions, the model can more comprehensively understand category features and reduce errors caused by inaccurate or incomplete descriptions.

[0046] The Similarity Calculation Module calculates the similarity between image features and text features for each category, resulting in a classification score. Similarity calculation typically uses cosine similarity, which measures the degree of similarity between image feature vectors and text feature vectors by calculating the cosine of the angle between them.

[0047] 2. Object Detection Module:

[0048] Existing zero-shot object detectors (OV-Detectors) are used to generate preliminary region proposals from input images, and to classify and regress each candidate region to obtain classification scores and detection boxes.

[0049] The Instance Probability Refinement Module integrates the classification ratings of the zero-shot object detector and the CLIP model to improve the model's generalization ability.

[0050] Table 1 shows the performance on the OV-COCO dataset.

[0051]

[0052] Table 2 shows the performance of the classroom scene motion recognition dataset.

[0053]

Claims

1. A zero-shot object detection method based on deep learning, the method comprising: Step 1: Obtain sample data, which includes images and corresponding labels; Step 2: Use the CLIP text encoder to extract features from the tags to obtain text features. ; Step 3: Use the CLIP image encoder to extract features from the image to obtain visual features. ; Step 4: Calculate text features and visual features Similarity; The similarity is ; , in, The dimension of the feature vector. Represents an image. Indicates the first The first category of text feature vectors One component; Step 5: Establish a zero-shot object detection model, and use the established zero-shot object detection model to initially obtain the classification results and detection boxes of the image; Step 5.1: Establish a zero-shot target detection model; Step 5.2: Preprocess the image. The preprocessed image is represented as follows: ; Step 5.3: Extract feature maps from the image A set of candidate boxes is generated on the feature map using a region proposal network; the set of candidate boxes is represented as... ,in The image represents the first There are a total of candidate boxes. One candidate box; Step 5.4: Input the features of the candidate boxes into the classification layer and calculate the preliminary classification probability of each candidate box belonging to each category; the classification head is a function. Then the image's first The initial classification probabilities of the candidate boxes are: in, It is a vector representing the probability that the candidate box belongs to each category; Step 5.5: Input the features of the candidate boxes into the regression head and calculate the detection box coordinates of each candidate box; the regression head is a function Then the image's first The coordinates of the detection boxes for each candidate box are: ;in, It is a vector representing the coordinates of the top-left and bottom-right corners of the candidate box; Step 6: Fuse the classification results from Step 5 with the similarity results from Step 4, and obtain the final detection result based on the fusion result; Step 6.1: First, perform a logit transformation on the classification probability of the zero-shot object detection model. ; Step 6.2: Perform exponentiation and logit transformation on the similarity obtained in Step 4: ,in The adjustment factor should be set according to the actual situation. Step 6.3: Add the results of Step 6.1 and Step 6.2 to obtain the fused classification score. Finally, the refined classification probability is obtained using the Sigmoid function: .

2. The zero-shot target detection method based on deep learning as described in claim 1, characterized in that, The data requirements for obtaining the dataset in step 1 are as follows: Let the set of all category names in the dataset be... The total number of elements is The set of class names is visible. Each set of invisible classes is Visible classes and invisible classes have no intersection. All categories are exactly divided into visible and invisible categories. The set of text descriptions for the target category is: ,in Indicates the first The first category A type of description.

3. The zero-shot target detection method based on deep learning as described in claim 1, characterized in that, The specific method for step 2 is as follows: Step 2.1: Tag multiple text descriptions of different categories to obtain a tag sequence; Step 2.2: Encode the labeled sequence using the CLIP text encoder, obtaining multiple different text features for each category; assume the text encoder is a function. ,but The text feature vector is: ; Step 2.3: Normalize the obtained text feature vectors to ensure that the norm of each vector is 1. The normalized feature vectors are as follows: ; Step 2.4: For the text features obtained from multiple descriptions of the i-th category, take the average value as the text feature of that category. ; The average eigenvector is Finally, the text features of this category are obtained. After normalization, its norm is ensured to be 1: .

4. The zero-shot target detection method based on deep learning as described in claim 1, characterized in that, The specific method for step 3 is as follows: For images Preprocessing is performed to resize the image to the input size required by the CLIP model. The preprocessed image is represented as follows. The preprocessed image is encoded using the CLIP image encoder to obtain a high-dimensional visual feature vector. Assume the image encoder is a function Then the visual feature vector of the image is: The obtained visual feature vectors are normalized to ensure that the norm of each vector is 1; the normalized visual feature vectors are as follows: .

Images