Zero sample target detection method based on deep learning

Abstract

The invention discloses a zero sample target detection method based on deep learning, and relates to the field of computer vision and machine learning. According to the method, on the premise of not increasing training expenditure, the detection precision of the base class and the generalization ability of the new class can be improved on the basis of an existing open vocabulary target detection model by only adding a simple module. Coding and feature extraction are performed on the image and text description, and the classification probability of each proposal in the image is refined by using a global classification scoring module, so that the accuracy and generalization ability of zero sample target detection are improved. The classification probability of each region candidate obtained by a zero sample target detection classifier is refined by using the classification probability of the CLIP model to the image, so that the detection accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and machine learning, and in particular to a zero-sample target detection system based on deep learning. Background Art

[0002] With the continuous development of deep learning technology, object detection has made significant progress. However, traditional object detection methods often require a large amount of labeled data for training when dealing with new categories of targets, which limits their flexibility in practical applications. To solve this problem, Zero-Shot Object Detection (ZSD) came into being, aiming to detect new categories of targets that did not appear in the training set.

[0003] Existing zero-shot object detection methods are mainly divided into embedding learning-based methods and generative adversarial-based methods. Zhou X, Girdhar R, Joulin A, et al. Detecting twenty-thousand classes using image-level supervision [C] / / European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022: 350-368 The image-level label is applied to the region with the largest size in the region proposal. This method does not require a complex prediction-based label assignment mechanism, and directly uses image-level annotations to constrain the object detection classifier. The detector trained by Detic under the constraint of image-level annotations can generalize to new datasets without fine-tuning without additional fine-tuning. However, this method relies on the maximum size region in the region candidate, which may lead to reduced accuracy when dealing with smaller targets. Lin J, ShenY, Wang B, et al. Weakly supervised open-vocabulary object detection[C] / / Proceedings of the AAAI Conference on Artificial Intelligence. 2024, 38(4): 3404-3412 Through three key strategies of dataset-level feature adaptation, image-level salient object localization, and region-level visual-language alignment, this paper realizes the detection and localization of open vocabulary concepts on diverse datasets with only image-level annotations. When facing large-scale datasets, the feature adaptation and alignment process may become a performance bottleneck, affecting the detection speed. Lin C, Jiang Y, Qu L, et al. Generative Region-Language Pretraining for Open-Ended Object Detection[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition.2024:13958-13968. The framework redefines the object detection problem as a generation problem and proposes an end-to-end learning framework consisting of a visual object detector and a language model. It directly generates object names in a free form and enriches label diversity through pseudo-label methods, thereby achieving open vocabulary object detection without pre-defining categories in the reasoning stage.Although the pseudo labels in this method increase the diversity of labels, their quality is difficult to guarantee, which will affect the detection effect. Pham C, Vu T, Nguyen K. Lp-ovod: Open-vocabulary object detection by linear probing [C] / / Proceedings of the IEEE / CVF Winter Conference on Applications of Computer Vision. 2024: 779-788. The sigmoid linear classifier is trained on the pseudo labels retrieved from the new relevant region proposals to discard low-quality boxes, thereby improving the detection performance of new categories. Although discarding low-quality boxes in this method can improve the detection performance, the detection accuracy of new categories still needs to be improved. JeongJ,Park G,Yoo J,et al.ProxyDet:Synthesizing Proxy Novel Classes via ClasswiseMixup for Open-Vocabulary Object Detection[C] / / Proceedings of the AAAIConference on Artificial Intelligence.2024,38(3):2462-2470. We propose to synthesize proxy new categories that are similar to new categories through linear mixing between a pair of base categories, and train our detector using these synthesized approximate new categories to reserve positions for unseen classes in the feature space.

[0004] The invention can improve the detection accuracy of the base class and the generalization ability of the new class on the basis of the existing open vocabulary object detection model by simply adding a simple module without increasing the training cost. Summary of the invention

[0005] Write a paragraph This invention solves the problem that the classification head of the prior art zero-shot target detection method has insufficient generalization ability for new classes when processing zero-shot target detection tasks. By encoding and extracting features from images and text descriptions, the classification probability of each proposal in the image is refined using a global classification scoring module, thereby improving the accuracy and generalization ability of zero-shot target detection. By using the CLIP model to classify the image, the classification probability of each region candidate obtained by the zero-shot target detection classifier is refined, thereby improving the accuracy of detection.

[0006] The technical solution of the present invention is: a zero-sample target detection method based on deep learning, the method comprising:

[0007] Step 1: Obtain sample data, which includes images and corresponding labels;

[0008] Step 2: Use CLIP text encoder to extract features from labels and obtain text features t i ′;

[0009] Step 3: Use CLIP image encoder to extract features from the image and obtain visual features v′;

[0010] Step 4: Calculate text feature t i ′ and the similarity of the visual feature v′;

[0011] Step 5: Establish a zero-shot target detection model, and use the established zero-shot target detection model to preliminarily obtain the classification results and detection frames of the image;

[0012] Step 6: Fuse the classification result of step 5 and the similarity of step 4, and obtain the final detection result based on the fusion result.

[0013] Furthermore, the data requirements for step 1 are as follows: Let the set of all category names of the data set be The total number of elements is N, and the name set of visible classes is Each set of invisible classes is There is no intersection between visible and invisible classes All categories are exactly divided into visible and invisible categories The text description set of the target category is {d ij} i＝1,2,…N；j＝1,2,…m , where d ij Represents the jth description of the ith category.

[0014] Furthermore, the specific method of step 2 is:

[0015] Step 2.1: Tokenize multiple text descriptions of different categories to obtain a token sequence;

[0016] Step 2.2: Encode the tag sequence through the CLIP text encoder, and obtain multiple different text features for each category; assume that the text encoder is function E t , then d ij The text feature vector is: e ij =E t (d ij );

[0017] Step 2.3: Normalize the obtained text feature vector to ensure that the norm of each vector is 1. The normalized feature vector is:

[0018] Step 2.4: Take the average value of the text features obtained from multiple descriptions of the i-th category as the text feature t of this category. i The average eigenvector is Finally, the text feature t of this category is obtained i After normalization to ensure its norm is 1:

[0019] Furthermore, the specific method of step 3 is:

[0020] Preprocess the image I and adjust the image size to the input size required by the CLIP model. The preprocessed image is represented as I′. Encode the preprocessed image through the CLIP image encoder to obtain a high-dimensional visual feature vector. Assume that the image encoder is a function E v , then the visual feature vector of the image is: v = E v (I′); the obtained visual feature vector is normalized to ensure that the norm of each vector is 1; the normalized visual feature vector is:

[0021] Furthermore, the similarity of step 4 is S i ;

[0022] Among them, d is the dimension of the feature vector, v j ′ represents the image, t′ ij Represents the jth component of the text feature vector of the i-th category.

[0023] Furthermore, the specific method of step 5 is:

[0024] Step 5.1: Build a zero-shot object detection model;

[0025] Step 5.2: Preprocess the image, and the preprocessed image is represented as X′;

[0026] Step 5.3: Extract the feature map F = E of the image d (X′); Use the region proposal network to generate a series of candidate boxes on the feature map; The set of candidate boxes is represented as Among them B i Represents the i-th candidate box of the image, there are n candidate boxes in total;

[0027] Step 5.4: Input the features of the candidate boxes into the classification layer and calculate the preliminary classification probability of each candidate box belonging to each category; the classification head is function C, then the preliminary classification probability of the i-th candidate box of the image is: p i =C(F,B i ) Among them, p iIs a vector that indicates the probability that the candidate box belongs to each category;

[0028] Step 5.5: Input the features of the candidate boxes into the regression head and calculate the detection box coordinates of each candidate box; the regression head is the function R, then the detection box coordinates of the i-th candidate box in the image are: b i =R(F,B i ), where b i Is a vector representing the coordinates of the upper left corner and the lower right corner of the candidate box.

[0029] Furthermore, the specific method of step 6 is:

[0030] Step 6.1: First, perform logit transformation on the classification probability of the zero-shot target detection model

[0031] Step 6.2: Perform power operation and logit transformation on the similarity obtained in step 4: Where α is the adjustment factor, which is set according to the actual situation;

[0032] Step 6.3: Add the results of step 6.1 and step 6.2 to get the fused classification score Finally, the final refined classification probability is obtained through the Sigmoid function:

[0033] Compared with the existing technology, the present invention has the following significant advantages and positive effects: by using the CLIP model to extract global features of the image, and combining it with the existing zero-sample target detection model, the classification probability of each proposal is refined, which can effectively handle new categories that do not appear in the training set, thereby improving the detection performance of new categories. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is the overall framework diagram of the strategy of the present invention. DETAILED DESCRIPTION

[0035] The present invention proposes a zero-shot target detection method based on the CLIP model and the existing zero-shot target detection model. By encoding and extracting features from images and text descriptions, the classification probability of each proposal in the image is refined using a global classification scoring module, thereby improving the accuracy and generalization ability of zero-shot target detection. By using the classification probability of the image using the CLIP model, the classification probability of each region candidate obtained by the zero-shot target detection classifier is refined, thereby improving the accuracy of detection. The specific implementation is as follows:

[0036] Step 1: Use CLIP's text encoder to extract text features for each category. The set of all category names in the dataset is The total number of elements is N, and the name set of visible classes is Each set of invisible classes is There is no intersection between the visible class and the invisible class All categories are exactly divided into visible and invisible categories Input the text describing the target category into the system. The text description can include m different description forms, such as "a photo of a{object}", "a close-up photo of a{object}", etc. Tokenize multiple text descriptions of different categories to convert natural language text into a sequence of tags that the model can process. Assume that the text description set of the target category is

[0037] {d ij} i＝1,2,…N；j＝1,2,…m , where d ij represents the jth description of the i-th category. The tag sequence is encoded by the CLIP text encoder, and each category obtains multiple different text features. Assume that the text encoder is a function E t , then d ij The text feature vector is: e ij =E t (d ij ). The obtained text feature vector is normalized to ensure that the norm of each vector is 1. The normalized feature vector is: The text features obtained from multiple descriptions of the i-th category are averaged as the text feature t of this category. i The average eigenvector is Finally, the text feature t of this category is obtained i After normalization to ensure its norm is 1:

[0038] Step 2: Use CLIP's image encoder to extract the global features of the image. Input system. Preprocess the image and adjust the image size to the input size 224*224 required by the CLIP model. The preprocessed image is represented as I′. Encode the preprocessed image through the CLIP image encoder to obtain a high-dimensional visual feature vector. Assume that the image encoder is function E v , then the visual feature vector of the image is: v = E v (I′). The obtained visual feature vector is normalized to ensure that the norm of each vector is 1. The normalized visual feature vector is:

[0039] Step 3: Calculate the cosine similarity between visual features and text features. The normalized text feature vector t of each category has been obtained through the CLIP text encoder. i ′, where i represents the category number. The normalized visual feature vector v′ of each image has been obtained through the CLIP visual encoder. The cosine similarity S between the input image and the i-th category is calculated. i . Where d is the dimension of the feature vector v′ j and t′ ij Represent the jth component of the image and text feature vector of the i-th category respectively.

[0040] Step 4: Use the existing zero-shot target detection model to obtain the preliminary classification results and detection boxes of the candidate regions. Input the image into the existing zero-shot target detection model (here, the Detic model is used as an example) to preprocess the image. The preprocessed image is represented as X′. Input the preprocessed image into the backbone network E of the Detic model. d In the above example, we extract the feature map F = E of the image. d (X′). A series of candidate boxes are generated on the feature map using the region proposal network. The set of candidate boxes is represented as Among them B i represents the i-th candidate box of the image, and there are n candidate boxes in total. The features of the candidate boxes are input into the classification layer to calculate the preliminary classification probability of each candidate box belonging to each category. Assuming that the classification head is function C, the preliminary classification probability of the i-th candidate box of the image is: p i =C(F,B i ) Among them, p i is a vector that represents the probability that the candidate box belongs to each category. The features of the candidate box are input into the regression head to calculate the detection box coordinates of each candidate box. Assuming that the regression head is a function R, the detection box coordinates of the i-th candidate box of the image are: i =R(F,B i ). Among them, b i Is a vector representing the coordinates of the upper left corner and the lower right corner of the candidate box.

[0041] Step 5: Fuse the classification results obtained by the Detic model with the classification results obtained by CLIP to improve the classification performance of the candidate box. In order to fuse the classification results of the Detic model and the CLIP model, first perform a logit transformation on the classification probability of the Detic model. To prevent the CLIP model from being biased towards the primary objective, the classification scores of the CLIP model were exponentially transformed and logit-transformed: Where α = 0.5. Add the two logits to get the fused classification score Finally, the final refined classification probability is obtained through the Sigmoid function:

[0042] The system of the present invention mainly includes the following two modules:

[0043] 1. Global Classification Scoring Module:

[0044] CLIP Image Encoder: The image encoder in the CLIP model is used to encode the input image and generate the global features of the image. The image encoder uses a pre-trained convolutional neural network (CNN), such as ResNet or VisionTransformer, to convert the image into a high-dimensional feature vector. These feature vectors represent the visual information of the image and are used for subsequent similarity calculations.

[0045] CLIP text encoder: The text encoder of the CLIP model is used to encode the category labels to obtain the text features of each category. The specific process is as follows: For each category label, such as "airplane", "bicycle", "chair", "sheep", etc., it is converted into a natural language description form, such as "a photo of a{object}", "a close-upphoto of a{object}", etc. These descriptions are encoded by the CLIP text encoder, and multiple different text features are obtained for each category. For multiple descriptions of the same category, the average value is taken as the text feature of the category. Multiple text descriptions can improve the robustness of the model and reduce the impact of uncertainty or bias on a single description. For example, although "a photo ofa bicycle" and "a close-up photo of a bicycle" both describe a bicycle, they focus on different aspects. By combining these different descriptions, the model can understand the category features more comprehensively and reduce the errors caused by inaccurate or incomplete descriptions.

[0046] Similarity Calculation Module: Calculates the similarity between the image features and the text features of each category to obtain the classification score. The similarity calculation usually uses cosine similarity, that is, by calculating the cosine value of the angle between the image feature vector and the text feature vector to measure their similarity.

[0047] 2. Object Detection Module:

[0048] Existing zero-shot object detector (OV-Detector): used to generate preliminary region proposals for the input image, and classify and regress each candidate region to obtain classification scores and detection boxes.

[0049] Instance Probability Refinement Module: It combines the classification scores of the zero-shot object detector with those of the CLIP model to improve the generalization ability of the model classification.

[0050] Table 1 Performance results on the OV-COCO dataset

[0051]

[0052] Table 2 Performance results on classroom character action recognition dataset

[0053]

Claims

1. A zero-sample target detection method based on deep learning, the method comprising: Step 1: Obtain sample data, which includes images and corresponding labels; Step 2: Use CLIP text encoder to extract features from labels and obtain text features t i ′; Step 3: Use CLIP image encoder to extract features from the image and obtain visual features v ′ ; Step 4: Calculate text feature t i ′ and the similarity of the visual feature v′; Step 5: Establish a zero-shot target detection model, and use the established zero-shot target detection model to preliminarily obtain the classification results and detection frames of the image; Step 6: Fuse the classification result of step 5 and the similarity of step 4, and obtain the final detection result based on the fusion result.

2. The method for zero-sample target detection based on deep learning according to claim 1, characterized in that: The data requirements obtained in step 1 are: Let the set of all category names of the data set be The total number of elements is N, and the name set of visible classes is Each set of invisible classes is There is no intersection between the visible class and the invisible class All categories are exactly divided into visible and invisible categories The text description set of the target category is {d ij } i＝1,2,…N；j＝1,2,…m , where d ij Represents the jth description of the ith category.

3. The method for zero-sample target detection based on deep learning according to claim 1, characterized in that: The specific method of step 2 is: Step 2.1: Tokenize multiple text descriptions of different categories to obtain a token sequence; Step 2.2: Encode the tag sequence through the CLIP text encoder, and obtain multiple different text features for each category; assume that the text encoder is function E t , then d ij The text feature vector is: e ij =E t (d ij ); Step 2.3: Normalize the obtained text feature vector to ensure that the norm of each vector is 1. The normalized feature vector is: Step 2.4: Take the average value of the text features obtained from multiple descriptions of the i-th category as the text feature t of this category. i The average eigenvector is Finally, the text feature t of this category is obtained i After normalization to ensure its norm is 1:

4. The method for zero-sample target detection based on deep learning according to claim 1, characterized in that: The specific method of step 3 is: Preprocess the image I and adjust the image size to the input size required by the CLIP model. The preprocessed image is represented as I′. Encode the preprocessed image through the CLIP image encoder to obtain a high-dimensional visual feature vector. Assume that the image encoder is a function E v , then the visual feature vector of the image is: v = E v (I′); the obtained visual feature vector is normalized to ensure that the norm of each vector is 1; the normalized visual feature vector is:

5. The method for zero-sample target detection based on deep learning according to claim 1, characterized in that: The similarity of step 4 is S i ; Where d is the dimension of the feature vector, v′ j represents the image, t′ ij Represents the jth component of the text feature vector of the i-th category.

6. The method for zero-sample target detection based on deep learning according to claim 1, characterized in that: The specific method of step 5 is: Step 5.1: Build a zero-shot object detection model; Step 5.2: Preprocess the image, and the preprocessed image is represented as X′; Step 5.3: Extract the feature map F = E of the image d (X′); Use the region proposal network to generate a series of candidate boxes on the feature map; The set of candidate boxes is represented as Among them B i Represents the i-th candidate box of the image, there are n candidate boxes in total; Step 5.4: Input the features of the candidate boxes into the classification layer and calculate the preliminary classification probability of each candidate box belonging to each category; The classification head is function C, then the initial classification probability of the i-th candidate box of the image is: p i =C(F,B i ) Among them, p i Is a vector that indicates the probability that the candidate box belongs to each category; Step 5.5: Input the features of the candidate boxes into the regression head and calculate the detection box coordinates of each candidate box; the regression head is the function R, then the detection box coordinates of the i-th candidate box in the image are: b i =R(F,B i ), where b i Is a vector representing the coordinates of the upper left corner and the lower right corner of the candidate box.

7. The method for zero-sample target detection based on deep learning according to claim 1, characterized in that: The specific method of step 6 is: Step 6.1: First, perform logit transformation on the classification probability of the zero-shot target detection model Step 6.2: Perform power operation and logit transformation on the similarity obtained in step 4: Where α is the adjustment factor, which is set according to the actual situation; Step 6.3: Add the results of step 6.1 and step 6.2 to get the fused classification score Finally, the final refined classification probability is obtained through the Sigmoid function:

Images