A novel detector method based on a two-stage detection model

Abstract

This invention discloses a novel detector method based on a two-stage detection model. This invention has certain versatility within two-stage detection models, and the improved detection model can also be used as a standalone detection framework. To address the problem that traditional two-stage detection models often misclassify new targets as easily confused base class targets in classification tasks, this invention adds a contrastive network to the detector to understand intra-class similarities and inter-class differences. By contrasting object encoding, inter-class errors can be reduced. The patent verifies the feasibility and usability of the algorithm through ablation and contrast experiments, thereby demonstrating the feasibility and rationality of the novel detector proposed in this invention.

Description

Technical Field

[0001] This invention relates to the field of object detection and classification in deep learning, and more particularly to a novel detector method based on a two-stage detection model. Background Technology

[0002] Due to the immense success of deep neural networks (DNNs) in image classification and detection tasks, they are now widely accepted as "feature extractors" for almost all computer vision applications, primarily because of their ability to extract features from data effectively. It is well known that training a regular DNN model from scratch requires a large amount of training data. However, in many practical applications, only a few training samples may be available for each class to learn and train the classifier. Compared to conventional training methods, this scenario involves very few new class samples and a large number of base class samples; this is known as the few-shot learning problem.

[0003] Recent research has made significant progress in few-shot learning using DNNs. This has proven applicable to domain-specific tasks such as face recognition and general category classification. However, few works have investigated object detection problems in few-shot learning, including few-shot object localization and object recognition, which can be viewed as a classification task. Currently, meta-learning methods are commonly used, with metric learning being a branch of meta-learning. Two-stage detection models can accurately localize the foreground, but in subsequent classification tasks, new class objects are often classified as easily confused base class objects, which is still detrimental to metric estimation of new class objects. To address the potential confusion between classes, this invention adds a contrastive network to the detector to understand intra-class similarity and inter-class differences. By contrasting object encoding, misclassification between classes can be reduced. Furthermore, traditional two-stage models suffer from misalignment between the original image and feature map; this invention improves the ROI pooling module to an ROI Align module to solve this problem. Summary of the Invention

[0004] This invention mainly improves the detection head of the traditional two-stage detection model (Faster-RCNN) by using a novel detector based on a DML subnet assisted by a comparison network to replace the traditional network framework. The improved detection model can then be used as a standalone detection framework.

[0005] The technical solution adopted in this invention is:

[0006] Step 1: Extract a single feature vector calculated by the network from the backbone and ROI of the two-stage detection model, and input it into the DML embedding module;

[0007] Step 2: The embedding module generates representative classes based on the support image. The method is to input a scalar 1 into an FC layer of size N*K*e and output an N*K*e tensor. N represents the number of new classes and K represents that each new class has K instances. An instance can also be understood as the number of samples in the training set.

[0008] Step 3: Using the embedding vector E obtained in Step 2, the distance d from this embedding vector E to Tij can be calculated. ij (E)=d(E,T ij And by using this distance, we can obtain the probability of the ROI in the j-th mode of class i;

[0009]

[0010] Step 4: Assume that all class distributions have variance σ 2 The framework is an isotropic multivariate Gaussian mixture. Instead of learning the mixture coefficients, it sets the class posterior as the maximum probability across all modalities, as follows:

[0011]

[0012] Where B = i represents class i, and its maximum value is the maximum of all patterns in its mixture model. This conditional probability is an upper bound on the actual class posterior probability;

[0013] Step 5: After obtaining the class posterior, this subnet can estimate the posterior probability of the open background class. This subnet does not model the background probability; instead, it estimates it using the lower bound of the foreground probability, as shown in the following formula.

[0014]

[0015] Step 6: Based on P(B = i|X) and P(α|X) calculated in Step 5, we train our model (DML subnet + backbone) using the sum of two loss functions. The first loss is the cross-entropy (CE) of the standard label of the ROI corresponding to X. The other is to ensure that there is at least a β margin between the distance E to the nearest representative of the correct class and the distance E to the nearest representative of the incorrect class, as follows:

[0016]

[0017] Among them, i * This is the correct class index for the current example, |...| + It is the ReLU function;

[0018] Step 7: Building on steps 1 to 6, we introduce a contrast branch at the ROI head, parallel to the DML subnet and regression branch;

[0019] Step 8: The contrast branch applies a 1-layer multilayer perceptron (MLP) head with negative logical cost to convert ROI features into contrastive features. Default D c =128;

[0020] Step 9: After step 8, this branch measures similarity scores on the ROI features encoded in the MLP header and the optimized contrast target to maximize consistency between the same category and improve the difference between different categories.

[0021] The ROI extraction network in step 1 is the ROI Align network.

[0022] In step 2, during training, the embedding module introduces gradients into the weights of the FC layer and learns representative classes.

[0023] Step 2 consists of two fully connected (FC) layers with a width of 1021, featuring BN and ReLU. The last layer is an FC layer with linear activation and a bandwidth of 256, followed by L2 normalization.

[0024] The contrast network in steps 7 to 9 can be adapted to end-to-end training of the two-stage detection model (Faster-RCNN). It requires an additional training on top of the basic training, using a dataset of new classes and randomly sampled base classes to fine-tune the original detector. The feature extractor is frozen during fine-tuning, while the ROI extractor is supervised by the contrast class.

[0025] Compared with the prior art, the beneficial effects of the present invention are:

[0026] (1) It can effectively solve the classification problem that occurs in the two-stage detection module. By adding a comparison network to the detector, it can understand the similarity within the class and the difference between the classes. By comparing the object encoding, it can reduce the misjudgment between classes.

[0027] (2) Traditional two-stage modules can cause misalignment between the original image and the feature image. This problem can be solved by improving ROI pooling to ROI Align. Attached Figure Description

[0028] Figure 1 This is the overall algorithm framework of the present invention.

[0029] Figure 2 This is the DML subnet framework of the present invention.

[0030] Figure 3 This invention presents a two-stage detection framework based on DML subnets.

[0031] Figure 4 The results are visualizations of the network loss after comparing the network with and without the present invention.

[0032] Figure 5 This invention provides a comparison network framework based on object encoding.

[0033] Figure 6 The loss terms are used to train the two-stage detection model.

[0034] Figure 7 The mAP value is the baseline comparison experiment of the two-stage detection model based on the DML subnet.

[0035] Figure 8 mAP values ​​for baseline comparison experiments (visible class) of a two-stage detection model based on DML subnets.

[0036] Figure 9 This is a performance comparison of object-based encoding contrast networks. Detailed Implementation

[0037] The invention will be further described below with reference to the accompanying drawings.

[0038] First, the base detector is trained using the standard two-stage detection model (Faster-RCNN) loss, with a binary cross-entropy loss L. rpn To obtain the foreground proposal from the anchor, the cross-entropy loss L cls Used for bounding box classification, smooth loss L reg Used for bounding box regression. After fine-tuning to transform the data into new data, this paper finds that the contrastive loss can be added to the main Faster-RCNN loss in a multi-task manner without compromising training stability, as follows:

[0039] L = L rpn +L cls +L reg +αL cpe

[0040] The initial value of α is set to 0.5, which is used to balance the proportion of loss. The loss function is calculated as follows: Figure 6 As shown.

[0041] To demonstrate the effectiveness of the DML subnet, the proposed two-stage detection model based on the DML subnet is compared with three baselines. First, the first baseline, called baseline-FR, fine-tunes the standard detector network on only a few N*K available samples of N new classes in each N-way, K-shot test set. Specifically, the experiment fine-tunes the linear decision layer in the classifier head of the traditional two-stage detection model (Faster-RCNN) detector, using it as the experimental backbone. Second, the second baseline, called baseline-DML, connects the DML subnet without an embedding module to the Faster-RCNN detector, using two additional fully connected (FC) layers as the embedding module. The Faster-RCNN detector in this baseline does not optimize for DML subnet classification. Finally, the third baseline, called baseline-DML-embedding, trains the DML subnet embedding module separately from the detector during offline training. The embedding module is trained on the sampled foreground and background ROIs using triple loss. Similar baseline performance was also obtained when training the embedding module with a non-typical network.

[0042] All baselines will be pre-trained on the same dataset as experimental models and tested on the same test set.

[0043] To train the model, the experiments used the first 100 classes (primarily animals and birds) from ImageNet LOC. For testing, the remaining 214 animal and bird classes were used to ensure that the training and testing categories expanded to the same conceptual domain. For the experimental model and all DML baselines, in each set, the detected class set was reset to N new class sets by replacing the representative class in the DML subnet with embedding vectors computed based on the ROIs corresponding to the training objects of that class. The RPN returns 2000 ROIs for each image by checking which ROIs meet the requirement of IOU ≥ 0.7 for the training object bounding boxes, and selects the appropriate ROIs from these. In the proposed method, the embedding layer and the backbone network are jointly optimized for use with the class posterior output by the DML subnet. Compared to the baseline, this method achieves an improvement in accuracy, such as... Figure 7 As shown, all results represent mAP%.

[0044] Figure 7 The report presents an evaluation of our method and the baseline for this set of unknown classes. The mAP% value was calculated using a 5-way detection task, where mAP was calculated by collecting and evaluating all detection boxes detected in all 500 sets of tests, with 50 query images per set.

[0045] To test the interdependence of the embedding module on specific representations in the vector space, this invention conducts another experiment on the class used in training (the visible class). The test results are as follows: Figure 8 As shown. The experiment was repeated twice: once using representatives extracted from the training objects of each class, and once using the original trained representatives. Because the experiment learned K=5 representative data points for each class during training, the results of the second test are listed in 5 instances. It can be seen that the trained representatives perform better than the randomly embedded class samples, further highlighting the benefits of joint training; moreover, the performance drop from the trained representatives to random class members is not as large, indicating that the learned embedding module is robust to changes in representatives and is likely to perform well on new, unseen categories.

[0046] To demonstrate the effectiveness of the comparison network module, it was validated using the PASCAL VOC and COCO datasets. Three random batches of new classes from the PASCAL VOC dataset were used for testing, and the results are as follows: Figure 9 As shown. The object encoding proposed in this invention outperforms all existing methods in any instance and all batches. The effectiveness of the object encoding-based contrastive network proposed in this invention is fully demonstrated.

Claims

1. A novel detector method based on a two-stage detection model, characterized in that, Includes the following steps: Step 1: Obtain the original image to be detected, input it into a two-stage detection model consisting of a backbone network and an ROI extraction network, where the ROI extraction network is the ROI Align network, and input the calculated single feature vector into the DML embedding module; Step 2: The embedding module generates representative classes based on the support image. The method is to input a scalar 1 into an FC layer of size N*K*e and output an N*K*e tensor. N represents the number of new classes and K represents that each new class has K instances. An instance can also be understood as the number of samples in the training set. Step 3: Using the embedding vector E obtained in Step 2, this embedding vector can be calculated. arrive distance And by using this distance, we can obtain the probability of the ROI in the j-th mode of class i; ; Step 4: Assume all class distributions are mixtures of isotropic multivariate Gaussians with variance The DML embedding module does not learn mixture coefficients, but instead sets the class posterior to be the maximum of the probabilities under each modality, as follows: in It represents class i and its maximum value is the maximum of all modes in its mixture model; this conditional probability is an upper bound of the actual class posterior probability. Step 5: After obtaining the posterior, the DML embedding module can estimate the posterior probability of the open background class; the DML embedding module does not model the background probability, but rather estimates it through a lower bound on the foreground probability, as follows ; Step 6: Based on the calculations in Step 5 and We train our model using the sum of two loss functions: a DML subnet and a backbone. The first loss is the cross-entropy (CE) of the standard label corresponding to the ROI of X; the other is to ensure that there is at least a certain distance between the distance E to the nearest representative of the correct class and the distance E to the nearest representative of the incorrect class. The margin is as follows: in, This is the correct class index for the current example. It is the ReLU function; Step 7: Building on steps 1 to 6, we introduce a contrast branch at the ROI head, parallel to the DML subnet and regression branch; Step 8: Contrast branch applies a 1 -layer multi-layer perceptron (MLP) head with negative logic cost to convert ROI features to contrast features default = 128; Step 9: After step 8, the comparison branch measures similarity scores on the ROI features encoded in the MLP header and the optimized comparison target to maximize consistency between the same category and improve the difference between different categories.

2. The method of claim 1, wherein, In step 2, during training, the embedding module introduces gradients into the weights of the FC layer and learns representative classes.

3. The method of claim 1, wherein, Step 2 consists of two fully connected (FC) layers, each 1021 pixels wide, with Batch Normalization (BN) and ReLU. The last layer is an FC layer with linear activation and a bandwidth of 256, followed by... Normalization.

4. The method of claim 1, wherein, The contrast network in steps 7 to 9 can be adapted to the end-to-end training of the two-stage detection model Faster-RCNN. It requires an additional training on top of the basic training, using a dataset of new classes and random sampling of the base classes to fine-tune the original detector. The feature extractor is frozen during fine-tuning, while the ROI extractor is supervised by the contrast class.

Images