Continuous Few-Shot Object Detection Method Based on Category Registration Mechanism and Regional Contrastive Learning
Through the category registration mechanism and regional comparison learning method, the problem of the object detection system forgetting the old category target after fine-tuning is solved, and effective detection of new category targets is achieved with few samples is achieved, the detection ability of the old category targets is maintained, and the detection performance and flexibility of the system are improved.
Patent Information
- Application Number
- CN202311231642.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-22
- Publication Date
- 2025-08-05
- Estimated Expiration
- 2043-09-22
AI Technical Summary
The existing object detection system is prone to forget the detection capabilities of known category targets after fine-tuning, and the effect is not good when detecting new category targets with few samples, especially in complex application scenarios, which is difficult to achieve a balance between continuous learning and small sample learning.
The class registration mechanism is used to isolate the parameters of the new class and the old class, calculate the classifier output using cosine similarity, and combine regional comparison learning to mine a large amount of labeled sample information during the pre-training process, and feature extraction and comparison learning are performed through the RPN network of the Faster-RCNN framework to ensure that the old class object detection capability is not lost.
The ability to continuously learn new category target detection under a small number of labeled samples is realized, while maintaining the detection ability of old category targets, improving the performance and system generalization ability of detecting new category targets.
Smart Images
Figure CN117292112B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of deep learning and object detection, and designs an Incremental Few-Shot Object Detection (iFSOD) algorithm based on category registration mechanism and region contrast learning. Background Art
[0002] Deep learning technology has achieved remarkable results in object detection tasks and is widely used in our daily lives. Conventional deep neural networks for object detection require training with large amounts of manually annotated image data, which limits their performance in some complex application scenarios. An ideal object detection system should be able to acquire the ability to detect new object categories through training with a small number of annotated examples while maintaining the ability to detect objects of known categories. This means it should possess both continuous learning and few-shot learning capabilities.
[0003] Few-Shot Object Detection (FSOD) methods, proposed to address the problem of having only a few labeled examples for some categories, typically first pre-train on a training set of base categories with a large amount of labeled examples to learn general object detection knowledge. They then fine-tune the system on novel categories with only a small amount of labeled examples. Incorporating the previously learned general knowledge allows the system to effectively detect novel objects. However, after fine-tuning, these methods tend to forget the base category knowledge and lose their ability to detect base-category objects. To address this issue, some generalized few-shot object detection (gFSOD) methods incorporate a balanced mix of labeled examples from some base categories into the training process during fine-tuning, preserving the system's ability to detect base-category objects. Furthermore, considering that in some application scenarios, for reasons such as user privacy or limited system storage capacity, it is not possible to store examples from previous categories, the system can still detect both base and novel categories after fine-tuning, and can continuously improve its ability to detect new categories through fine-tuning. A method to address this issue is the continuous few-shot object detection algorithm.
[0004] This paper proposes a continuous few-shot object detection method based on a class registration mechanism and region contrastive learning. A class registration mechanism is designed for the classifier, isolating parameters of new and old classes. Cosine similarity is used to calculate the classifier output, ensuring stable classifier knowledge updates and learning the ability to detect new classes of objects without sacrificing the ability to detect old classes. Region contrastive learning is introduced during pre-training to exploit the potential information in a large number of annotated samples. This results in a more generalizable object detection feature extractor, improving the detection capabilities of new classes of objects. Summary of the Invention
[0005] The present invention proposes a continuous few-sample target detection algorithm based on category registration mechanism and regional contrast learning, which enables the target detection system to continuously learn new category target detection based on a small number of labeled samples in some complex application scenarios, update system knowledge, and not lose the ability to detect old category targets.
[0006] The present invention adopts the typical target detection algorithm Faster-RCNN as the system network. Faster-RCNN and some other target detection algorithms usually use a fully connected layer as a regional target classifier, and the network before the classifier input is regarded as a feature extractor. Due to the complex nonlinear structure of the feature extractor, it is very easy to produce overfitting when trained with a small number of labeled samples, resulting in poor detection effect of the test set. In order to alleviate this problem, in the present invention, the feature extractor only updates parameters during pre-training, and the feature extractor network weights will no longer change in the subsequent process. Therefore, the technical solution of the present invention is a continuous few-sample target detection method based on category registration mechanism and regional contrast learning, which includes:
[0007] Step 1: Select a suitable object detection dataset and divide all categories of the dataset into base classes and novel classes in proportion. All training samples of the base class are retained for pre-training, and a small number of samples are randomly sampled from the training set for each novel class as training samples for fine-tuning.
[0008] Step 2: Initialize the system network, which includes a feature extractor, a classifier, and a regressor. The classifier and regressor are initialized with the same number of weight vectors based on the number of categories in pre-training.
[0009] The output of the classifier is passed through the softmax activation layer and then subjected to the cross entropy loss, as follows:
[0010]
[0011] Among them, l i,j represents the j-th classifier output value of the i-th sample, gt represents the label category, bg represents the background category, N o represents the total number of samples;
[0012]
[0013] Among them, α represents the scale factor, and the cosine similarity between the feature vector output by the feature extractor and the classifier weight vector is used as the output of the classifier, which is expressed as:
[0014]
[0015] Among them, α is the same as the scale factor in the previous formula, r i Represents the feature extractor output feature vector, w j is the classifier weight vector;
[0016] Step 3: Perform pre-training. The steps for each iteration are as follows:
[0017] Step 3.1: Randomly sample data from the training set, each sample contains image x, box category y gt and box position y loc , perform four augmentation steps of random cropping, random flipping, padding, and size transformation on each image in the data to obtain two views v o =A o (x) and v t =A t (x), the annotation information is also transformed with the above transformation to obtain the annotation corresponding to the view and The two views are used as the input of the neural network, and the forward propagation processes of the two views are recorded as the online branch and the target branch respectively, where the target branch does not calculate the gradient;
[0018] Step 3.2: Input the view into the backbone network to extract features and obtain feature maps; the feature maps of the online branches are input into the Region Proposal Network (RPN) of Faster-RCNN to calculate the RPN loss and the target branch proposal box p o Proposal boxes representing online branches;
[0019] Step 3.3: Add a random offset to the coordinates of the proposal box position as augmented data. The proposal box of the online branch is calculated based on the information in the augmentation operation and corresponds to the position of the view in the target branch as the proposal box of the target branch. A random offset within ±20% is added to the positions of the proposal boxes of the two branches, and proposal box pairs with an intersection over union (IoU) less than 0.6 are screened out.
[0020] Step 3.4: The proposal frames of each branch are pooled through RoI-Align to obtain the feature map f of the region of interest (RoI) corresponding to the proposal frame o and f t , the feature map is input into the RoI feature extractor F, the online branch calculates the category feature vector and position feature vector corresponding to each proposed region, and the target branch only calculates the category feature vector;
[0021] Step 3.5: The feature vectors of the online branch are respectively calculated with the weight vectors of each class in the classifier to obtain the output of the classifier. The classification loss is calculated by softmax activation and cross entropy loss. The position feature vector is input into the regressor to calculate the position regression loss.
[0022] Step 3.6: The category feature vectors of each branch are input into a multi-layer perceptron to obtain the contrast space vector. The contrast space vector of the online branch is input into the predictor to obtain the prediction vector. The prediction vector and the contrast space vector of the corresponding target branch are used to calculate the contrast loss L. ctr ;
[0023]
[0024] Where <·,·> is the cosine similarity, R(·,·) is the abbreviation of the mapping G(F(RoIAlign(·,·))), and It represents the proposed box set pair that is finally involved in contrastive learning through augmentation and screening. and represents the corresponding proposal box in the set;
[0025] Step 3.7: Add the losses and backpropagate to update the neural network weights;
[0026] L=L rpn +L cls +L reg +λL ctr
[0027] Among them, L rpn , L cls and L reg They represent the RPN loss, RoI classification loss and position regression loss of Faster-RCNN network training respectively, and λ represents the contrast loss scaling coefficient;
[0028] Step 4: Enter a new incremental phase. Based on the neural network trained in the previous phase, fix the existing weights and do not update them in the future. Then initialize the same number of weight vectors according to the number of categories to be learned in this phase.
[0029] Step 5: Fine-tune the neural network. During fine-tuning, only the classifier calculation method is the same as step 3.5 during system network training, and the rest of the methods are trained normally.
[0030] Step 6: If new categories need to be learned, loop back to step 4; otherwise, end the system training process.
[0031] Beneficial effects:
[0032] The proposed continuous few-shot object detection algorithm based on a category registration mechanism and region contrastive learning utilizes the classifier's category registration mechanism to enable the object detection system to detect new classes of objects based on a small number of labeled samples without sacrificing the system's previously learned ability to detect objects of older classes. The proposed region contrastive learning algorithm fully exploits training sample information by comparing the regions corresponding to proposal boxes calculated by the RPN network, improving feature versatility and enhancing the object detection system's pre-training effectiveness on a large number of labeled base class samples, thereby improving the performance of detecting new classes of objects. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] Figure 1 Schematic diagram of the target detection classifier under the category registration mechanism of the present invention.
[0034] Figure 2 This is a schematic diagram of pre-training under regional contrast learning enhancement of the present invention.
[0035] Figure 3 It is the overall process framework diagram of the present invention. DETAILED DESCRIPTION
[0036] The present invention designs a category registration mechanism based on the fully connected classifier; the weight vectors corresponding to different categories in the classifier are separated, and a corresponding number of weight vectors are registered for the classifier in each training stage according to the number of categories to be learned; the weight vectors of the old class inherit the previous stage and are no longer updated in the new training stage, and only the newly registered weight vectors are updated; the output of the classifier is passed through the softmax activation layer and then subjected to the cross entropy loss, as shown in the following formula:
[0037]
[0038] where l i,j represents the j-th classifier output value for the i-th sample, gt represents the label class, and bg represents the background class. When the feature extractor weights and the classifier's old class weight vector remain unchanged, the distribution of the classifier's old class output remains unchanged. Since the training data only contains new class labeled samples, as the loss decreases during training, the new class output continues to increase. As training continues, the mean new class output will significantly exceed the old class output, suppressing the system's prediction probability for the old class, thereby reducing the system's ability to detect old class targets.
[0039] In order to solve the above problem, the present invention adopts the cosine similarity between the feature vector output by the feature extractor and the classifier weight vector as the output of the classifier, which is expressed as
[0040]
[0041] Where α is the scale factor. Compared with the output calculated by the ordinary fully connected layer classifier, the output value calculated using cosine similarity satisfies -α <l i,j <α, the above problem does not exist. In this calculation method, the weight vector can be regarded as the prototype of the category, and the classifier can be regarded as a nearest neighbor classification method based on cosine distance during the inference process. The background class is a special category in the classification of the detection task, which contains a variety of different semantics. Even some potential targets are regarded as background in the labeled data in the early training stage. Therefore, the category registration mechanism of the present invention will not register the weight vector for the background class. In the actual training process, the background class output of the classifier is indirectly calculated from other outputs:
[0042]
[0043] Taking into account that the feature extractor weights are not updated in the fine-tuning stage in the present invention in order to alleviate overfitting, the system's ability to learn to detect new types of targets is limited. Since the knowledge required to detect new types of targets is transferred from the network pre-trained with a large amount of labeled sample base class data, the effect of optimizing the detection ability of new types of targets can be achieved by enhancing pre-training. Unlike the training images used in image classification tasks, the training images for target detection tasks usually contain richer semantic information. There may be potential targets in the background of some training images, and these potential targets are the key to extracting general target detection knowledge. Since the background in the training image does not contain annotations, knowledge cannot be extracted from the background through supervised learning methods. The present invention introduces contrastive learning under the self-supervised paradigm into the training process to solve this problem.
[0044] Typical contrastive learning approaches treat each image as an instance, guiding the neural network to learn image representation encodings by distinguishing between positive and negative examples. Applying contrastive learning to object detection presents two challenges: First, each image typically contains multiple objects, and different image locations exhibit significant semantic differences, making contrastive learning infeasible for a single image as an instance. Second, some objects have overlapping bounding boxes within an image, making them infeasible to directly consider as negative pairs. Consequently, the relationship between positive and negative examples is complex.
[0045] To address the above two issues, this paper proposes a region contrast learning algorithm to enhance pre-training. In this algorithm, the regions corresponding to the proposal boxes predicted by the RPN network in the Faster-RCNN object detection framework are used as examples. During the training process, the network is treated as two twin branches, denoted as the online branch and the target branch. During training, each input image is augmented with random data to obtain two different views v o =A o (x), v t =A t (x), the position label also changes accordingly The two views are used as inputs of the two branches respectively. Among them, the online branch performs conventional target detection training while executing the contrastive learning algorithm. The RPN network in the online branch calculates the proposed box set Each suggestion box The proposal box corresponding to the target branch view is calculated according to the augmentation function Considering the translation invariance of the convolutional neural network in the backbone network, p o and p t The corresponding feature maps have a certain similarity. The present invention adds a random offset to the coordinates of the proposal box position as augmentation to enhance the contrastive learning effect. Due to the cropping operation in image augmentation, some proposal boxes are beyond the view range and these proposal boxes do not participate in the training. The final proposal box set participating in contrastive learning is obtained through augmentation and screening. and According to the feature map f of each branch in the proposed box in the set o and f t Perform the RoI-Align pooling operation on the RoI pooling feature to obtain the RoI pooling feature as the input of the RoI feature extractor F. The output feature is then input into the multi-layer perceptron G. The output of the online branch G will be used as a predictor P composed of a multi-layer perceptron. Finally, the outputs of the two branches are substituted into the contrast loss, as shown in the following formula:
[0046]
[0047] Where <·,·> is the cosine similarity, and R(·,·) is the abbreviation of the mapping G(F(RoIAlign(·,·))). As the loss decreases, the positive sample pairs gradually approach each other in the feature space, improving the expression effect and versatility of the features, thereby improving the performance of detecting new types of objects. The loss function of the entire pre-training process is defined as:
[0048] L=L rpn +L cls +L reg +λL ctr
[0049] The first three items are the target detection training losses of the Faster-RCNN algorithm, and λ is the scale factor of the contrast loss.
[0050] The present invention is implemented on the MMFewShot framework for few-shot learning based on the Pytorch deep learning platform, and specifically includes the following steps:
[0051] Step 1: Select a suitable object detection dataset and divide all classes in the dataset into base classes and novel classes according to a certain ratio (3:1 is used in this paper). All training samples of the base class are retained for pre-training, and a small number of samples are randomly selected from the training set for each novel class as training samples for fine-tuning. The number can be 1, 2, 3, 5, 10, etc.
[0052] Step 2: Initialize the neural network, where the classifier initializes the same number of weight vectors as the number of categories during pre-training;
[0053] Step 3: Perform pre-training. The steps for each iteration are as follows:
[0054] Step 3.1: Randomly sample data from the training set and perform four augmentation steps on each image in the data: random cropping, random flipping, padding, and resizing, to obtain two views. The annotation information is also transformed along with these transformations to obtain the annotations corresponding to the views. The two views are used as the input of the neural network, and the forward propagation of the two views is denoted as the online branch and the target branch, respectively. The target branch does not calculate gradients.
[0055] Step 3.2: The view is input into the backbone network to extract features and obtain feature maps. The feature maps of the online branch are fed into the RPN to calculate the RPN loss and proposal box.
[0056] Step 3.3: The proposal box of the online branch is calculated based on the information in the augmentation operation and the position of the view in the target branch as the proposal box of the target branch. A random offset within ±20% is added to the positions of the proposal boxes of the two branches, and proposal box pairs with IoU less than 0.6 are screened out;
[0057] Step 3.4: The proposal boxes of each branch are all applied to RoI-Align pooling to obtain the feature map of the area corresponding to the proposal box. The feature map is input into the RoI feature extractor. The online branch calculates the category feature vector and position feature vector corresponding to each proposal area, and the target branch only calculates the category feature vector.
[0058] Step 3.5: Calculate the cosine similarity between the feature vector of the online branch and the weight vector of each class in the classifier to obtain the output of the classifier. The classification loss is calculated by softmax activation and cross entropy loss. The position feature vector is input into the regressor to calculate the position regression loss.
[0059] Step 3.6: The category feature vectors of each branch are input into a multi-layer perceptron to obtain the contrast space vector. The contrast space vector of the online branch is input into the predictor to obtain the prediction vector. The contrast loss is calculated between the prediction vector and the contrast space vector of the corresponding target branch.
[0060] Step 3.7: Add the losses calculated in the above steps and backpropagate to update the neural network weights.
[0061] Step 4: Enter a new incremental phase. Based on the neural network trained in the previous phase, fix the existing weights and do not update them in the future. Then initialize the same number of weight vectors according to the number of categories to be learned in this phase.
[0062] Step 5: Fine-tune the neural network. The rest of the training method is the same as Faster-RCNN, except that the classifier calculation method is the same as step 3.5.
[0063] Step 6: If new categories need to be learned, loop back to step 4; otherwise, end the system training process.
[0064] The patented invention utilizes a cosine similarity classifier under a category registration mechanism and region contrast learning for continuous few-shot object detection. This method demonstrates superior object detection performance and flexibility in continuous few-shot learning scenarios. Experiments on the popular object detection dataset COCO demonstrated that the average precision (AP) performance of the base class decreased by no more than 0.5% with a single incremental learning pass, and by no more than 2.2% over 20 consecutive incremental runs.
Claims
1. A continuous few-shot object detection method based on a category registration mechanism and region contrast learning, the method comprising: Step 1: Select a suitable target detection dataset and divide all categories of the dataset into base classes and novel classes in proportion; The base class retains all training samples for pre-training, and each novel class randomly extracts a small number of samples from the training set as training samples for fine-tuning; Step 2: Initialize the system network, which includes a feature extractor, a classifier, and a regressor. The classifier and regressor are initialized with the same number of weight vectors based on the number of categories in pre-training. The output of the classifier is passed through the softmax activation layer and then subjected to the cross entropy loss, as follows: Among them, l i,j represents the j-th classifier output value of the i-th sample, gt represents the label category, bg represents the background category, N o represents the total number of samples; Among them, α represents the scale factor, and the cosine similarity between the feature vector output by the feature extractor and the classifier weight vector is used as the output of the classifier, which is expressed as: Among them, α is the same as the scale factor in the previous formula, r i Represents the feature extractor output feature vector, w j is the classifier weight vector; Step 3: Perform pre-training. The steps for each iteration are as follows: Step 3.1: Randomly sample data from the training set, each sample contains image x, box category y gt and box position y loc , perform four augmentation steps of random cropping, random flipping, padding, and size transformation on each image in the data to obtain two views v o =A o (x) and v t =A t (x), the annotation information is also transformed with the above transformation to obtain the annotation corresponding to the view and The two views are used as the input of the neural network, and the forward propagation processes of the two views are recorded as the online branch and the target branch respectively, where the target branch does not calculate the gradient; Step 3.2: The view is input into the backbone network to extract features and obtain feature maps; the feature maps of the online branches are input into the Region Proposal Network (RPN) of Faster-RCNN to calculate the RPN loss and the target branch proposal box. p o Proposal boxes representing online branches; Step 3.3: Add a random offset to the proposal box position coordinates as augmented data. The proposal box of the online branch is calculated based on the information in the augmentation operation and corresponds to the position of the view in the target branch as the proposal box of the target branch. A random offset within ±20% is added to the positions of the proposal boxes of the two branches, and proposal box pairs with an intersection over union (IoU) less than 0.6 are screened out. Step 3.4: The proposal boxes of each branch are pooled through RoI-Align to obtain the feature map f of the region of interest (RoI) corresponding to the proposal box o and f t , the feature map is input into the RoI feature extractor F, the online branch calculates the category feature vector and position feature vector corresponding to each proposed region, and the target branch only calculates the category feature vector; Step 3.5: Calculate the cosine similarity between the feature vector of the online branch and the weight vector of each class in the classifier to obtain the output of the classifier. The classification loss is calculated by softmax activation and cross entropy loss. The position feature vector is input into the regressor to calculate the position regression loss. Step 3.6: The category feature vectors of each branch are input into a multi-layer perceptron to obtain the contrast space vector. The contrast space vector of the online branch is input into the predictor to obtain the prediction vector. The prediction vector and the contrast space vector of the corresponding target branch are used to calculate the contrast loss L. ctr ; Where <·,·> is the cosine similarity, R(·,·) is the abbreviation of the mapping G(F(RoIAlign(·,·))), and It represents the proposed box set pair that is finally involved in contrastive learning through augmentation and screening. and represents the corresponding proposal box in the set; Step 3.7: Add the losses and backpropagate to update the neural network weights; L=L rpn +L cls +L reg +λL ctr Among them, L rpn 、L cls and L reg They represent the RPN loss, RoI classification loss and position regression loss of Faster-RCNN network training respectively, and λ represents the contrast loss scaling coefficient; Step 4: Enter a new incremental phase. Based on the neural network trained in the previous phase, fix the existing weights and do not update them in the future. Then initialize the same number of weight vectors according to the number of categories to be learned in this phase. Step 5: Fine-tune the neural network. During fine-tuning, only the classifier calculation method is the same as step 3.5 during system network training, and the rest of the methods are trained normally. Step 6: If new categories need to be learned, loop back to step 4; otherwise, end the system training process.