A few-shot object detection method based on transfer learning

Abstract

The application discloses a few-shot target detection method based on transfer learning, total data sets are divided into a few-shot class set and a base class set according to the number of samples, and the base class set is used to train the whole target detector; a copy of the detector in the base training stage is taken, the backbone network and part of the feature network are taken, and part of the backbone network is frozen; the total data set is used, the copied frozen backbone network and the original network are combined by using a stable constraint loss, and a back propagation algorithm is used for fine-tuning training; during testing, the copied network is discarded, and the original model after fine-tuning is used to output classification and regression results. In the case that no additional inference overhead is increased, better few-shot class and base class target detection performance is brought.

Description

Technical Field

[0001] This invention relates to a few-shot target detection method based on transfer learning, belonging to the fields of computer vision and deep learning technology in computer technology. It can effectively improve the detection performance of both base class and few-shot class, and achieve a higher overall detection accuracy. Background Technology

[0002] Few-shot object detection is a crucial topic in computer vision. In recent years, the deep learning AI paradigm, driven by computing power and data, seems to have become the norm, and the reliance on large-scale, high-quality datasets has become a common bottleneck for many deep learning algorithms. However, possessing human intelligence means having the ability to reason, inductively, and summarize from limited data. In many real-world scenarios, humans struggle to collect large amounts of finely labeled, effective samples, leaving many deep learning algorithms stretched thin. Therefore, the concept of the few-shot problem has gradually gained attention and research, becoming a classic problem in machine learning and artificial intelligence with significant practical implications. The few-shot problem also exists in many areas of computer vision, including image classification, object detection, and semantic segmentation, but research on tasks such as object detection has lagged behind.

[0003] Mainstream algorithms in the field of few-shot object detection fall into two categories: meta-learning-based few-shot detection algorithms and transfer learning-based few-shot detection algorithms. Meta-learning teaches the model "how to learn" by constructing specific network structures and losses, allowing the classifier to learn how to distinguish image features from different categories, rather than the specific information of all classes themselves. Since everything is "classified," meta-learning-based few-shot detection methods borrow this idea and adapt it to object detection frameworks. Transfer learning-based few-shot detection is more intuitive, employing a classic pre-training-fine-tuning process: training the feature extractor with a large amount of common class data, and then fine-tuning the classifier with rare class data. The transfer learning framework is simple, clear, and highly scalable, and generally exhibits better overall performance than meta-learning framework detectors. Recently, although transfer learning-based few-shot detection frameworks have accumulated some research, many problems still exist due to inconsistencies in low-level models and evaluation criteria. For transfer learning itself, the core lies in how to control the gradient backpropagation range during the fine-tuning stage to simultaneously prevent overfitting and catastrophic forgetting, balancing the performance of the base class and the few-shot class. Previous studies have often suggested that these two aspects are mutually exclusive, so there is an urgent need to propose a more effective few-shot detection method that can better balance the performance of the base class and the few-shot class, and achieve better target detection for image data and other data. Summary of the Invention

[0004] Objective: To address the problems and shortcomings of existing technologies, this invention provides a few-shot object detection method based on transfer learning. By reusing the base training model and utilizing base class information during the fine-tuning stage to prevent catastrophic forgetting of common classes, the method further improves the accuracy of rare classes by incorporating constraint features learned during fine-tuning. It adds no overhead during the inference stage and is applicable to most transfer learning-based models, delivering consistent performance improvements to solve the problems encountered in object detection using image data, as mentioned in the background.

[0005] Technical solution: A few-shot target detection method based on transfer learning, mainly including the following steps:

[0006] Step 1: Divide the total dataset into a small sample class set and a base class set according to the number of samples, and use the base class set to train the entire object detector;

[0007] Step 2: Make a copy of the detector from the base training phase, take the backbone network and part of the feature network of the copied detector, and freeze the entire backbone network and part of the frozen feature network.

[0008] Step 3: Using the total dataset, combine the replicated frozen backbone network and the original detector backbone network with the stability constraint loss, and use the backpropagation algorithm to fine-tune the original detector network to obtain the fine-tuned target detection model.

[0009] Step 4: During testing, discard the copied network and output classification and regression results using the fine-tuned object detection model.

[0010] Furthermore, the total dataset is an object detection image dataset, the samples in the total dataset are image data, and the fine-tuned object detection model is a model used for object detection; when the fine-tuned object detection model is tested, it takes image data as input and outputs the object detection results, including the object's category and location.

[0011] Furthermore, training the entire target detector as described in step 1 specifically includes the following steps:

[0012] Step 1.1: The total dataset is divided into a set of fewer classes, D. novel and the base class set D base Each contains a small number of samples of category C. novel and common category C base Images;

[0013] Step 1.2: Train a deep convolutional neural network on an image classification dataset, the deep convolutional neural network being used to initialize the object detector;

[0014] Step 1.3: Embed the convolutional neural network into a two-stage object detection framework, setting the number of categories in the classification layer of the detection head to the number of common instance categories in the few-shot detection dataset, and using the base class set D. base Train the entire detector.

[0015] Furthermore, the step 2, which involves copying the detector from the base training phase, taking the backbone network and part of the feature network of the copied detector, and freezing the entire backbone network and part of the frozen feature network, specifically includes the following steps:

[0016] Step 2.1: Make a copy of the entire detector trained on the base and set it aside for later use. The detector includes the backbone network φ and the region of interest feature network ψ.

[0017] Step 2.2: Take the backbone network, feature pyramid network (if any), and the first two layers of the region of interest feature network of the copied detector;

[0018] Step 2.3: Freeze the backbone network, feature pyramid network (if any), and the first two layers of the region of interest feature network of the detector copied in Step 2.2. This involves canceling gradient backpropagation and weight updates for the frozen portion. The frozen backbone network is denoted as φ. b The first two layers of the frozen region of interest feature network are denoted as ψ. b .

[0019] Furthermore, in step 3, the replicated frozen backbone network and the original backbone network are combined using a stability constraint loss, and fine-tuned using the backpropagation algorithm. This specifically includes the following steps:

[0020] Step 3.1, during forward propagation, the base class image I from the total dataset... base Simultaneously, the original detector (parameters can be fine-tuned) and the freezing part of step 2.3 (parameters are frozen and cannot be fine-tuned) are entered to obtain the generated region of interest features ψ(RPN(φ(I)) respectively. base ))) and ψ b (RPN(φ b (I base And apply stability constraint loss L to the two features. SR ;

[0021] Step 3.2, during forward propagation, all images I from the total dataset are fed into the fine-tuned detector, and the detection head classification and regression losses are calculated. Region-based network classification and regression loss

[0022] Step 3.3 involves upsampling the minority class images using a class-based image sampler, making their frequency of participation in training close to that of the base class images; further, the specific method for upsampling the minority class images is as follows: Define In one training cycle (including all D) base In the epoch of the image, every From the images, randomly select one d novel The images in the training image queue are inserted.

[0023] Step 3.4, the total loss for fine-tuning is set to the stability constraint loss L. SR Detection head classification and regression loss And regional suggested network classification and regression loss With rights and.

[0024] Furthermore, the detailed process of obtaining the two sets of region of interest features in step 3.1 is as follows:

[0025] f bb =φ(I base )

[0026] f RoI =ψ(RPN(f) bb ))

[0027]

[0028]

[0029] Where f RoI and The regions of interest (ROIs) are generated by the fine-tuning network (original network) and the copying network, respectively. The RPN is the original region proposal network, which is shared by both.

[0030] Furthermore, in step 3.1, a negative cosine loss is used to calculate the stability constraints for the two regions of interest features:

[0031]

[0032] Where B represents a single small batch, B base R represents the base class image in the current batch, and R is the characteristic function.

[0033] Furthermore, the specific formula for calculating the total loss in step 3.4 is as follows:

[0034]

[0035] Where the stability constraint loss L SRThe weighting coefficient α is empirically set to 0.1, and the weighting coefficient η is used to balance the loss of the detection head and the region proposal network, and is empirically set to 1.

[0036] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the few-shot target detection method based on transfer learning as described above.

[0037] A computer-readable storage medium storing a computer program that performs the few-shot target detection method based on transfer learning as described above.

[0038] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0039] This invention provides a few-shot target detection method based on transfer learning. After pre-training the detection model using a base training set, the pre-trained model is copied. During the fine-tuning transfer learning stage, the model is frozen and reused, ensuring that the instance features generated by the original model are as close as possible to the features generated by the frozen pre-trained model, thus constraining the training in the fine-tuning stage. Starting from the fine-tuning layer, a key focus of transfer learning, this method explores the potential value of the base training set and the base training model. Without introducing additional overhead, it significantly improves the performance of the few-shot class and greatly enhances the performance of the base class, solving the catastrophic forgetting problem. The algorithm is clear in principle, simple to use, and can achieve advanced results when superimposed on existing detection models without additional overhead, achieving better target detection for image data. Attached Figure Description

[0040] Figure 1 This is a flowchart illustrating the main steps in an embodiment of the present invention.

[0041] Figure 2 : This is a flowchart illustrating the steps involved in fine-tuning the model design in an embodiment of the present invention;

[0042] Figure 3 This is a flowchart illustrating the steps involved in designing the dataset sampler in this invention. Detailed Implementation

[0043] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading the present invention, any modifications of the present invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0044] A few-shot target detection method based on transfer learning will be described in detail below with reference to the accompanying drawings and embodiments.

[0045] Reference Figure 1 As shown, the present invention mainly includes the following steps:

[0046] Step 1: Divide the total dataset into a small sample class set and a base class set according to the number of samples, and use the base class set to train the entire object detector;

[0047] Step 2: Copy the detector from the base training phase, take its backbone network and part of the feature network, and freeze the entire backbone network and part of the frozen feature network.

[0048] Step 3: Using the total dataset, combine the replicated frozen backbone network and the original network with the stable constraint loss, and fine-tune the original detector network using the backpropagation algorithm.

[0049] Step 4: During testing, discard the copied network and use the fine-tuned original model to output classification and regression results.

[0050] Training the entire target detector as described in step 1 specifically includes the following steps:

[0051] Step 1.1: The total dataset is divided into a set of fewer classes, D. novel and the base class set D base Each contains a small number of samples of category C. novel and common category C base Images;

[0052] Step 1.2: Train a deep convolutional neural network on an image classification dataset;

[0053] Step 1.3: Embed the convolutional network into a two-stage object detection framework, setting the number of categories in the classification layer of the detection head to the number of common instance categories in the few-shot detection dataset, and using the base class set D. vase Train the entire detector.

[0054] Specifically, the set of few classes D novel In this context, typically only a few dozen or even just a few instances per category participate in training. base These images comprise over 95% of the total, making them suitable for training a high-performance general-purpose object detector. This step pre-trains the entire model, enabling it to learn a better backbone and feature network, thus providing stability constraints for the region of interest features generated during the fine-tuning stage.

[0055] Reference Figure 2 As shown, step 2 involves copying the detector from the base training phase, taking its backbone network and part of the feature network, and partially freezing it. This specifically includes the following steps:

[0056] Step 2.1: Make a copy of the entire detector trained on the base and set it aside for later use;

[0057] Step 2.2: Take the first two layers of the backbone network φ, feature pyramid network (if any), and region of interest feature network ψ of this part (i.e., remove the region proposal network, final classification and regression modules);

[0058] Step 2.3: Freeze all the above modules, that is, cancel the gradient backpropagation and weight update for this part, and obtain {φ} b ,ψ b}

[0059] Specifically, this step yields an excellent feature extractor for the base class images, preventing catastrophic forgetting caused by the few-shot class during fine-tuning of the original network. Note that the region proposal network of the base-trained model is not saved here, but shared with the original model during fine-tuning to ensure that features are generated for the same set of regions of interest.

[0060] Step 3, which involves combining the replicated frozen backbone network with the original network using a stable constraint loss and fine-tuning the training using the backpropagation algorithm, specifically includes the following steps:

[0061] Step 3.1, during forward propagation, the base class image I from the total dataset... base Simultaneously, the fine-tuning detector and the detector partial copy module are entered, respectively obtaining the generated region of interest features ψ(RPN(φ(I)). base ))) and ψ b (RPN(φ b (I base And apply stability constraint loss L to the two features. sR ;

[0062] Step 3.2, during forward propagation, all images I from the total dataset... base Enter the fine-tuned detector and calculate the detection head classification and regression loss. Region-based network classification and regression loss

[0063] Step 3.3: Use a class-based image sampler to upsample the few class images so that their frequency of participation in training is close to that of the base class images;

[0064] Step 3.4, the training loss is set to the stability constraint loss L. SR Detection head classification and regression loss And regional suggested network classification and regression loss With rights and.

[0065] Specifically, this section details the fine-tuning process. The first two steps reveal a difference in how the base class images and the fine-tuned images in a mini-batch are processed. The former requires simultaneous participation in both the old and new feature extractors and the calculation of the stability constraint loss; the latter only needs to participate in the training of the regular detector. To prevent class imbalance, the third step upsamples the minority classes. The fourth step presents the formula for the total loss.

[0066] The detailed process of obtaining the two sets of region of interest features in step 3.1 is as follows:

[0067] f bb =φ(I base )

[0068] f RoI =ψ(RPN(f) bb ))

[0069]

[0070]

[0071] Where f RoI and The regions of interest (ROIs) are generated by the fine-tuning network (original network) and the copying network, respectively. The RPN (Reference Region Proposal Network) is shared by both. Different classic two-stage detection models have different ROI feature networks ψ, which may be multi-layer neural networks. Experiments have shown that choosing the last layer as the feature mimicry layer usually yields good performance, as described in the formula above.

[0072] In step 3.1, a negative cosine loss is used to calculate the stability constraints for the two regions of interest features:

[0073]

[0074] Where B represents a single small batch, B base For this batch of base class images, E is the indicator function. Here, negative cosine loss is only one feasible solution; other feasible solutions include l1 loss, l2 loss, etc., all of which can bring good results.

[0075] Reference Figure 3 As shown, the specific method for upsampling the few-sample class images in step 3.3 is as follows: Define In one training cycle (including all D) base In the epoch of the image, every From the images, randomly select one d novelThe images are then inserted into the training data. This ensures that the frequency of minority class images participating in training is the same as the proportion of minority class images to the total number of classes, thus preventing extreme class imbalance.

[0076] The specific formula for calculating the total loss in step 3.4 is as follows:

[0077]

[0078] Where the stability constraint loss L SR The weight coefficient α is empirically set to 0.1 to prevent the constrained features from deteriorating.

[0079] Obviously, those skilled in the art should understand that the steps of the few-shot target detection method based on transfer learning in the above embodiments of the present invention can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by the computing device. Furthermore, in some cases, the steps shown or described can be performed in a different order than presented herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the embodiments of the present invention are not limited to any particular hardware and software combination.

[0080] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A few-shot target detection method based on transfer learning, characterized in that, Training with additional stability constraint branches includes the following steps: Step 1: Divide the total dataset into a small sample class set and a base class set according to the number of samples, and use the base class set to train the entire object detector; Step 2: Make a copy of the detector from the base training phase, take the backbone network and part of the feature network of the copied detector, and freeze the entire backbone network and part of the frozen feature network. Step 3: Using the total dataset, combine the replicated frozen backbone network and the original detector backbone network with the stability constraint loss, and use the backpropagation algorithm to fine-tune the original detector network to obtain the fine-tuned target detection model. Step 4: During testing, discard the copied network and output classification and regression results using the fine-tuned object detection model; Training the entire target detector as described in step 1 specifically includes the following steps: Step 1.1: The total dataset is divided into a set of fewer samples. and base class collection Each contains a small number of sample categories. and common categories Images; Step 1.2, in the image classification dataset A deep convolutional neural network is trained on the target detector; Step 1.3: Embed the convolutional neural network into a two-stage object detection framework, setting the number of categories in the classification layer of the detection head to the number of common instance categories in the few-shot detection dataset, and using the base class set. Training the entire detector; Step 2 involves making a copy of the detector from the base training phase, taking the backbone network and part of the feature network of the copied detector, and partially freezing the backbone network and feature network. This specifically includes the following steps: Step 2.1: Make a copy of the entire detector trained on the base network for later use. The detector includes the backbone network. and region of interest feature network ; Step 2.2: Take the first two layers of the backbone network and the region of interest feature network of the copied detector. If the detector has a feature pyramid network, the first two layers of the feature pyramid network also need to be copied. Step 2.3: Undo the gradient backpropagation and weight update of the frozen portion. The frozen backbone network is denoted as... The first two layers of the frozen region of interest feature network are denoted as ; Step 3, which involves combining the replicated frozen backbone network and the original backbone network using a stability constraint loss and fine-tuning the training using the backpropagation algorithm, specifically includes the following steps: Step 3.1, during forward propagation, the base class images from the total dataset Simultaneously, the original detector and the detector partial copy module are entered, respectively obtaining the features of their respective generated regions of interest. and And apply stability constraint loss to the two features. ; Step 3.2, during forward propagation, all images from the total dataset... Enter the fine-tuned detector and calculate the detection head classification and regression loss. Regional suggested network classification and regression loss ; Step 3.3: Upsample the few-sample class images using a class-based image sampler; Step 3.4: Set the total loss for fine-tuning to the stability constraint loss. Detection head classification and regression loss And regional suggested network classification and regression loss With rights and.

2. The few-shot target detection method based on transfer learning according to claim 1, characterized in that, The total dataset is a target detection image dataset, and the samples in the total dataset are image data. The fine-tuned target detection model is a model used for target detection. When the fine-tuned target detection model is tested, it takes image data as input and outputs the target detection results, including the target's category and location.

3. The few-shot target detection method based on transfer learning according to claim 1, characterized in that, The detailed process of obtaining the two sets of region of interest features in step 3.1 is as follows: in and These are the region-of-interest features generated by the fine-tuning network and the copying network, respectively. It is the original regional suggested network, which is shared by both.

4. The few-shot target detection method based on transfer learning according to claim 1, characterized in that, In step 3.1, a negative cosine loss is used to calculate the stability constraints for the two regions of interest features: in Represents a single small batch. The base class image in the current batch. This is an indicator function.

5. The few-shot target detection method based on transfer learning according to claim 1, characterized in that, The specific method for upsampling the few-sample class images in step 3.3 is as follows: Define In a training cycle, every One image is randomly selected from the images. Images are placed within them.

6. A computer device, characterized in that: The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the few-sample target detection method based on transfer learning as described in any one of claims 1-5.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program that performs the few-shot target detection method based on transfer learning as described in any one of claims 1-5.

Images