A small-sample object detection method based on mutual distillation and self-supervised latent knowledge mining

Abstract

The invention discloses a small-sample target detection method for mutual distillation self-supervised latent knowledge mining, which belongs to the field of small-sample learning and target detection. The technical problems solved by the present invention include: a fine-tuning method based on first training the target detector on a large number of base class samples, and then fine-tuning the detector on a small number of new class samples. During the fine-tuning process, the feature extractor parameters are almost unchanged, that is, the feature extractor in the final new class detector is mainly obtained by training the base class samples. The feature extractor trained on the base class image will lack the ability to extract the discriminative features of the new class target. The present invention proposes to use the representation obtained by self-supervised pre-training to guide the base class training process to improve the generalization of the feature encoder of the detector in the base class training.

Description

Technical Field

[0001] The present invention belongs to the field of small sample learning and target detection. Background Art

[0002] General object detection refers to finding all objects of predefined categories within an image, and providing their locations and category labels. For each predefined category, current deep neural network-based methods require a large number of training samples to drive network learning, resulting in detectors that generalize across all categories. However, in real-world scenarios, due to difficulties in image acquisition and high labeling costs, it is difficult to obtain a large number of training samples for some categories. Few-shot object detection (FSOD) addresses the problem of how to train an object detector that generalizes well to new categories when there are a large number of base class samples and a small number of new class sample pairs.

[0003] Current FSOD algorithms are primarily categorized into two types: meta-learning-based and fine-tuning-based. Early algorithms primarily focused on meta-learning approaches, studying "learning how to learn," using a meta-transformer to transfer meta-knowledge from base classes to novel classes. Recently, researchers have discovered that simply training a detector with a large amount of base class data and then fine-tuning it with a small amount of novel class data can achieve good performance. Extensive research has focused on the second step of fine-tuning approaches: fine-tuning on novel classes.

[0004] Self-supervised learning involves training a neural network using defined training tasks to extract useful information from large amounts of unsupervised data, enabling the trained neural network to extract relatively generalized representations. Common self-supervised learning methods currently used in computer vision include purely visual self-supervised pre-training, such as MoCO, SimCLR, and SimSiam. These pre-train on large amounts of unlabeled images to obtain robust feature extraction networks. Furthermore, driven by the development of multimodal technologies, recent vision-language self-supervised pre-training models such as CLIP and ALIGN have emerged. These pre-train on image-text pairs collected from the internet to collaboratively extract image and text feature extraction networks. The learned image and text feature extraction networks can map images and text into the same feature space. However, due to the domain gap between the upstream self-supervised pre-training task and the downstream specific task (such as few-shot object detection), the generalized representations extracted by the self-supervised pre-trained feature extractor cannot be directly applied to downstream tasks. Therefore, a proper transformation of the generalized representations is required to better utilize the latent knowledge of self-supervision.

[0005] Knowledge distillation is a knowledge transfer technology that transfers knowledge from a pre-trained high-performance teacher network to a student network with smaller capacity to improve the performance of the student network. Distillation methods are mainly divided into two types: feature distillation and logits distillation. Feature distillation directly makes the middle-layer features of the student network imitate the middle-layer features of the teacher network, and logits distillation makes the logits classified by the student network align with the logits of the teacher network. In addition to the one-way knowledge transfer from the teacher network to the student network, some researchers have also proposed mutual distillation, that is, letting the teacher network and the student network distill each other. This method can improve the performance of the teacher network and the student network at the same time. The present invention improves the ordinary mutual distillation and designs a regional decoupled asymmetric mutual distillation method for the representation characteristics of small sample target detection. It is used to mine the potential knowledge in the base class training images, and better realizes the transfer of potential knowledge from the self-supervised pre-training network to small sample target detection. Summary of the Invention

[0006] The technical problems solved by the present invention include: the fine-tuning method first trains the target detector on a large number of base class samples, and then fine-tunes the detector on a small number of new class samples. During the fine-tuning process, the feature extractor parameters are almost unchanged, that is, the feature extractor in the final new class detector is mainly trained by the base class samples. The feature extractor trained on the base class images will lack the ability to extract the discriminative features of the new class targets. The present invention proposes to use the representation obtained by self-supervised pre-training to guide the base class training process to improve the generalization of the feature encoder of the detector in the base class training.

[0007] The technical solution of the present invention is a small sample target detection method for mutual distillation self-supervised potential knowledge mining, which includes:

[0008] Step 1: Initialize an object detector and a self-supervised pre-trained feature extractor, and a converter from self-supervised features to detection features; the object detector contains a feature extractor E D ;

[0009] Step 2: Use the feature extractor E of the target detector respectively D and self-supervised feature extractor E P Extract the image feature spectrum of the base class image; obtain the detection feature spectrum F D and self-supervised feature spectrum F P ;

[0010] Step 3: Use the annotation box to detect the feature spectrum F D and self-supervised feature spectrum F P The foreground features are extracted by ROI pooling to obtain F D-O and F P-O ; Detect characteristic spectrum F DSend it to the region candidate network to generate a candidate box that may contain the target, select the part of the candidate box with a smaller intersection than the annotation box as the background box, and use the background box to generate the candidate box in F. D and F P The background features are extracted by ROI pooling to obtain F D-B and F P-B ;

[0011] Step 4: Optimize the parameters of the converter from self-supervised features to detection features by learning the mapping relationship between foreground features of self-supervised features and background features of detection features;

[0012] Step 5: Use the converter in step 4 to map the self-supervised features of the background to obtain the predicted background detection features, and optimize the feature extraction network in the detector by making the background detection features extracted by the detector close to the predicted background detection features.

[0013] Step 6: Fine-tune the detector obtained in step 5 using a small number of new class samples to obtain an object detector for the new class.

[0014] The present invention proposes a small sample representation algorithm for self-supervised latent knowledge mining based on mutual distillation, which improves the performance of the feature extraction network obtained in base class pre-training with the help of self-supervised representation with strong generalization. Compared with the ordinary fine-tuning method based on the benchmark point, it has higher prediction accuracy on new classes. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 Provides the overall process framework for small sample target detection algorithms based on fine-tuning;

[0016] Figure 2 This is an illustration of the self-supervised latent knowledge mining method based on asymmetric mutual distillation proposed in the present invention. DETAILED DESCRIPTION

[0017] Step 1: Define a Faster R-CNN target detector D, where the feature extraction network E D Initialized by Imagnet pre-training parameters, and the rest are randomly initialized; at the same time, an additional feature extraction network E is defined P , initialized by large-scale self-supervised pre-training; E D and E P Both are ResNet-101 residual networks;

[0018] Step 2: Use E D and E P Extract the features of a base class image I respectively and obtain the detection feature spectrum F D and self-supervised feature spectrum F P , use the callout box to D and F PThe foreground features are extracted by ROI pooling to obtain F D-O and F P-O ; Select the part of the background box generated by the detector that is smaller than the intersection of the annotation box as the background box, and use the background box to D and F P The background features are extracted by ROI pooling to obtain F D-B and F P-B ;

[0019] Step 3: Define a converter T that converts self-supervised features into detection features, and convert F P-O ,F P-B Send it to the converter to get the converted features The structure of the converter is: fully connected layer-ReLU activation function-fully connected layer;

[0020] Step 4: Define a multi-layer perceptron MLP(.) and transform F D-O ,F D-B , Send it to the multi-layer perceptron to get the mapped feature F D ′ -O ,F D ′ -B ,

[0021] Step 5: Define a prediction network P(.) and define the loss function for optimizing the converter T as follows:

[0022]

[0023] Among them, detach(.) means disconnecting the gradient, ‖,‖ means finding the Euclidean distance;

[0024] Step 6: Optimize E using background pixel signature D ,The loss function of background pixel feature spectrum is defined as follows:

[0025]

[0026] Step 7: Detection loss L in Faster R-CNN object detector det 、Converter T loss L t and self-supervised knowledge transfer loss L s The detector base class training is completed under joint supervision, and the total loss is as follows:

[0027] L=L det +L t +L s

[0028] Step 8: Fine-tune the object detector using samples from the new class. The fine-tuning process only uses the detection loss of the detector to obtain a high-performance object detector suitable for the new class.

Claims

1. A small sample target detection method for mutual distillation self-supervised latent knowledge mining, the method comprising the following steps: 1: Define a Faster R-CNN target detector D, where the feature extraction network E D Initialized by Imagnet pre-training parameters, and the rest are randomly initialized; at the same time, an additional feature extraction network E is defined P , initialized by large-scale self-supervised pre-training; E D and E P Both are ResNet-101 residual networks; Step 2: Use E D and E P Extract the features of a base class image I respectively and obtain the detection feature spectrum F D and self-supervised feature spectrum F P , use the callout box to D and F P The foreground features are extracted by ROI pooling to obtain F D-O and F P-O ; Select the part of the background box generated by the detector that is smaller than the intersection of the annotation box as the background box, and use the background box to D and F P The background features are extracted by ROI pooling to obtain F D-B and F P-B ; Step 3: Define a converter T that converts self-supervised features into detection features, and convert F P-O ,F P-B Send it to the converter to get the converted features The structure of the converter is: fully connected layer-ReLU activation function-fully connected layer; Step 4: Define a multi-layer perceptron MLP(.) and transform F D-O ,F D-B , Send it to the multi-layer perceptron to get the mapped features Step 5: Define a prediction network P(.) and define the loss function for optimizing the converter T as follows: Among them, detach(.) means disconnecting the gradient, ‖,‖ means finding the Euclidean distance; Step 6: Optimize E using background pixel signature D ,The loss function of background pixel feature spectrum is defined as follows: Step 7: Detection loss L in Faster R-CNN object detector det 、Converter T loss L t and self-supervised knowledge transfer loss L s The detector base class training is completed under joint supervision, and the total loss is as follows: L=L det +L t +L s Step 8: Fine-tune the object detector using samples from the new class. The fine-tuning process only uses the detection loss of the detector to obtain a high-performance object detector suitable for the new class.

Images