A multi-detection head two-stage small sample target detection method based on an improved SPP structure
By improving the SPP structure and multi-detector head design, the problem of insufficient feature information in small sample target detection of the FPN structure is solved, realizing more flexible and accurate target detection, adapting to targets of different scales, and reducing computational costs.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- TONGJI UNIV
- Filing Date
- 2024-05-31
- Publication Date
- 2026-05-29
AI Technical Summary
In small sample target detection, existing technologies cannot fully utilize deep network information due to the FPN structure, resulting in poor performance on low-resolution or inaccurate feature maps and difficulty in effectively detecting new types of sample targets of different sizes.
An improved SPP structure is used to design a multi-detector head two-stage target detection method. Features from different network levels are extracted through a five-branch SPP structure, and average pooling is introduced into the detector head to construct a multi-detector head structure to fuse multi-scale information.
This improved the model's detection performance for targets of different sizes, increased the accuracy and efficiency of detection, reduced computational costs, and enhanced the model's robustness and adaptability.
Smart Images
Figure CN118691792B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of small sample target detection, and in particular to a multi-head, two-stage small sample target detection method based on an improved SPP structure. Background Technology
[0002] Object detection is a core area of computer vision, tasked with locating and identifying objects of interest in images. However, real-world object categories exhibit a long-tailed distribution, making it difficult to obtain samples for many categories. Furthermore, labeling samples and training models requires significant human and hardware resources. Few-shot object detection, on the other hand, utilizes the ample annotation information of the base class dataset to train the model, and then leverages the limited annotation information of the new class dataset and the prior knowledge of the base class model to predict new class objects, effectively overcoming these shortcomings.
[0003] However, due to the extremely small number of new class samples, their scale distribution is often sparse and severely imbalanced compared to the overall class distribution. To address this challenge, many algorithms employ the Feature Pyramid Network (FPN) structure to construct a multi-scale feature pyramid. However, this strategy, which generates feature maps of different scales through simple upsampling and downsampling operations, may not be able to fully capture the feature information of complex scenes, resulting in insufficient feature representation. In the prior art, Chinese patent CN116704260A proposes a meta-learning small sample image target detection method based on multi-scale feature fusion. This method uses an FPN network to achieve the fusion of feature maps of different scales in the support set, but it does not fully utilize the information of deep networks, resulting in poor performance for low-resolution or inaccurate feature maps. Therefore, to improve the model's detection performance for new class samples of different sizes, this invention proposes a multi-detector head, two-stage small sample target detection method based on an improved SPP structure. This method can extract features from different network layers more flexibly, thereby improving the model's performance. Summary of the Invention
[0004] The present invention aims to overcome the shortcomings of the existing technology by providing a multi-head, two-stage, small-sample target detection method based on an improved SPP structure.
[0005] The objective of this invention can be achieved through the following technical solutions:
[0006] This invention provides a multi-head, two-stage, small-sample target detection method based on an improved SPP structure, comprising the following steps:
[0007] Step S1: Using the two-stage object detection network Faster-RCNN as the base model, a base class detection model is constructed through a feature extraction backbone network, a region proposal network, an RoI pooling module, an object classifier, and a bounding box regressor.
[0008] Step S2: Freeze the parameters of the feature extraction backbone network, region proposal network, and RoI pooling module;
[0009] Step S3: Design a five-branch SPP structure based on convolutional layers, connection layers, and pooling layers;
[0010] Step S4: Constructing the detection network, including: adding the five-branch SPP structure to the new detection head branch; inserting the new detection head branch between the feature extraction backbone network and the RoI pooling module; and adding a second target classifier and a second bounding box regressor that are exactly the same as the target classifier and bounding box regressor.
[0011] Step S5: Use a small sample dataset of new classes with limited labeled information as input to fine-tune the parameters of the detection head part of the detection network;
[0012] Step S6: Input the dataset to be detected, which contains samples of the new class, into the finely tuned detection network to obtain the detection results.
[0013] Step S1 specifically includes:
[0014] Step S11: Construct the base class dataset;
[0015] Step S12, Model Selection: Select the two-stage object detection network Faster-RCNN as the base model, and load the pre-trained VGG16 network as the feature extraction backbone network;
[0016] Step S13, Model Construction: Construct a base class detection model by using a feature extraction backbone network, a region proposal network, an RoI pooling module, an object classifier, and a bounding box regressor, and set the hyperparameters of the base class detection model;
[0017] Step S14: Define the loss function: Select the classification cross-entropy loss as the classification loss, and select the smoothing L1 loss as the bounding box regression loss;
[0018] Step S15, Model Training: Input the base class dataset into the base class detection model, update the parameters of the base class detection model through the backpropagation algorithm to minimize the loss function, and dynamically adjust the hyperparameters of the base class detection model.
[0019] The parameter freezing in step S2 includes setting the "required_grad" attribute of the feature extraction backbone network, the region proposal network, and the RoI pooling module to "False".
[0020] Step S3 specifically includes:
[0021] Step S31, Initialization function design: First, determine the number of input channels C1 and the number of output channels C2, create the first convolutional layer, use a 1×1 convolutional kernel to reduce the number of input channels to C1 / 2, then create the second convolutional layer, use a 1×1 convolutional kernel to map the feature map after spatial pyramid pooling to the number of output channels;
[0022] Step S32, Hardware acceleration settings: In the initialization function, move the two created convolutional layers to the CUDA device for computation to accelerate the model training process;
[0023] Step S33, Pooling layer design: Create four pooling layers, including: two max pooling layers, using 3×3 and 5×5 pooling kernels respectively, and two average pooling layers, also using 3×3 and 5×5 pooling kernels respectively, for feature extraction at different scales.
[0024] Step S34, Forward Propagation Design: First, the input feature map is reduced in dimensionality by passing it through the first convolutional layer. Then, the dimensionality-reduced feature map is fed into four pooling layers to obtain pooled feature maps of different scales. Next, the result after passing through the first convolutional layer is concatenated with the four pooled feature maps. Finally, the concatenated feature map is fed into the second convolutional layer to obtain the feature map output from the five-branch SPP structure.
[0025] Furthermore, the five-branch SPP structure is calculated using the following formula:
[0026] X = conv1(X in )
[0027] Y1 = MaxPool1(X)
[0028] Y2 = MaxPool2(X)
[0029] Y3 = AvgPool1(X)
[0030] Y4 = AvgPool2(X)
[0031] X out =conv2(concat(X,Y1,Y2,Y3,Y4))
[0032] Among them, X in As input features, X out For output features.
[0033] Step S4 specifically includes:
[0034] Step S41: Extract features from the feature extraction backbone network and pass them to the five-branch SPP structure;
[0035] Step S42: The output of the five-branch SPP structure is used as a feature representation and passed to the RoI pooling module to map the features into a visual feature representation with a fixed number of channels and size.
[0036] Step S43: After the output of the RoI pooling module is processed by the activation layer and the fully connected layer, it is passed as input to the second target classifier and the second bounding box regressor to obtain the target classification and location regression.
[0037] Furthermore, the new detection head branch adopts the same loss function as the original detection head branch, and the loss functions of the two detection heads are added together to jointly optimize the network parameters.
[0038] Furthermore, the outputs of the new detection head branch and the original detection head branch are extracted together as the final detection result to obtain comprehensive detection results for targets at different scales and locations.
[0039] The number of samples in each category of the new class small sample dataset with limited annotation information in step S5 is specified by the task.
[0040] Furthermore, the number of samples included in each category is set to 1, 2, 3, 5, and 10, respectively.
[0041] Compared with the prior art, the present invention has the following beneficial effects:
[0042] 1) This invention extracts feature layers at different scales and designs a multi-detector head structure during the fine-tuning stage of new sample types. This allows for the fusion of multi-scale information, thereby improving the model's detection performance for targets of various sizes. By extracting features at multiple levels, the model can simultaneously capture information about objects at different scales in the image, which helps improve detection accuracy and reduce missed detections. Simultaneously, the multi-detector head structure enables the model to handle targets of various scales more flexibly, thus improving the model's efficiency.
[0043] 2) This invention integrates the Spatial Pyramid Pooling (SPP) structure into the newly added detection head, enabling more flexible and efficient handling of targets at various scales and further improving detection performance. The SPP structure allows the model to perform feature pooling at different scales, adapting to input features of different sizes without adjusting their dimensions. This design reduces computational costs, decreases the number of parameters, and improves the stability of feature representation, contributing to improved target detection accuracy. Through this method, the model can better adapt to targets of different scales in real-world scenarios, achieving more accurate detection.
[0044] 3) This invention improves upon the existing SPP structure, primarily by introducing average pooling and employing a relatively small pooling kernel. A relatively small kernel is chosen because the SPP structure is placed before the newly added detection head, which uses the output features of a shallow feature extraction network—that is, not belonging to a very deep layer—therefore, a large pooling kernel is not suitable. The advantages of introducing average pooling are twofold: First, unlike max pooling, average pooling focuses on the features of the entire region rather than just the most salient parts, which helps the model to understand the target's features more comprehensively. Second, using different types of pooling operations helps increase the model's robustness, thereby reducing the possibility of overfitting. Attached Figure Description
[0045] Figure 1 This is a flowchart illustrating the implementation of the present invention;
[0046] Figure 2 This is an overall framework diagram of the present invention;
[0047] Figure 3 This is a diagram of the improved five-branch SPP structure of the present invention. Detailed Implementation
[0048] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.
[0049] Example
[0050] This embodiment provides a multi-head, two-stage, small-sample target detection method based on an improved SPP structure, such as... Figure 1 and Figure 2 As shown, it includes the following steps:
[0051] Step 1: Train the two-stage object detection network Faster-RCNN using a base class dataset with a large amount of labeled information to obtain the base class detection model.
[0052] Specifically, the base class detection model in step 1 is obtained through the following calculation method:
[0053] Step 11) Construct the base class dataset. This example uses the PASCAL VOC 2012 dataset for experiments. The PASCAL VOC 2012 dataset contains 20 categories, from which 15 categories are selected as the base classes for this experiment. Specifically, the 15 selected base classes are: aeroplane, bicycle, boat, bottle, car, cat, chair, dining table, dog, horse, person, potted plant, sheep, train, and tvmonitor. For each base class, the number of samples is greater than 300.
[0054] Step 12) Model selection. The two-stage object detection network Faster-RCNN was selected as the base model, and a pre-trained VGG16 network was loaded as the feature extraction backbone network.
[0055] Step 13) Model Construction. Construct the base class detection model (Faster-RCNN model), which includes a Region Proposal Network (RPN), an object classifier, and a bounding box regressor. Set the model's hyperparameters, including: initial learning rate of 0.001 and batch size of 16.
[0056] Step 14) Define the loss function. Choose the classification cross-entropy loss as the classification loss and the smoothing L1 loss as the bounding box regression loss. Specifically:
[0057]
[0058]
[0059] in, This indicates that the model's predicted probability of the target class depends on the target's true class. This indicates the location of the bounding box predicted by the model. This indicates the actual location of the bounding box.
[0060] Step 15) Model Training. Input the base class dataset into the base class detection model, update the parameters of the base class detection model through the backpropagation algorithm to minimize the loss function, and dynamically adjust the hyperparameters of the base class detection model.
[0061] Step 2: Freeze the parameters of the feature extraction backbone network, RPN, and RoI parts in the base class detection model. Specifically, set the "required_grad" attribute of the above parameters to "False".
[0062] Step 3: Design a five-branch SPP structure, such as Figure 3As shown, this includes designing two convolutional layers based on the number of input and output channels, designing four pooling layers with different pooling methods and sizes, and designing forward propagation.
[0063] Specifically, the five-branch SPP structure in step 3 is designed using the following method:
[0064] Step 31) Initialization function design: First, determine the number of input channels (C1) and the number of output channels (C2). Then, create the first convolutional layer using a 1×1 convolutional kernel to reduce the number of input channels to C1 / 2. Next, create the second convolutional layer using a 1×1 convolutional kernel to map the feature map after spatial pyramid pooling to the number of output channels.
[0065] Step 32) Hardware acceleration setup: In the initialization function, move the two convolutional layers to the CUDA device (GPU) for computation to accelerate the model training process;
[0066] Step 33) Pooling Layer Design: Create four pooling layers for feature extraction at different scales, including two max pooling layers using 3×3 and 5×5 kernels respectively, and two average pooling layers using 3×3 and 5×5 kernels respectively. To keep the width and height dimensions of the features constant, the stride of all four pooling layers is set to 1, the fill parameter of the 3×3 kernel is set to 1, and the fill parameter of the 5×5 kernel is set to 2.
[0067] Step 34) Forward propagation design: First, the input features are reduced in dimensionality by passing through the first convolutional layer. Then, the dimensionality-reduced feature maps are fed into four pooling layers to obtain pooled feature maps of different scales. Next, the original feature map, i.e. the result after passing through the first convolutional layer, is concatenated with the four pooled feature maps. Finally, the concatenated feature map is fed into the second convolutional layer to obtain the final output feature map.
[0068] Specifically, the calculation formula for the five-branch SPP structure is as follows:
[0069] X = conv1(X in )
[0070] Y1 = MaxPool1(X)
[0071] Y2 = MaxPool2(X)
[0072] Y3 = AvgPool1(X)
[0073] Y4 = AvgPool2(X)
[0074] X out =conv2(concat(X,Y1,Y2,Y3,Y4))
[0075] Among them, X in Let X be the input feature. in If the dimension of the input SPP structure is C×H×W, then X, as the output feature after dimensionality reduction by the first convolutional layer, has a dimension of C / 2×H×W. Y1, Y2, Y3, and Y4 are the output features of the four pooling layers, each with a dimension of C / 2×H×W. Therefore, the concatenated feature dimension is 5C / 2×H×W. After dimensionality reduction by the second convolutional layer, the number of channels becomes the specified number. In this embodiment, the number of feature channels in the input SPP structure is 256, and the number of feature channels in the output SPP structure is 512.
[0076] Step 4: Add the improved five-branch SPP structure obtained in step S3 to the new detection head branch, and insert this branch into the existing network structure to construct the detection network.
[0077] Specifically, the improved five-branch SPP structure and the new detection head branch are incorporated into the existing network structure using the following method:
[0078] Step 41) Extract features from the intermediate layer of the feature extraction network (in this embodiment, the last convolutional layer in the VGG16 network with 256 output feature channels) and pass these features as input to the improved five-branch SPP structure mentioned in step S3.
[0079] Step 42) The output of the improved five-branch SPP structure is used as a feature representation and passed to the RoI pooling module of the original network to map the features into visual feature representations with a fixed number of channels and size.
[0080] Step 43) After the output of the RoI pooling module is processed by the activation layer and the fully connected layer, it is passed as input to the target classifier and bounding box regressor that are exactly the same as the original network to obtain the target classification and location regression.
[0081] Specifically, the new detection head branch uses the same loss function as the original detection head branch, that is, selecting the classification cross-entropy loss as the classification loss and the smoothing L1 loss as the bounding box regression loss. The loss functions of the two detection heads are added together to optimize the network parameters. In addition, the final detection result is a comprehensive extraction of the outputs from the two detection heads to obtain comprehensive detection results of targets at different scales and locations.
[0082] Step 5: Use a small dataset of new classes with limited labeled information as input to fine-tune the parameters of the detection head part of the detection network;
[0083] Specifically, the new class small sample dataset consists of 5 other classes in the PASCAL VOC 2012 dataset besides the 15 classes of the base class, namely: cow, bus, motorbike, sofa, and bird.
[0084] Step 6: Input the detection dataset containing the new class samples into the detection network to obtain the detection results. Specifically, the test dataset contains both base class data and new class data, but when calculating the detection accuracy, the detection accuracy of the base class and the new class will be counted separately to intuitively understand the network's detection performance for small sample classes.
[0085] The number of samples in each class of a new class small sample dataset with limited annotation information is specified by the task. In the k-shot task, each class of the small sample dataset will contain k samples, where k is usually equal to 5, 10, 30, etc., which is much smaller than the size of the base class dataset.
[0086] In this embodiment, the PASCAL VOC 2012 dataset was used as the test data. The number of samples for each small sample category was set to 1, 2, 3, 5, and 10 respectively, following the conventional settings for small sample target detection tasks. The detection accuracy results are shown in Tables 1 to 5. Furthermore, the experimental conditions and operations before the improvement were basically the same as after the improvement, except that the multi-detector head with the improved SPP structure was not introduced. All other experimental conditions and operations were exactly the same.
[0087] Table 1. Accuracy comparison before and after improvement under 1-shot settings
[0088] Cow Motorbike Bus Sofa Bird AP (before improvement) (%) 50.42 38.04 39.54 6.26 4.30 AP (after improvement) (%) 50.90 39.16 42.87 9.57 4.54
[0089] Table 2. Accuracy comparison before and after improvement under 2-shot settings
[0090] Cow Motorbike Bus Sofa Bird AP (before improvement) (%) 60.90 50.59 36.27 21.18 7.41 AP (after improvement) (%) 63.63 50.73 36.49 21.81 11.39
[0091] Table 3. Accuracy comparison before and after improvement under 3-shot settings
[0092] Cow Motorbike Bus Sofa Bird AP (before improvement) (%) 61.16 56.75 48.36 24.89 14.55 AP (after improvement) (%) 61.99 57.74 48.17 27.16 14.78
[0093] Table 4. Accuracy comparison before and after improvement under 5-shot settings
[0094] Cow Motorbike Bus Sofa Bird AP (before improvement) (%) 59.78 55.23 52.94 35.28 17.29 AP (after improvement) (%) 59.14 54.98 55.22 36.59 19.61
[0095] Table 5. Accuracy comparison before and after improvement under 10shot settings
[0096] Cow Motorbike Bus Sofa Bird AP (before improvement) (%) 66.44 52.47 58.37 37.47 17.18 AP (after improvement) (%) 66.30 52.35 58.77 40.46 22.28
[0097] This embodiment compares the detection accuracy of new classes before and after improvement under five conditions: the number of new class samples is 1, 2, 3, 5, and 10. The results show that the detection accuracy before improvement is significantly lower than that after improvement. Specifically, the mAP of the proposed method in this embodiment increases by 1.70%, 1.93%, 0.83%, 1.00%, and 1.65% respectively compared to the mAP before improvement when the number of new class samples is 1, 2, 3, 5, and 10. This demonstrates that the method proposed in this embodiment can effectively improve the network's ability to detect targets when the number of samples is small, better adapt to targets of different scales in real-world scenarios, and improve detection accuracy.
[0098] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.
Claims
1. A multi-head, two-stage, small-sample target detection method based on an improved SPP structure, characterized in that, Includes the following steps: Step S1: Using the two-stage object detection network Faster-RCNN as the base model, a base class detection model is constructed through a feature extraction backbone network, a region proposal network, an RoI pooling module, an object classifier, and a bounding box regressor. Step S2: Freeze the parameters of the feature extraction backbone network, region proposal network, and RoI pooling module; Step S3: Design a five-branch SPP structure based on convolutional layers, connection layers, and pooling layers, specifically including: Step S31, Initialization Function Design: First, determine the number of input channels. and number of output channels Create the first convolutional layer using a 1×1 convolutional kernel to reduce the number of input channels to [a smaller number]. Next, a second convolutional layer is created, using a 1×1 convolutional kernel, to map the feature map after spatial pyramid pooling to the number of output channels; Step S32, Hardware acceleration settings: In the initialization function, move the two created convolutional layers to the CUDA device for computation to accelerate the model training process; Step S33, Pooling layer design: Create four pooling layers, including: two max pooling layers, using 3×3 and 5×5 pooling kernels respectively, and two average pooling layers, also using 3×3 and 5×5 pooling kernels respectively, for feature extraction at different scales. Step S34, Forward Propagation Design: First, the input feature map is reduced in dimensionality by passing it through the first convolutional layer. Then, the dimensionality-reduced feature map is fed into four pooling layers to obtain pooled feature maps of different scales. Next, the result after passing through the first convolutional layer is concatenated with the four pooled feature maps. Finally, the concatenated feature map is fed into the second convolutional layer to obtain the feature map output from the five-branch SPP structure. Step S4: Constructing the detection network, including: adding the five-branch SPP structure to the new detection head branch; inserting the new detection head branch between the feature extraction backbone network and the RoI pooling module; and adding a second target classifier and a second bounding box regressor that are exactly the same as the target classifier and bounding box regressor. Step S5: Use a small sample dataset of new classes with limited labeled information as input to fine-tune the parameters of the detection head part of the detection network; Step S6: Input the dataset to be detected, which contains samples of the new class, into the finely tuned detection network to obtain the detection results.
2. The multi-head, two-stage, small-sample target detection method based on an improved SPP structure according to claim 1, characterized in that, Step S1 specifically includes: Step S11: Construct the base class dataset; Step S12, Model Selection: Select the two-stage object detection network Faster-RCNN as the base model, and load the pre-trained VGG16 network as the feature extraction backbone network; Step S13, Model Construction: Construct a base class detection model by using a feature extraction backbone network, a region proposal network, an RoI pooling module, an object classifier, and a bounding box regressor, and set the hyperparameters of the base class detection model; Step S14: Define the loss function: Select the classification cross-entropy loss as the classification loss, and select the smoothing L1 loss as the bounding box regression loss; Step S15, Model Training: Input the base class dataset into the base class detection model, update the parameters of the base class detection model through the backpropagation algorithm to minimize the loss function, and dynamically adjust the hyperparameters of the base class detection model.
3. The multi-head, two-stage, small-sample target detection method based on an improved SPP structure according to claim 1, characterized in that, The parameter freezing in step S2 includes setting the "required_grad" attribute of the feature extraction backbone network, region proposal network, and RoI pooling module to "False".
4. The multi-head, two-stage, small-sample target detection method based on an improved SPP structure according to claim 1, characterized in that, The five-branch SPP structure is calculated using the following formula: in, For input features, For output features.
5. The multi-head, two-stage, small-sample target detection method based on an improved SPP structure according to claim 1, characterized in that, Step S4 specifically includes: Step S41: Extract features from the feature extraction backbone network and pass them to the five-branch SPP structure; Step S42: The output of the five-branch SPP structure is used as a feature representation and passed to the RoI pooling module to map the features into a visual feature representation with a fixed number of channels and size. Step S43: After the output of the RoI pooling module is processed by the activation layer and the fully connected layer, it is passed as input to the second target classifier and the second bounding box regressor to obtain the target classification and location regression.
6. The multi-head, two-stage, small-sample target detection method based on an improved SPP structure according to claim 5, characterized in that, The new detection head branch uses the same loss function as the original detection head branch. The loss functions of the two detection heads are added together to optimize the network parameters.
7. The multi-head, two-stage, small-sample target detection method based on an improved SPP structure according to claim 6, characterized in that, The outputs of the new detection head branch and the original detection head branch are extracted together as the final detection result to obtain comprehensive detection results for targets at different scales and locations.
8. The multi-head, two-stage, small-sample target detection method based on an improved SPP structure according to claim 1, characterized in that, The number of samples in each category of the new class small sample dataset with limited annotation information is specified by the task.
9. A multi-head, two-stage, small-sample target detection method based on an improved SPP structure according to claim 8, characterized in that, The number of samples in each category is set to 1, 2, 3, 5, and 10, respectively.