A few-shot object detection method based on Meta RCNN

Abstract

The present invention belongs to the field of computer vision technology, and specifically relates to a few-shot target detection method based on MetaRCNN. To overcome the shortcomings of existing detection technologies in terms of detection accuracy and poor generalization ability to new classes, the deep learning network constructed by the present invention includes ResNet as the backbone network, a region proposal network to extract image target proposal boxes, and RoIAlign to process basic features and regions of interest. After feature aggregation, the features are fed into the predictor. The output is then fed into the objective function along with the corresponding true value to calculate the loss. Backpropagation is used to adjust the parameters until the objective function converges. The collected and constructed dataset is then input into the trained deep network model to obtain pixel-level prediction output, and the prediction results are measured by the AP value.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and specifically relates to a few-sample target detection method based on Meta RCNN. Background Art

[0002] In recent years, deep learning-based image object detection technology has achieved remarkable success, and many mature detection models have emerged. However, these models all require large numbers of labeled samples for training. Even the best methods struggle to generalize well to unknown categories that the system has never encountered during training or for which only a small number of examples are available. Furthermore, in real-world scenarios, it is often difficult to obtain large-scale, high-quality labeled samples, limiting their application in specific fields. Furthermore, acquiring and accurately labeling thousands of valid data points is costly. This is particularly true in fields such as medicine, military, and defense, where data is scarce and requires highly skilled experts to label. Conventional deep learning fine-tuning approaches are already helpless when faced with this challenge of only having a single or small number of samples.

[0003] Therefore, detecting objects with very few samples is a highly practical problem and is attracting increasing attention. The problem of few-shot object detection was proposed to address the problem of object detection when training samples are limited. Traditional object detection algorithms rely on abundant training samples with labeled data. However, few-shot object detection suffers from a lack of training samples. Consequently, the learned object detection network performs poorly, with detection accuracy lower than that of traditional object detection algorithms. Therefore, generalizing the trained object detection network to new classes is a key research focus in few-shot object detection algorithms.

[0004] Since the research on few-shot object detection is still in its infancy, in order to better generalize to new types of objects, it is necessary to identify some key issues that need to be addressed:

[0005] 1) When extracting and processing features from image data, it is easily affected by noise-like information features, especially in the few-shot setting where only a few labeled samples are provided for the new class;

[0006] 2) Fine-tuning the object detection network with only a few instances of new categories can easily lead to overfitting.

[0007] Meta R-CNN introduces meta-learning into a two-stage object detection approach. Leveraging the RoI features of Faster R-CNN and Mask R-CNN, it partially addresses a long-standing problem in the study of few-shot object detection algorithms in complex backgrounds and images with multiple objects. The network incorporates a Predictor-Head Remodeling Network (PRN), which shares a backbone with Faster R-CNN or Mask R-CNN. The PRN is fully convolutional and receives few-shot objects from base classes and novel classes, along with their bounding boxes or masks, to infer a class attention vector corresponding to the class of the few-shot input object. This makes Meta R-CNN a lightweight network that improves the generalization capabilities of traditional Faster R-CNN or Mask R-CNN on novel classes. Summary of the Invention

[0008] Existing few-shot target detection methods, when performing target detection based on large datasets, usually use new classes (uncommon classes) with a small amount of annotation information to fine-tune the network model. The lack of annotation information leads to a lack of available knowledge learned by the network, which increases the difficulty of accurately detecting new classes.

[0009] In order to overcome the shortcomings of existing few-shot object detection methods, such as the lack of research and low accuracy in detecting new classes, the present invention provides a few-shot object detection method based on Meta RCNN, which mainly solves the following problems: (1) the feature information extracted by the backbone network from the input data of the new class is relatively poor; (2) the generalization ability of the network trained on the base class on the new class after fine-tuning is poor, and the separability between the base class and the new class is poor.

[0010] In order to achieve the above object, the present invention adopts the following technical solutions:

[0011] A few-shot object detection method based on Meta RCNN includes the following steps:

[0012] Step 1: Collect raw data: Use an RGB camera to shoot the object to be predicted and obtain an RGB image;

[0013] Step 2: Generate a training dataset: Obtain bounding box information data and mask data information corresponding to the RGB image to construct a dataset in PASCAL VOC format;

[0014] Step 3: Build a deep learning network model: including a data input module, a feature processing module, a feature aggregation module, and a predictor module connected in sequence;

[0015] The data input module uses a query image and a class data image as input, wherein the query image is a 1-dimensional 224×224 depth map, and the class data image is a binary mask image of the region of interest generated based on image feature prediction;

[0016] The feature processing module includes a query feature processing module and a class feature processing module. Both the query feature processing module and the class feature processing module include a backbone part, an RPN network and a suggestion-level feature alignment module. In the query feature processing module, the input image is first subjected to a two-dimensional convolution with a convolution kernel of 7. The feature map obtained by the convolution is sequentially batch normalized, ReLU activated, and two-dimensional maximum pooled, and fed to the hierarchical network for processing to obtain basic features. Subsequently, the extracted basic feature map is fed to the RPN network together with the image information, the true bounding box, and the number of bounding boxes to obtain the predicted region of interest features. The suggestion-level feature alignment module performs roi based on the predicted region of interest features. Pooling: Three POOLING_MODE modes are proposed for the pooling method. The final pooling method is selected according to the actual effects of different modes. The obtained feature map is fed into the hierarchical network with the corresponding number of channels to obtain the final query feature. In the class feature processing module, a PRN sharing the FasterRCNN backbone network is introduced. The PRN receives image data to infer their class attention vectors. The input image is first processed by the backbone part to obtain the basic features of the class data. Then the basic features are max-pooled, hierarchical network with the corresponding number of channels and sigmoid processing to generate the class attention vector, i.e., the class data feature. The hierarchical network is created and generated by inputting the number and type of blocks in each layer.

[0017] The feature aggregation module performs channel multiplication and feature subtraction on the class data features and query features obtained by the feature processing module, and concatenates the obtained results with the query features by channel to complete feature aggregation and obtain the aggregated features. The feature aggregation formula is as follows:

[0018] Α(f roi ,f cls )=[f roi ⊙f cls ,f roi -f cls ,f roi ] (1)

[0019] Among them, f roi represents the query feature, f cls Represents class data characteristics;

[0020] Predictor module: for bounding box classification and regression, including bounding box classifier and bounding box regressor, both of which are implemented as two fully connected layers of size 4096, outputting N train=|C train | classification scores and N corresponding to each RoI train Frame regression;

[0021] Step 4: Train the deep learning network model: pre-process the images in the generated training dataset and input them into the deep learning network model to obtain the output images and the corresponding true values. Figure 1 The objective function is fed into the system to calculate the loss, and the parameters are adjusted through backpropagation until the objective function converges.

[0022] Step 5, output: The final output image contains object probability and bounding box parameters. The results of bounding box classification and regression output are processed to obtain bounding box information. At the same time, the aggregated features are fed to a fully connected layer with the input feature number equal to the aggregated feature dimension and the output feature number equal to the number of categories to calculate the object category probability. The labeled image containing bounding box information and mask information is input into the trained deep network model to obtain the predicted output.

[0023] Furthermore, the bounding box information data and mask data information corresponding to the RGB image in step 2 are obtained by annotating the captured RGB image using Labelimg and Labelme annotation tools. The specific steps are as follows:

[0024] First, use the Labelimg annotation tool to annotate the corresponding bounding box of the captured image and indicate the category of the object in the box. The annotation generates an xml file in the corresponding format of PASCAL VOC. Then use the Labelme annotation tool to annotate the image contour and category and generate the corresponding json file.

[0025] Furthermore, the bounding box information data and mask data information corresponding to the RGB image in step 2 are obtained by using a target detection algorithm and an image segmentation algorithm. The specific steps are as follows:

[0026] Add corresponding category data and missing categories to the yaml files of the training data and dataset, then use the traditional target detection algorithm and the existing weight file for training to generate a new weight file. Based on this, the constructed dataset is detected, and the detection generates an image with bounding box annotations and saves its corresponding txt label file. At the same time, the instance segmentation algorithm is used to process the image and obtain instance segmentation type data.

[0027] Furthermore, the bounding box classifier in step 3 sets two classifiers to decouple the classification branches of the base class and the new class. One classifier is used only to identify the base class, and the other is used to identify the new class and the background. The outputs of the two classifiers are then merged. The weight matrix of the classifier is W = [ω1,ω2,…,ω c], the classification score of the i-th RoI and class c is shown in formula (2):

[0028]

[0029] Where α is the scaling factor, is the classification weight vector, and d is the dimension of the aggregated features.

[0030] Furthermore, the classifier uses a cosine similarity-based classifier.

[0031] Furthermore, the output image obtained in step 4 and the corresponding true value Figure 1 The objective function is sent to calculate the loss. The specific steps are:

[0032] The cross entropy loss function is used for both RPN network classification and final target classification. The network classification score and the corresponding label are fed into the loss function for calculation. The smooth L1 loss function is used for both RPN network regression and final bounding box regression. The predicted bounding box, the actual bounding box and the weight of the balance loss are fed into the loss function for calculation. The meta-loss adopts the cross entropy loss function. After the attention vector obtains the corresponding score, it is fed into the function together with the PRN classification output for calculation. The loss function is shown in formula (4):

[0033] L=L rpn +L cls +L loc +L meta (4)

[0034] In formula (4), L is the total loss function, L rpn Applied to the output of the RPN network to distinguish foreground from background and refine the proposal, L loc represents the Huber loss for box regression, L meta is the cross entropy loss, which encourages the diversification of class features of different categories, L cls Represents the cross entropy loss function for the classification of base and new class boxes, trying to maximize With any other class The margin of the decision boundary between is defined as follows:

[0035]

[0036]

[0037] in, and s j Class and The classification score of , ε is a constant used to maintain the stability of the value;

[0038]

[0039] Among them, α, β, and γ are hyperparameters that control the margins of base class samples, new class samples, and negative samples, respectively.

[0040] Furthermore, the output in step 5 includes: image category prediction score classscore, the image contains the bounding box position information corresponding to the object in the original image: the center point x, y coordinates and width w, height h, and then obtains the image coordinate prediction result, as shown in formula (3):

[0041] (cls i,c ,box i,c )=P(Α(f roi ,f cls )) (3)

[0042] Where c∈C train , C train is the set of all training classes, cls i,c and box i,c are the predicted classification scores and object locations for the ith RoI and class c in the query image.

[0043] Compared with the prior art, the present invention has the following advantages:

[0044] 1. The backbone network uses the ResNet module, supplemented by its weight files pre-trained on ImageNet, to accelerate network training, achieve faster model convergence, reduce training time, and improve model performance. The network also uses bounding box-annotated image data and mask-annotated data as input. These two types of data are referred to as query data and class data, respectively. The network integrates features from both types of data, enriching the features extracted during network training and fine-tuning, and providing more comprehensive image information. This ensures that after the first phase of basic training and the second phase of network fine-tuning, the deep learning network achieves more accurate detection of sample features.

[0045] 2. Aggregate the feature vectors generated by processing the two input data. Building on the reweighted feature aggregation method of Meta R-CNN, the new features generated by channel-wise concatenation of the two types of base features, combined with the query features, are aggregated. Feature subtraction is a different but equally effective method for measuring the similarity between image features. The image query features themselves are not reweighted but still contain relevant information.

[0046] 3. During the classification phase, to ensure separability between the base and new classes, the classification branches of the base and new classes are decoupled; and to further enhance inter-class separability across all classes, a new marginal loss function is added. Using Meta RCNN-based few-shot object detection, we first experimented on a self-constructed dataset, achieving a maximum average precision of 64.9 and a minimum average precision of 32.3. We then experimented on public traditional object detection datasets, PASCAL VOC and MS-COCO, achieving a maximum average precision of 64.1 and a minimum average precision of 30.1 under three class segmentation methods on PASCAL VOC, respectively, as well as average precision and average recall on MS-COCO, outperforming existing few-shot object detection methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 is a schematic diagram of image annotation;

[0048] Figure 2 This is the overall flow chart of this method;

[0049] Figure 3 This is the prediction result of this embodiment. DETAILED DESCRIPTION

[0050] Example 1

[0051] like Figure 2 As shown in FIG, a few-shot target detection method based on Meta RCNN is characterized by comprising the following steps:

[0052] Step 1: Collect raw data: Use an RGB camera to capture the object to be predicted and obtain an RGB image. The image capture and acquisition process does not require image size, but requires that the clarity of the captured image be guaranteed for image annotation.

[0053] Step 2: Generate training dataset: The few-shot target detection method based on Meta RCNN requires a dataset containing image bounding box and mask annotation information. Therefore, Labelimg and Labelme annotation tools are used to annotate the captured RGB images to obtain the bounding box information data and mask data information corresponding to the RGB images (such as Figure 1 As shown), to construct the PASCAL VOC format dataset, the specific steps are:

[0054] First, use the Labelimg annotation tool to annotate the corresponding bounding box of the captured image and indicate the category of the object in the box. The annotation generates an xml file in the corresponding format of PASCAL VOC. Then use the Labelme annotation tool to annotate the image contour and category and generate the corresponding json file. The xml file contains the image category and bounding box coordinate information, and the json file contains the image category and contour annotation coordinate point information.

[0055] In addition, you can also use traditional target detection algorithms and image segmentation algorithms with higher accuracy to achieve image annotation. To ensure the accuracy of image annotation, the prediction results need to be calibrated after the program is finished running to ensure that the image category information, bounding box coordinate information, and mask information are consistent with the original image. The specific steps are as follows:

[0056] Add corresponding category data and missing categories to the yaml files of the training data and dataset, and then use the traditional target detection algorithm and the existing weight file for training to generate a new weight file. Based on this, the constructed dataset is detected, and the detection generates an image with a bounding box annotation and saves its corresponding txt label file. At the same time, the instance segmentation algorithm (yolov7 segmentation algorithm) is used to process the image and obtain instance segmentation type data.

[0057] Step 3: Build a deep learning network model: including a data input module, a feature processing module, a feature aggregation module, and a predictor module connected in sequence;

[0058] The data input module uses a query image and a class data image as input, where the query image is a 1-dimensional 224×224 depth map, and the class data image is a binary mask image of the region of interest generated based on image feature prediction. The input order of the data set is disrupted to reduce the difficulty of model convergence and improve model performance.

[0059] The feature processing module includes a query feature processing module and a class feature processing module. Both the query feature processing module and the class feature processing module include a backbone part, an RPN network, and a suggestion-level feature alignment module. In the query feature processing module, the input image is first expanded to 64 layers through a two-dimensional convolution with a convolution kernel of 7. The feature maps obtained by the convolution are sequentially batch normalized, ReLU activated, and two-dimensional maximum pooled, and fed into a network structure (hierarchical network) with output channels of 64, 128, and 256 respectively. The networks with output channels of 128 and 256 are first downsampled by a two-dimensional convolution with a kernel of 1. The number of channels is doubled after each downsampling. Batch normalization is performed after each convolution layer. The features obtained after downsampling are added to the input through residual addition to obtain a basic feature map. The extracted basic feature map is then fed into the RPN network together with image information, true bounding box and the number of bounding boxes to obtain the predicted region of interest features. The proposal-level feature alignment module performs roi pooling based on the predicted region of interest features. Three POOLING_MODE modes are proposed for the pooling method. The final pooling method is selected according to the actual effect of different modes. The obtained feature map is fed to the hierarchical network with the corresponding number of channels to obtain the final query feature. In the class feature processing module, a PRN sharing the Faster RCNN backbone network is introduced. The PRN receives image data to infer their class attention vectors. The input image is first processed by the backbone part to obtain the basic features of the class data. Then the basic features are subjected to maximum pooling, a hierarchical network with a specified number of output channels and sigmoid processing to generate the class attention vector, i.e., the class data feature. The hierarchical network is created and generated by inputting the number of blocks and their types in each layer.

[0060] The feature aggregation module performs channel multiplication and feature subtraction on the class data features and query features obtained by the feature processing module, and concatenates the obtained results with the query features by channel. The two new features obtained by processing are transmitted to the fully connected layer with input and output dimensions of 2048 and 1024 respectively, and batch normalization and ReLU activation are performed in sequence. Then, the two tensor sequences output above are connected on dimension 1, and the obtained features are connected with the query features originally input on dimension 1 to complete feature aggregation and obtain the aggregated features. The features are used to realize the final bounding box prediction and classification score calculation. The feature aggregation formula is as follows:

[0061] Α(f roi ,f cls )=[f roi ⊙f cls ,f roi -f cls ,f roi ] (1)

[0062] Among them, f roi represents the query feature, f cls Represents class data characteristics;

[0063] Predictor module: for bounding box classification and regression, including bounding box classifier and bounding box regressor, both of which are implemented as two fully connected layers of size 4096, outputting N train =|C train | classification scores and N corresponding to each RoI train The bounding box classifier sets two classifiers to decouple the classification branches of the base class and the new class. One classifier is used only to identify the base class, and the other is used to identify the new class and background. The outputs of the two classifiers are then merged. Both use a classifier based on cosine similarity, and the classifier weight matrix is ​​W = [ω1,ω2,…,ω c ], the classification score of the i-th RoI and class c is shown in formula (2):

[0064]

[0065] in, is the classification weight vector, d is the dimension of the aggregated features, and α is the scaling factor, all set to 20;

[0066] Step 4: Train the deep learning network model: pre-process the images in the generated training dataset and input them into the deep learning network model to obtain the output images and the corresponding true values. Figure 1 The objective function is fed into the system to calculate the loss, and the parameters are adjusted through backpropagation until the objective function converges. The specific steps are as follows:

[0067] The cross entropy loss function is used for both RPN network classification and final target classification. The network classification score and the corresponding label are fed into the loss function for calculation. The smooth L1 loss function is used for both RPN network regression and final bounding box regression. The predicted bounding box, the actual bounding box and the weight of the balance loss are fed into the loss function for calculation. The meta-loss adopts the cross entropy loss function. After the attention vector obtains the corresponding score, it is fed into the function together with the PRN classification output for calculation. The loss function is shown in formula (4):

[0068] L=L rpn +L cls +L loc +L meta (4)

[0069] In formula (4), L is the total loss function, L rpn Applied to the output of the RPN network to distinguish foreground from background and refine the proposal, L loc represents the Huber loss for box regression, L metais the cross entropy loss, which encourages the diversification of class features of different categories, L cls Represents the cross entropy loss function for the classification of base and new class boxes, trying to maximize With any other class The margin of the decision boundary between is defined as follows:

[0070]

[0071]

[0072] in, and s j Class and The classification score, ε is a constant (1e -7 ), used to maintain numerical stability;

[0073]

[0074] Among them, α, β, and γ are hyperparameters that control the margins of base class samples, novel class samples, and negative samples, respectively. Intuitively, β is larger than α because novel classes are more challenging, while γ is a very small value to balance the overwhelming negative samples.

[0075] Step 5, output (such as Figure 3 The final output image contains object probability and bounding box parameters. The results of bounding box classification and regression output are processed to obtain bounding box information. At the same time, the aggregated features are fed to a fully connected layer with the input feature number being the aggregated feature dimension and the output feature number being the number of categories to calculate the object category probability. The labeled image containing bounding box information and mask information is input to the trained deep network model to obtain the predicted output: image category prediction score class score. The image contains the bounding box position information corresponding to the object in the original image: the center point x, y coordinates and width w, height h, and then the image coordinate prediction result is obtained, as shown in formula (3):

[0076] (cls i,c ,box i,c )=P(Α(f roi ,f cls )) (3)

[0077] Where c∈C train , C train is the set of all training classes, cls i,c and box i,c are the predicted classification scores and object locations for the ith RoI and class c in the query image.

[0078] Example 2

[0079] Dataset experiment evaluation criteria:

[0080] This embodiment uses Average Precision (AP) and Mean Average Precision (mAP) as evaluation metrics for the network designed by the present invention. The AP is usually reported at a single Intersection over Union (IoU) threshold of 0.5, while mAP represents the average AP at multiple IoU thresholds ranging from 0.5 to 0.95:

[0081] 1. The IoU threshold between the predicted bounding box and the true annotation bounding box is 0.5, as shown in the following formula:

[0082]

[0083] Where A represents the predicted box, B represents the ground-truth box, A∩B represents the intersection of the two, and A∪B represents the union of the two. When the IoU is greater than the threshold of 0.5, it is considered a successful detection; otherwise, it is considered an error.

[0084] 2. Precision: The ratio of correctly predicted targets among all targets predicted by the model, i.e., the number of correctly predicted positive targets / all positive targets found; Recall: The ratio of correctly predicted targets among all true (positive) targets, i.e., the number of correctly predicted positive targets / all positive targets that should have been found. AP: The area under the PR curve. As shown in the following formula:

[0085]

[0086]

[0087] Among them, TP is true positive (predicting the positive class as the positive class), FP is false positive (predicting the positive class as the negative class), TN is true negative (predicting the negative class as the negative class), and FN is false negative (predicting the negative class as the positive class).

[0088] Dataset experimental environment:

[0089] This example completes the data set experiment on the Ubuntu 16.04 operating system. The specific configuration includes CPU E5-2683 v3, frequency is 2.00GHz, 16GB memory, NVIDIA Tesla 2070SUPER graphics card with 8G video memory, CUDA 8.0 acceleration toolbox, Pytorch 0.4.0 deep learning framework.

[0090] Dataset experiments:

[0091] This paper presents experimental results on the traditional object detection datasets PASCAL VOC and MS-COCO. PASCAL VOC: Testing is performed using the VOC2007 test set and training is performed using the VOC0712 trainval set. Three common few-shot splits are used to randomly select 5 classes from each of the 20 object categories in this dataset as novel classes, while retaining the remaining 15 classes as base classes. Evaluation is performed on these three splits, assuming that only K annotated bounding boxes are provided for each novel class during training, where K equals 1, 2, 3, 5, or 10. MS-COCO: Testing is performed using 5,000 images from the mini-val set and training using the remaining 118,287 images from the training-val set. Of the 80 object categories, 20 classes commonly found in PASCAL VOC are selected as novel classes, while the remaining 60 classes are used as base classes. For this dataset, testing is performed on K = 10 or 30 annotated bounding boxes for each novel class.

[0092] Use SGD optimizer with an initial learning rate of 10 -3 , the batch size is 4, the weight decay and momentum are set to 0.0005 and 0.9 respectively. In the basic training phase, 20 epochs are trained, and the learning rate is divided by 10 after every 5 epochs. In the fine-tuning phase, 5 epochs are trained with a learning rate of 10 -3 , and then train for 4 epochs with a learning rate of 10 -4 For anchor box scales, PASCAL VOC uses three scales (128 2 ,256 2 ,512 2 ), and add a fourth scale of 64 for MS-COCO 2 The three aspect ratios of the anchors are set to 1:2, 1:1, and 2:1. Data is augmented by horizontal flipping. The experimental results are as follows.

[0093] Table 1 Evaluation of few-shot object detection on PASCAL VOC

[0094]

[0095]

[0096] Table 2 Evaluation of few-shot object detection on MS-COCO

[0097]

[0098]

[0099]

[0100] Ablation experiment:

[0101] To verify the effectiveness of the feature aggregation method used in this example, ablation experiments on the PASCAL VOC dataset are shown in Table 3. Using three different class splits for the PASCAL VOC dataset, the performance of few-shot object detection was measured on new classes with 3 and 10 labeled samples. As can be seen, the performance of the last aggregation method significantly outperforms the previous methods. This demonstrates the importance of the method used in this example in few-shot object detection.

[0102] Table 3 Ablation experiments of feature aggregation schemes

[0103]

[0104]

[0105] This embodiment is tested on the PASCAL VOC and MS-COCO datasets. Compared with a series of methods proposed previously, the average precision of the present invention on these two datasets has been improved to a certain extent.

Claims

1. A few-shot target detection method based on MetaRCNN, characterized in that: The following steps are involved: Step 1: Collect raw data: Use an RGB camera to shoot the object to be predicted and obtain an RGB image; Step 2: Generate a training dataset: Obtain bounding box information data and mask data information corresponding to the RGB image to construct a dataset in PASCAL VOC format; Step 3: Build a deep learning network model: including a data input module, a feature processing module, a feature aggregation module, and a predictor module connected in sequence; The data input module uses a query image and a class data image as input, wherein the query image is a 1-dimensional 224×224 depth map, and the class data image is a binary mask image of the region of interest generated based on image feature prediction; The feature processing module includes a query feature processing module and a class feature processing module. Both the query feature processing module and the class feature processing module include a backbone part, an RPN network and a suggestion-level feature alignment module. In the query feature processing module, the input image is first subjected to a two-dimensional convolution with a convolution kernel of 7. The feature map obtained by the convolution is sequentially batch normalized, ReLU activated, and two-dimensional maximum pooled, and fed to the hierarchical network for processing to obtain basic features. Subsequently, the extracted basic feature map is fed to the RPN network together with the image information, the true bounding box, and the number of bounding boxes to obtain the predicted region of interest features. The suggestion-level feature alignment module performs roi based on the predicted region of interest features. Pooling: Three POOLING_MODE modes are proposed for the pooling method. The final pooling method is selected according to the actual effects of different modes. The obtained feature map is fed into the hierarchical network with the corresponding number of channels to obtain the final query feature. In the class feature processing module, a PRN sharing the FasterRCNN backbone network is introduced. The PRN receives image data to infer their class attention vectors. The input image is first processed by the backbone part to obtain the basic features of the class data. Then the basic features are max-pooled, hierarchical network with the corresponding number of channels and sigmoid processing to generate the class attention vector, i.e., the class data feature. The hierarchical network is created and generated by inputting the number and type of blocks in each layer. The feature aggregation module performs channel multiplication and feature subtraction on the class data features and query features obtained by the feature processing module, and concatenates the obtained results with the query features by channel to complete feature aggregation and obtain the aggregated features. The feature aggregation formula is as follows: Α(f roi ,f cls )=[f roi ⊙f cls ,f roi -f cls ,f roi ] (1) Among them, f roi represents the query feature, f cls Represents class data characteristics; Predictor module: for bounding box classification and regression, including bounding box classifier and bounding box regressor, both of which are implemented as two fully connected layers of size 4096, outputting N train =|C train | classification scores and N corresponding to each RoI train Frame regression; Step 4: Train the deep learning network model: Preprocess the images in the generated training dataset and input them into the deep learning network model. The output images and the corresponding ground truth images are fed into the objective function to calculate the loss, and the parameters are adjusted by backpropagation until the objective function converges. Step 5, output: The final output image contains object probability and bounding box parameters. The results of bounding box classification and regression output are processed to obtain bounding box information. At the same time, the aggregated features are fed to a fully connected layer with the input feature number equal to the aggregated feature dimension and the output feature number equal to the number of categories to calculate the object category probability. The labeled image containing bounding box information and mask information is input into the trained deep network model to obtain the predicted output.

2. The method for few-shot target detection based on Meta RCNN according to claim 1, wherein: The bounding box information data and mask data information corresponding to the RGB image in step 2 are obtained by annotating the captured RGB image using Labelimg and Labelme annotation tools. The specific steps are as follows: First, use the Labelimg annotation tool to annotate the corresponding bounding box of the captured image and indicate the category of the object in the box. The annotation generates an xml file in the corresponding format of PASCAL VOC. Then use the Labelme annotation tool to annotate the image contour and category and generate the corresponding json file.

3. The few-shot target detection method based on Meta RCNN according to claim 1, wherein: The bounding box information data and mask data information corresponding to the RGB image in step 2 are obtained by using the target detection algorithm and the image segmentation algorithm. The specific steps are as follows: Add corresponding category data and missing categories to the yaml files of the training data and dataset, then use the traditional target detection algorithm and the existing weight file for training to generate a new weight file. Based on this, the constructed dataset is detected, and the detection generates an image with bounding box annotations and saves its corresponding txt label file. At the same time, the instance segmentation algorithm is used to process the image and obtain instance segmentation type data.

4. The method for few-shot target detection based on Meta RCNN according to claim 1, wherein: In step 3, the bounding box classifier sets two classifiers to decouple the classification branches of the base class and the new class. One classifier is only used to identify the base class, and the other is used to identify the new class and the background. The outputs of the two classifiers are then merged. The weight matrix of the classifier is W = [ω1,ω2,…,ω c ], the classification score of the i-th RoI and class c is shown in formula (2): Where α is the scaling factor, is the classification weight vector, and d is the dimension of the aggregated features.

5. The method for few-shot target detection based on Meta RCNN according to claim 4, wherein: The classifier uses a cosine similarity-based classifier.

6. The method for few-shot target detection based on Meta RCNN according to claim 1, wherein: The output image obtained in step 4 and the corresponding ground truth image are sent to the objective function to calculate the loss. The specific steps are: The cross entropy loss function is used for both RPN network classification and final target classification. The network classification score and the corresponding label are fed into the loss function for calculation. The smooth L1 loss function is used for both RPN network regression and final bounding box regression. The predicted bounding box, the actual bounding box and the weight of the balance loss are fed into the loss function for calculation. The meta-loss adopts the cross entropy loss function. After the attention vector obtains the corresponding score, it is fed into the function together with the PRN classification output for calculation. The loss function is shown in formula (4): L=L rpn +L cls +L loc +L meta (4) In formula (4), L is the total loss function, L rpn Applied to the output of the RPN network to distinguish foreground from background and refine the proposal, L loc represents the Huber loss for box regression, L meta is the cross entropy loss, which encourages the diversification of class features of different categories, L cls Represents the cross entropy loss function for the classification of base and new class boxes, trying to maximize C yi With any other class The margin of the decision boundary between is defined as follows: in, and s j Class and The classification score of , ε is a constant used to maintain the stability of the value; Among them, α, β, and γ are hyperparameters that control the margins of base class samples, new class samples, and negative samples, respectively.

7. The method for few-shot target detection based on Meta RCNN according to claim 1, wherein: The output in step 5 includes: image category prediction score class score, the image contains the bounding box position information corresponding to the object in the original image: the center point x, y coordinates and width w, height h, and then obtains the image coordinate prediction result, as shown in formula (3): (cls i,c ,box i,c )=P(Α(f roi ,f cls )) (3) Where c∈C train , C train is the set of all training classes, cls i,c and box i,c are the predicted classification scores and object locations for the ith RoI and class c in the query image.

Images