A small sample target detection method based on attention and contrast learning

Abstract

The application relates to a small sample target detection method based on attention and contrast learning, and belongs to the fields of artificial intelligence and image processing. The application is a small sample target detection method combining data enhancement, an attention region proposal network (Attention RPN) and contrast learning. The method is based on a FasterR-CNN network and comprises a small sample mosaic (Few-shot Mosaic) data enhancement module used for enriching the contrast between small sample background instances and novel classes (Novel classes) and base class (Base classes) instances, an Attention RPN module based on regression coordinate compensation for strengthening the attention of the model to the foreground, and a contrast learning module for improving the expression of instance-level features. The application improves the detection accuracy of novel classes of FasterR-CNN on small samples while maintaining high base class detection accuracy. The application reduces the dependence of FasterR-CNN on the amount of training samples of novel classes, improves the migration ability of novel classes, and verifies the effectiveness of the method on COCO and VOC data sets.

Description

Technical Field

[0001] This invention relates to a method for few-sample object detection, specifically a method for few-sample object detection based on attention and contrastive learning, belonging to the fields of artificial intelligence and image processing. Background Technology

[0002] Object detection involves simultaneously detecting and locating objects in an image, making it more complex than image classification, as it involves both classification and regression tasks. Data samples often exhibit long-tailed distributions, and some samples are difficult to obtain, resulting in significant labeling costs. Therefore, object detection also faces the few-shot problem. This involves training on a large number of base class samples, followed by fine-tuning on a small number of new classes to achieve performance comparable to the base class. Because object detection is more complex than classification, existing techniques for few-shot object detection cannot be simply ported from few-shot classification tasks.

[0003] The development of Convolutional Neural Networks (CNNs) has brought tremendous progress to general object detection. Deep learning object detectors require large amounts of annotated training data to improve their performance. In few-shot learning scenarios, object detectors suffer from severe overfitting, and the gap between few-shot object detection and general object detection is much larger than the corresponding gap in few-shot image classification. Therefore, few-shot object detection models are much more complex than few-shot image classification models.

[0004] Contrastive learning and attention networks are increasingly being used to improve model performance. The success of self-supervised contrastive learning is attributed to renewed research into the subject. It optimizes contrastive instances while maximizing consistency between similar instances (defined as positive pairs) and encouraging differences between dissimilar instances or negative pairs. Through contrastive learning, the network learns to construct representations that are not focused on pixel-level details, but rather encode high-level features sufficient to distinguish different images. However, supervised contrastive learning extends batch contrastive methods to supervised training, but these are primarily used in image classification. By modifying contrastive learning, it can be transferred to few-shot object detection tasks, improving the model's ability to represent instance-level features.

[0005] Mosaic data augmentation, first proposed in YOLOv4 for object detection, is based on CutMix data augmentation. It utilizes four images, enriching the background of the detected object. Furthermore, since it computes data from all four images simultaneously in a batch, the training batch size doesn't need to be large, allowing for good results with a small number of GPUs. To better suit small-sample object detection by comparing new and old classes, certain modifications to mosaic data augmentation are made.

[0006] Attention methods are widely used in object detection, such as DETR based on multi-head self-attention networks, MetaR-CNN based on meta-learning attention, and DAnA based on spatial attention. Attention is not limited to a specific form or structure. The attention involved in this invention is essentially channel multiplication, which stores the instance coordinate offset prediction information through an attention matrix.

[0007] The baseline of this invention is Faster R-CNN, a classic two-stage detection algorithm proposed by Ross B. Girshick in 2016. Its advantages include high accuracy and strong localization capabilities, while its disadvantages include slow detection speed and a large number of parameters. Over the years, a series of improved methods have been developed based on Faster R-CNN, with small-sample object detection being a major focus of these developments.

[0008] Few-shot object detection addresses the challenging problem of few-shot object detection (FSOD) through two main approaches. One is based on meta-learning attention, such as Meta R-CNN, FSRW, and FSIW; the other is based on fine-tuning methods such as TFA and MPSR.

[0009] Meta-learning methods are essentially attention mechanisms based on the attention of support and query sets, but they don't fully utilize the relationships between instances. Attention focuses more on the differences between different instances, while contrastive learning can simultaneously focus on both differences and commonalities. Fine-tuning methods vary in parameter settings depending on the dataset, and are not end-to-end self-learning models. Although empirically feasible, they do not solve the fundamental problem of forgetting during knowledge transfer. Therefore, based on the above problems, this invention is specifically proposed. Summary of the Invention

[0010] The purpose of this invention is to address the shortcomings and defects of the existing technologies by providing a few-sample target detection method based on attention and contrastive learning for the few-sample target detection task.

[0011] The technical solution adopted in this invention is: a few-sample object detection method based on attention and contrastive learning. The few-sample object detection task refers to the object detection model achieving good performance after training on a small number of samples. This method is characterized by: a few-sample mosaic data augmentation module, a coordinate-compensated attention-based region proposal network, a new / old class discrimination module, and a contrastive learning module. The few-sample mosaic data augmentation module enriches the instance background and reduces the data block size; the coordinate-compensated attention-based region proposal network prompts the model's region proposal network to focus on foreground instances; the new / old class discrimination module and the contrastive learning module enhance the expression and contrast of instance features, increasing the difference in feature expression between new and base class instances, and also enhancing knowledge transfer from the base class to the new class. The specific composition of each module is as follows:

[0012] The small-sample mosaic data augmentation module includes sample selection strategies and mosaic data augmentation methods.

[0013] The coordinate-compensated attention region proposal network includes a region proposal network, a regression coordinate-compensated attention branch, and a coordinate-compensated loss.

[0014] The new / old category discrimination module includes a feature extraction layer and a category discrimination layer;

[0015] The contrastive learning module includes a feature mapping layer, contrastive loss, and a joint training strategy;

[0016] The specific steps of the method are as follows:

[0017] (1) The small sample mosaic data augmentation module generates training samples by randomly selecting samples from the base class and new class training samples;

[0018] (2) Faster R-CNN extracts features from the training samples generated in step (1) through the backbone network ResNet 101, and then the feature pyramid network generates backbone output features: P2, P3, P4, P5, P6.

[0019] (3) The coordinate compensation attention region proposal network takes the backbone output features generated in step (2) as input, feeds them into a 3x3 convolution feature extraction network, and then feeds them into a 1x1 convolution network to produce regression coordinate prediction and foreground / background classification prediction respectively. At the same time, it feeds them into the regression coordinate compensation attention branch to generate regression coordinate compensation. The final coordinate prediction is generated by the aforementioned regression coordinate prediction and regression coordinate compensation, and then passes through the anchor box generator and post-processing to obtain the proposed region.

[0020] (4) The ROIPooling layer performs ROIPooling operation on the features P2, P3, P4, P5, and P6 generated in step (2) based on the suggested regions generated in step (3) to generate feature R;

[0021] (5) The feature R generated in step (4) is simultaneously sent to the ROI feature extraction module ROIHead and the new and old class discrimination module Discriminator;

[0022] (6) The new and old class discrimination module feeds the feature R into two fully connected layers to generate binary classification prediction scores for the new class and the base class;

[0023] (7) The ROI feature extraction module has two fully connected layers to extract the feature R into a 1024-dimensional feature F, which is then fed into the bounding box coordinate regression head, classification head, and contrastive learning head. The regression head and classification head results are then processed to generate the final detection result.

[0024] Specifically, the operational steps of the small-sample mosaic data augmentation module include:

[0025] (2.1) According to the proportion, respectively from the base class sample set D base and the new class sample set D novel Samples were drawn from the middle;

[0026] (2.2) Each set of four samples is processed using different data augmentation methods at a ratio of 1:3 between the new class and the base class. After being cropped to a certain random size, they are spliced ​​together to form a single sample. The purpose is to fully mix the rare new class samples with the base class samples, thereby performing data augmentation at the image level and laying the groundwork for subsequent instance-level comparative learning.

[0027] (2.3) Repeat step (2.2) to generate training samples D of batch size. train .

[0028] Specifically, the coordinate-compensated attention region proposal network operates through the following steps:

[0029] (3.1) Input the output features of FPN into a 3x3 convolutional network to generate feature A;

[0030] (3.2) The feature A generated in step (3.1) is passed through two 1x1 convolution layers to generate foreground and background prediction scores for each pixel. With regression coordinate prediction j∈(0,1), where 0 represents the foreground score and 1 represents the background score, and i∈(0,1,2,3) represent the center (x,y) and width and height (w,h) values, respectively;

[0031] (3.3) The feature A generated in step (3.1) is also fed into the regression coordinate compensation attention branch to generate compensation coordinates. Its calculation formula is:

[0032]

[0033] Among them W K Here is the learnable attention matrix, and conv(·) is a 1x1 convolution operation;

[0034] (3.4) Generated from steps (3.2) and (3.3) and Together, we generate the final regression coordinate prediction. Its calculation formula is:

[0035]

[0036] (3.5) The loss function of the coordinate-compensated attention region proposal network is:

[0037]

[0038] Where L RPN For the total loss of the RPN network, and The loss for the RPN network is calculated as follows: classification task loss, regression task loss, and attention branch loss.

[0039]

[0040]

[0041] Where t i The values ​​are the regression coordinate labels, and smooth() is a loss function of type smooth.

[0042] Specifically, the operation steps of the new / old class discrimination module include:

[0043] (4.1) The new and old class discrimination module takes R as input, which is a tensor with dimensions of 7x7xN. It first passes through an expansion layer to generate a 49xN two-dimensional vector R. * ;

[0044] (4.2) The result of step (4.1) R * The prediction α for both the old and new classes is generated through two fully connected layers. It is a 1xN dimensional vector, and its calculation formula is as follows:

[0045] α = FC(P(R))

[0046] In this context, FC(·) represents two fully connected layers, and P(·) represents the flattening operation.

[0047] Specifically, the operational steps of the contrastive learning module include:

[0048] (5.1) The feature extraction result vector F from the ROI is mapped to a 128-dimensional instance vector Z through a two-layer MLP network. i ;

[0049] (5.2) The instance vector Z generated in step (5.1) i To calculate the contrast loss, the formula for the contrast loss is:

[0050]

[0051]

[0052] Where N is Z i The total number, For Z i The instance label is y i The total number, α i It is the value of the i-th row in step (3.2), where τ is the temperature coefficient, which is a hyperparameter. It is the instance vector Z i Regularization, For instance with index j, feature Z j Regularization, where f(u) i The calculation formula is as follows:

[0053] f(u i )=I{u i ≥0.7}·u i

[0054] Among them, u i For the corresponding Z i The corresponding IOU value of the anchor box, where IOU is the intersection-union ratio of the anchor box and the label box;

[0055] (5.3) The total loss is composed of the contrastive loss calculated in step (5.2), the loss of the upregression head and the classification head, and the loss of the coordinate-compensated attention region proposal network.

[0056] L = L c +L reg +L RPN +e 1-β (L CPE +L D )

[0057] Among them, L c and L reg These are the classification loss and regression loss of the Faster R-CNN output head, respectively. RPN L represents the total loss of the RPN network described in step (3.5). D β is the loss function of the new and old class discriminators, and β is the number of training iterations.

[0058] The beneficial effects of this invention are as follows: In the few-shot detection model based on attention and contrastive learning, mosaic data augmentation and contrastive learning are effectively combined to enhance the robustness of feature representation and enrich instance feature representation; this method also combines the attention region proposal network with attention, guiding the model to pay more attention to foreground instances, thus improving training efficiency and quality; the new and old class discrimination network is an organic supplement to contrastive learning, improving the learning rate of similar new and old classes without increasing the model burden. In summary, it reduces the model's dependence on new class training samples while preserving the base class detection accuracy as much as possible. Attached Figure Description

[0059] Figure 1 This is a summary diagram of few-sample object detection methods based on attention and contrastive learning;

[0060] Figure 2 This is a detailed framework diagram of a few-sample object detection method based on attention and contrastive learning;

[0061] Figure 3 This is a schematic diagram of the small sample mosaic data augmentation module;

[0062] Figure 4 This is a schematic diagram of the attention region suggestion network module;

[0063] Figure 5 This is a schematic diagram of the network module for distinguishing between old and new categories;

[0064] Figure 6 This is a schematic diagram of the comparative learning module. Detailed Implementation

[0065] The specific implementation of the present invention will be described below with reference to the accompanying drawings, so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby making a clearer and more explicit definition of the scope of protection of the present invention.

[0066] Example 1: As Figure 1-6 As shown, a few-shot target detection method based on attention and contrastive learning is characterized by including a few-shot mosaic data augmentation module, a coordinate-compensated attention region proposal network, a new and old class discrimination module, and a contrastive learning module.

[0067] The small-sample mosaic data augmentation module includes sample selection strategies and mosaic data augmentation methods.

[0068] The coordinate-compensated attention region proposal network includes a region proposal network, a regression coordinate-compensated attention branch, and a coordinate-compensated loss.

[0069] The new / old category discrimination module includes a feature extraction layer and a category discrimination layer;

[0070] The contrastive learning module includes a feature mapping layer, contrastive loss, and a joint training strategy;

[0071] Based on the general two-stage object detection baseline R-CNN, the above modules are added. Among them, the few-sample mosaic data augmentation module enriches the instance background, especially for a small number of novel class instances, while effectively comparing the novel class with the base class, improving the efficiency of knowledge transfer from the base class to the novel class; the coordinate-compensated attention region proposal network intuitively improves the coordinate accuracy of the RPN network's proposal boxes, essentially shifting the RPN network's attention from the background to the foreground instances, increasing the model's focus on novel class instances; the new and old class discrimination module identifies difficult-to-distinguish pairs of new and old class instances, indicating that more useful information is contained within them, so the model's learning of instance pairs is improved by increasing the weights of the corresponding loss function; the contrastive learning module improves the model's representation of instance-level features, increasing the similarity of features among instances of the same class and reducing the similarity of features among instances of different classes.

[0072] The above modules enhance the model's instance representation capabilities, reduce the number of samples required for new classes, and improve the detector's knowledge transfer capabilities from base classes to old classes. Furthermore, while improving the detection accuracy for new classes, the detection accuracy for base classes is preserved as much as possible.

[0073] Detailed model structure as follows Figure 2 As shown, the training process is divided into two stages: In the first stage, the model is trained only on the base classes, using a standard mosaic data augmentation module and a coordinate-compensated attention region proposal network, without using the new / old class discrimination module or the contrastive learning module; In the second stage, the model is trained simultaneously on both the base and new classes, using all the new class data and a small amount of base class data randomly sampled using the few-sample mosaic data augmentation module. For result validation, the detection accuracy of both the base and new classes is simultaneously tested, and validation is performed on both the COCO 2014 and PASCALVOC datasets, using the nAP50 detection metric.

[0074] The specific steps of the method are as follows:

[0075] (1) The small sample mosaic data augmentation module randomly selects samples from the training samples of the base class and the new class according to a certain proportion and number. Every 4 samples generate a training sample according to the mosaic data augmentation method. Every 4 samples contain a certain proportion of new class samples. The ratio of new class to base class samples is 1:3. The new class samples adopt richer data augmentation methods. The sample size is uniformly 640x640x3.

[0076] (2) Faster R-CNN uses the training samples generated in step (1) to extract features through the backbone network. The backbone network uses ResNet 101, where the five layers are Conv1(BN+ReLU), Conv2_x, Conv3_x, Conv4_x, and Conv5_x. BN represents normalization, ReLU represents the activation function, and Conv_x represents the i-th convolutional layer. The 2nd, 3rd, 4th, and 5th layers generate Res2, Res3, Res4, and Res5 features, respectively. These features are fed into the FPN (Feature Pyramid Network) to generate P2, P3, P4, P5, and P6 features. The FPN network consists of four Conv2d 1x1 layers, five Conv2d 3x3 layers, and one Maxpool 1x1 layer. Conv2d 1x1 represents a 1x1 convolutional layer, Conv2d 3x3 represents a 3x3 convolutional layer, and Maxpool 1x1 represents the maxpool operation. Upsample represents upsampling, such as... Figure 2 As shown;

[0077] (3) The coordinate compensation attention region proposal network (AttentionRPN) takes the P2, P3, P4, P5, and P6 features generated in step (2) as input and feeds them into the convolutional feature extraction network (RPN Head). Then, the 1x1 convolutional network Conv 1x1 produces regression coordinate prediction (RPN logistic) and foreground / background classification prediction respectively. At the same time, it is also fed into the regression coordinate compensation attention branch (Attention) to generate regression coordinate compensation. The final coordinate prediction is generated by the aforementioned regression coordinate prediction and regression coordinate compensation. Then, the proposed region is obtained through the anchor generator (AnchorGenerator) and post-processing (Proposals, Filter Proposals). The RPN loss is to randomly select a certain number of positive and negative samples in the feature map based on the anchor boxes and prediction scores. The selection operation is Filter Proposals. The positive and negative samples are divided based on the label (Ground truth).

[0078] (4) The ROIPooling layer performs ROIPooling operation on the P2, P3, P4, P5, and P6 features generated in step (2) based on the suggested regions generated in step (3) to generate instance features R, the size of which is normalized to 7x7xN.

[0079] (5) The feature R generated in step (4) is simultaneously sent to the ROI header and the new / old class discrimination module (Discriminator):

[0080] (6) The new and old class discrimination module feeds the feature R into the flatten layer, two fully connected layers (Fc1, Fc2) and the ReLU layer to generate binary classification prediction scores for the new class and the base class;

[0081] (7) The ROI feature extraction module has two fully connected layers to extract features R into 1024-dimensional features F, which are then fed into the contrastive learning output head. This head includes the Faster R-CNN output head and the contrastive learning module. Figure 2 As shown, the bounding box coordinate regression head and classification head consist of two fully connected layers (FC). Contrastive learning is as follows: Figure 6 As shown in the figure. The losses for Regression and Classification are cross-entropy loss (Softmax cross entropy loss) and L1 norm loss (Smooth L1 loss), respectively. The Faster R-CNNHead results are used to generate the final detection results through post-processing of the prediction results.

[0082] Furthermore, the specific steps for operating the small-sample mosaic data augmentation module include:

[0083] (2.1) Proportionally from base class D base And new class D noval Samples are drawn from the sample;

[0084] (2.2) Each set of 4 samples is processed using different data augmentation methods at a ratio of 1:3 between the new class and the base class. The samples are then cropped to a random size and spliced ​​together to form a single sample. The new class samples will be reused using different data augmentation methods.

[0085] (3.3) Repeat step (1.2) to generate batch training samples D of batch size. train Where the base class samples and the new class samples are of different categories, D train =D base UD novel D base ∩D novel =φ, such as Figure 3 As shown.

[0086] Furthermore, the specific operational steps of the coordinate-compensated attention region proposal network include:

[0087] (3.1) Input the output features of FPN into a 3x3 convolutional network to generate feature A, with a feature size of NxNx256. The size of N depends on the input size, such as... Figure 4 As shown;

[0088] (3.2) The feature A generated in step (2.1) is passed through two 1x1 convolution layers to generate foreground and background prediction scores for each pixel. With regression coordinate prediction j∈(0,1), 0 represents the foreground score, 1 represents the background score, and i∈(0,1,2,3) represent the center (x,y) and width / height (w,h) values ​​respectively. The calculation formula is:

[0089]

[0090]

[0091] Where conv(·) is a 1x1 convolution operation;

[0092] (3.3) The feature A generated in step (3.1) is also fed into the regression coordinate compensation attention branch to generate compensation coordinates. For error prediction, the calculation formula is as follows:

[0093]

[0094] Among them W K This is a learnable attention matrix, the size of which varies with the size of A, and is usually 5-dimensional, corresponding to different input feature layers. conv(·) is a 1x1 convolution operation.

[0095] (3.4) Generated from steps (3.2) and (3.3) and Together, we generate the final regression coordinate prediction. Its calculation formula is:

[0096]

[0097] (3.5) The loss function of the coordinate-compensated attention region proposal network is:

[0098]

[0099] Where L RPN For the total loss of the RPN network, and The loss for the RPN network is calculated as follows: classification task loss, regression task loss, and attention branch loss.

[0100]

[0101]

[0102] Where t i The values ​​are the regression coordinate labels, and smooth(·) is a loss function of type smooth. Gradient backpropagation is performed across the entire RPN network, except for the attention branch. The gradient is only propagated back to the attention branch.

[0103] Furthermore, the specific operational steps of the new / old class discrimination module include:

[0104] (4.1) The new and old class discrimination module takes R as input, which is a tensor with dimensions of 7x7xN. N is usually 512 and is a hyperparameter. It first passes through an expansion layer to generate a 49xN two-dimensional vector R. * ;

[0105] (4.2) The result of step (4.1) R * The prediction α for both the old and new classes is generated through two fully connected layers. This α is a 1xN dimensional vector, where a label value of 0 represents the base class, 1 represents the new class, and N is the total number of instances. The calculation formula is as follows:

[0106] α = FC(P(R))

[0107] Wherein, FC(·) represents two fully connected layers (FC1, FC2), and P(·) represents the flattening operation.

[0108] (4.3) The loss function of the new and old class discrimination module is:

[0109] L D =smooth(y-α)

[0110] Where y represents the old and new class labels, 0 indicates the base class, and 1 indicates the new class. The gradient is only returned to the old and new class discrimination module, such as... Figure 5 As shown.

[0111] Furthermore, the specific operational steps of the contrastive learning module include:

[0112] (5.1) The feature extraction result vector F from the ROI is mapped to a 128-dimensional instance vector Proposal(p) through a two-layer MLP network. i y i ), where i is the index of the i-th instance feature, and the MLP is composed of fully connected layers (FC), ReLU activation layers, and fully connected layers (FC) respectively. Figure 2 As shown. The comparative learning module is as follows. Figure 6 As shown, Proposal(p i y i p) i Yes, the i-th instance feature, yi This refers to the corresponding label value; Box Regressor and Box Classifier are... Figure 1 The Faster R-CNN output header includes the bounding box coordinates of the regression header and the classification header.

[0113] (5.2) From Proposal(p i y i The instance vector Z after comparison with the contrastive head. i The contrast loss is calculated for ∈Z, and the formula for the contrast loss is as follows:

[0114]

[0115]

[0116] Where N is Z i The total number, For Z i The instance label is y i The total number, α i This is the predicted value of the old and new classes for the i-th instance in step (3.2), where τ is the temperature coefficient, a hyperparameter, usually taken as 0.2. It is the instance vector Z i Regularization, For instance with index j, feature Z j Regularization. Where f(u) i The calculation formula is as follows:

[0117] f(u i )=I{u i ≥0.7}·u i

[0118] Among them, u i For the corresponding Z i The corresponding IOU value of the anchor box, where IOU is the intersection-union ratio of the anchor box and the label box;

[0119] (5.3) The total loss is composed of the contrastive loss calculated in step (5.2), the loss of the upregression head and the classification head, and the loss of the coordinate-compensated attention region proposal network.

[0120] L = L c +L reg +L RPN +e 1-β (L CPE +L D )

[0121] Among them, L c and Lreg These are the classification loss and regression loss of the Faster R-CNN output head, respectively. RPN β represents the total loss of the RPN network described in step (3.5), and β represents the number of training rounds, indicating that the contrastive loss gradually decreases with the proportion of training rounds.

[0122] Example 2: This example illustrates the invention in detail with specific data.

[0123] Model training is divided into two stages: the first stage is base class training; the second stage is fine-tuning with a small number of new classes. The base class training samples are sufficiently abundant, while the new class training data is divided into task-specific sets of 1, 3, 5, and 10 samples per class. The model is trained and validated on the COCO and VOC datasets respectively. The COCO dataset has 80 classes, with 60 base classes and 20 new classes; the VOC dataset has 20 classes, with 15 base classes and 5 new classes.

[0124] Hyperparameter settings: Gradient descent uses standard SGD, momentum is 0.9, and weight decay is e. -4 The batch size is 16. The software environment is Ubuntu 18.04, CUDA 10.2, PyTorch 1.4.0, and Python 3.6; the hardware environment consists of 8 RTX 2080ti graphics cards and an AMD Ryzen 7 R7 3500x CPU.

[0125] The model structure differs between the first and second training phases. The first phase involves standard mosaic data augmentation, with the data fed into the backbone network ResNet 101, such as... Figure 2 As shown, the generated res2, res3, etc. features are then fed into the FPN structure. Both the backbone network and the FPN network are pre-trained on the ImageNet classification dataset. Next, an attention region proposal network is introduced. The attention branch of this network is already included during base class training, and its loss function gradient is only propagated back to itself, not to other structures. The result of the attention region proposal network is fed into the ROI head to generate instance-level ROI features, where the new and old class discriminators are not used. The ROI features are then fed into the contrast learning head, where the gradients of all three loss functions are propagated back to the entire network.

[0126] The second stage of data processing involves small-sample mosaic data augmentation. Specifically, sample sets of 3 times the batch size and 1 times the batch size are randomly selected from the base class and new class data at a 3:1 ratio. Each time, 3 images are randomly selected from the base class and 1 image is selected sequentially from the new class. These 4 images are then processed using standard data augmentation methods. Finally, these 4 images are cropped and stitched together to a new image at random sizes. This process is repeated batch-size times to generate a training set of batch size, which is then fed into the model. The backbone network, FPN network, and attention region proposal network remain unchanged from the first stage. A new / old class discrimination module is connected before the ROI head. The parameters of the fully connected layer 1 of this module are initialized the same as the first software layer of the ROI head. The module's function is to distinguish the similarity between the new class and the base class. The contrastive learning head follows, consistent with the first stage, but the parameters of the backbone network and FPN network are frozen and do not participate in training.

[0127] During the testing phase, data augmentation, the new and old class discriminators, and the contrastive learning module will be deprecated; at the same time, all loss mechanisms will no longer be used.

[0128] Results: The results in PASCAL VOC are shown in Table 1. These results are the detection data after the second stage of training, with the parameter settings being the same as described above. The base class detection result after the first stage of base class training has an AP50 of 80.8.

[0129] Table 1: Results of the invention on PASCAL VOC, where AP50 represents mAP>0.5.

[0130]

[0131] The results for the COCO dataset are shown in Table 2:

[0132] Table 2: Results of this invention on COCO, where AP75 represents mAP > 0.75

[0133]

[0134] Compared to other models, this model enhances instance representation capabilities, reduces the number of samples required for new classes, and improves the detector's knowledge transfer ability from base classes to older classes. Furthermore, it improves detection accuracy for new classes while preserving detection accuracy for base classes as much as possible. Experimental results demonstrate that this invention possesses excellent performance and broad applicability.

[0135] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A few-sample target detection method based on attention and contrastive learning, characterized in that: This system utilizes a few-sample mosaic data augmentation module, a coordinate-compensated attention region proposal network, a new / old class discrimination module, and a contrastive learning module. The specific structure of each module is as follows: The small-sample mosaic data augmentation module includes sample selection strategies and mosaic data augmentation methods. The coordinate-compensated attention region proposal network includes a region proposal network, a regression coordinate-compensated attention branch, and a coordinate-compensated loss. The new / old category discrimination module includes a feature extraction layer and a category discrimination layer; The contrastive learning module includes a feature mapping layer, contrastive loss, and a joint training strategy; The specific steps of the method are as follows: (1) The small-sample mosaic data augmentation module generates training samples by randomly selecting samples from the base class and new class training samples; (2) Faster R-CNN extracts features from the training samples generated in step (1) through the backbone network ResNet 101, and then the feature pyramid network generates backbone output features: P2, P3, P4, P5, P6. (3) The coordinate compensation attention region proposal network takes the backbone output features generated in step (2) as input, feeds them into a 3x3 convolution feature extraction network, and then feeds them into a 1x1 convolution network to produce regression coordinate prediction and foreground / background classification prediction respectively. At the same time, it feeds them into the regression coordinate compensation attention branch to generate regression coordinate compensation. The final coordinate prediction is generated by the aforementioned regression coordinate prediction and regression coordinate compensation, and then passes through the anchor box generator and post-processing to obtain the proposed region. (4) The ROIPooling layer performs ROIPooling operations on the P2, P3, P4, P5, and P6 features generated in step (2) based on the suggested regions generated in step (3) to generate features. R ; (5) Features generated in step (4) R Simultaneously, the data is fed into the ROI feature extraction module ROIHead and the new / old class discrimination module Discriminator; (6) The new and old class discrimination module will use features R After being fed into two fully connected layers, binary classification prediction scores for the new class and the base class are generated; (7) The ROI feature extraction module has two fully connected layers that extract features. R Extracted as 1024-dimensional features F Simultaneously, the bounding box coordinate regression head, classification head, and contrastive learning head are fed in, with the regression head and classification head results being post-processed to generate the final detection result.

2. The few-shot target detection method based on attention and contrastive learning according to claim 1, characterized in that: The specific steps for operating the small-sample mosaic data augmentation module include: (2.1) Samples from the base class set proportionally. and new class sample sets Samples were drawn from the middle; (2.2) Each set of four samples is processed using different data augmentation methods at a ratio of 1:3 between the new class and the base class. After being cropped to a certain random size, they are spliced ​​together to form a single sample. The purpose is to allow the rare new class samples to be fully mixed with the base class samples, so as to perform data augmentation at the image level and lay the groundwork for subsequent instance-level comparative learning. (2.3) Repeat step (2.2) to generate training samples of batch size. .

3. The few-sample target detection method based on attention and contrastive learning according to claim 1, characterized in that: The specific operational steps of the coordinate-compensated attention region proposal network include: (3.1) Input the output features of FPN into a 3x3 convolutional network to generate features. A ; (3.2) The features generated in step (3.1) A Foreground and background prediction scores are generated for each pixel using two 1x1 convolutional layers. With regression coordinate prediction , 0 represents the foreground score, and 1 represents the background score. Representing the center and width and height The value; (3.3) Features generated in step (3.1) A It also feeds into the regression coordinate compensation attention branch to generate compensation coordinates. Its calculation formula is: ； in A learnable attention matrix, This is a 1x1 convolution operation; (3.4) Generated from steps (3.2) and (3.3) and Together, we generate the final regression coordinate prediction. Its calculation formula is: ； (3.5) The loss function of the coordinate-compensated attention region proposal network is: ； in For the total loss of the RPN network, , and The loss for the RPN network is calculated as follows: classification task loss, regression task loss, and attention branch loss. ； ； in The values ​​are the regression coordinate labels, and smooth() is a loss function of type smooth.

4. The few-sample target detection method based on attention and contrastive learning according to claim 1, characterized in that: The specific steps of the new / old class discrimination module include: (4.1) The new and old class discrimination module is based on R For input, R For dimension 7 x 7 x N The tensor is first generated through an expansion layer. 49 x N Two-dimensional vector ; (4.2) Results of step (4.1) New and old class predictions are generated through two fully connected layers. ,for 1 x N A 3D vector has the following operation formula: ； in, It consists of two fully connected layers. For flattening operation.

5. The few-sample target detection method based on attention and contrastive learning according to claim 3, characterized in that: The specific steps of the comparative learning module include: (5.1) Feature extraction result vector from ROI F After being mapped by a two-layer MLP network, it becomes a 128-dimensional instance vector. ; (5.2) Instance vectors generated from step (5.1) To calculate the contrast loss, the formula for the contrast loss is: ； ； in, N for The total number, for Instance tags are The total number, It is the value of the i-th row in step (3.2). The temperature coefficient is a hyperparameter. It is an instance vector Regularization, Features of the instance with index j Regularization, where The calculation formula is as follows: ； in, For the corresponding The corresponding IOU value of the anchor box, where IOU is the intersection-union ratio of the anchor box and the label box; (5.3) The total loss is composed of the contrastive loss calculated in step (5.2), the loss of the upregression head and the classification head, and the loss of the coordinate-compensated attention region proposal network. ； in, and These represent the classification loss and regression loss of the Faster R-CNN output head, respectively. The total loss of the RPN network described in step (3.5) is... It is the loss function of the old and new class discriminators. This refers to the number of training sessions.

Images