Small sample target detection method based on fine-grained feature purification and spatial alignment

Abstract

The invention relates to the technical field of image processing, in particular to a small sample target detection method based on fine-grained feature purification and spatial alignment, and the method comprises the steps: respectively obtaining a query image and a plurality of support images; inputting the query image and the support image into a twin ResNet101 network sharing a trunk, and outputting a candidate box feature map and a category feature map; and inputting the candidate frame feature map and the category feature map into a feature alignment module and a prototype fusion module in sequence to obtain prototype fusion features, inputting the prototype fusion features into a detection head, and outputting the category and position of the query image. The method solves the problems that in an existing method, double-branch feature interaction is insufficient, and the effect is poor under a minimum sample.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a small sample target detection method based on fine-grained feature purification and spatial alignment. Background Technology

[0002] Deep learning has driven significant progress in the field of object detection, but the strong dependence of traditional methods on large-scale labeled data leads to significant shortcomings in small sample scenarios. Few sample object detection (FSOD) aims to achieve new class detection with limited samples through knowledge transfer from base class to new class. Currently, it is mainly divided into methods based on transfer learning and meta-learning.

[0003] Existing meta-learning-based methods typically handle query and support branches independently, performing feature fusion only through simple operations such as element-wise multiplication, lacking refined information interaction; the extraction of key class prototype information is coarse, lacking fine-grained semantic features; the spatial misalignment between query and support features is significant, leading to inaccurate feature matching; they only capture inter-class similarity, ignoring dissimilar features, resulting in insufficient discriminative ability; the high-level feature fusion method is fixed and cannot adapt to the feature relationships of different samples; the shortcomings of existing technologies lead to significant performance degradation in extremely small sample scenarios, making it difficult to cope with target detection tasks in complex scenarios; therefore, solving the problems of insufficient feature interaction between two branches and poor performance in extremely small sample scenarios in existing methods is an urgent issue to be addressed. Summary of the Invention

[0004] To address the shortcomings of existing methods, this invention combines a fine-grained feature coupling module with spatial feature alignment and foreground attention mechanisms to achieve refined spatial alignment between class prototypes and query features, enhancing feature discriminability and thus improving the performance of target detection in very small samples. Taking into account the differences and similarities between the ROI features of the query image and the class prototype, a dynamic weighting mechanism is introduced, and a prototype fusion module with adaptive fusion weights is designed to enhance the complementarity of high-level representations and alleviate the feature fusion and interaction problems between the two-branch networks.

[0005] The technical solution adopted in this invention is: a small sample target detection method based on fine-grained feature purification and spatial alignment, comprising the following steps: Step 1: Obtain the query image and several supporting images respectively; Step 2: Input the query image and supporting images into a twin ResNet101 network with a shared backbone, and output candidate box feature maps and category feature maps; In a preferred embodiment of the present invention, step two specifically includes: The query image is input into the first branch of the ResNet101 network in the first to third layers, and the first feature map is output. The support image is input into the second branch of the ResNet101 network in the first to third layers, and the second feature map is output. The first and second feature maps are input into FFA and RPN and fused to obtain the third feature map. The third feature map is input into the first branch of the ResNet101 network in the fourth layer, and the fifth feature map is output. The second feature map is input into the second branch of the ResNet101 network in the fourth layer, and the sixth feature map is output.

[0006] In a preferred embodiment of the present invention, the first to fourth layers of the ResNet101 network are composed of 3, 4, 23, and 3 Bottleneck residual blocks, respectively.

[0007] In a preferred embodiment of the present invention, the Bottleneck residual block includes: 1x1 convolution, 3x3 convolution for feature extraction, and 1x1 convolution, and the output is fused with the input through an addition operation.

[0008] Step 3: Input the candidate box feature map and the category feature map into the feature alignment module and the prototype fusion module in sequence to obtain the prototype fusion feature. Input the fifth feature map into the detection head and output the category and location of the query image. In a preferred embodiment of the present invention, the feature alignment module includes: Calculate the similarity matrix , , These are the candidate bounding box feature map and the category feature map, respectively. Will Softmax normalization is performed along the feature dimension to obtain the feature matrix. A ’ ; Calculate the category prototype for candidate region alignment ; Computing foreground attention mask ; The candidate bounding box feature map and category feature map output by the feature alignment module are calculated using the following formula: ; and These are learnable parameters.

[0009] In a preferred embodiment of the present invention, a prototype fusion module based on nonlinear fusion and adaptive weights is constructed to perform element-wise multiplication, subtraction, and concatenation of feature maps, thereby learning inter-class relationships; the formula of the prototype fusion module is:

[0010] in, , , and This indicates an independent fully connected layer; It is the dot product.

[0011] In a preferred embodiment of the present invention, the nAP50 metric is used to evaluate the model based on the Siamese ResNet101 network, the feature alignment module, the prototype fusion module, and the detection head.

[0012] In a preferred embodiment of the present invention, the detection head employs a fusion of ROI classification loss, regression loss, and meta-classification loss.

[0013] As a preferred embodiment of the present invention, a small sample target detection system based on fine-grained feature purification and spatial alignment includes: a memory for storing instructions executable by a processor; and a processor for executing the instructions to implement a small sample target detection method based on fine-grained feature purification and spatial alignment.

[0014] As a preferred embodiment of the present invention, a computer-readable medium storing computer program code implements a small sample target detection method based on fine-grained feature purification and spatial alignment when executed by a processor.

[0015] The beneficial effects of this invention are: 1. Fine-grained feature coupling module: It realizes fine-grained interaction between class prototypes and query features through spatial alignment and foreground attention mechanism, thereby improving feature discriminativeness; 2. Prototype Fusion Module: Based on a dynamic fusion mechanism with adaptive weights, it simultaneously captures the similarity and differences of features, enhancing the complementarity of high-level representations; 3. Dual-branch feature interaction framework: A multi-level information interaction mechanism from shallow feature distillation to high-level dynamic fusion, optimizing knowledge transfer efficiency in small sample scenarios; 4. Learnable parameter optimization: The adaptive weights in the fine-grained feature coupling module and the dynamic weight generation network in the prototype fusion module improve the model's adaptability to different samples. Attached Figure Description

[0016] Figure 1 This is a flowchart of the small sample target detection method based on fine-grained feature purification and spatial alignment of the present invention; Figure 2 This is a schematic diagram of the feature alignment module structure of the present invention; Figure 3 This is a schematic diagram of the prototype fusion module structure of the present invention. Detailed Implementation

[0017] The present invention will be further described below with reference to the accompanying drawings and embodiments. The drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0018] like Figure 1 As shown, a small-sample target detection method based on fine-grained feature extraction and spatial alignment includes the following steps: Step 1: Obtain the query image and several supporting images respectively; Step 2: Construct a twin ResNet101 network with a shared backbone. Input the query image into the first branch of the ResNet101 network in the first to third layers, and output the first feature map; input the support image into the second branch of the ResNet101 network in the first to third layers, and output the second feature map; input the first and second feature maps into FFA and RPN and fuse them to obtain the third feature map; input the third feature map into the first branch of the ResNet101 network in the fourth layer, and output the fifth feature map; input the second feature map into the second branch of the ResNet101 network in the fourth layer, and output the sixth feature map. The ResNet101 network has four layers. Layer 1 consists of 3 Bottleneck residual blocks; Layer 2 consists of 4 Bottleneck residual blocks; Layer 3 consists of 23 Bottleneck residual blocks; and Layer 4 consists of 3 Bottleneck residual blocks. Each Bottleneck residual block contains three convolutional layers: a 1x1 convolution for dimensionality reduction; a 3x3 convolution for feature extraction; and a 1x1 convolution for dimensionality recovery. The three convolutional layers are surrounded by Batch Normalization and ReLU activation functions. The output of the residual block is fused with the input through an addition operation to form the final block output.

[0019] The first branch ResNet101 network and the second branch ResNet101 network together constitute a twin ResNet101 network with a shared backbone; The Fine-Grained Feature Aggregation (FFA) module distills fine-grained prototypes from supporting images and assigns them to query features, enabling shallow information interaction between the two branches.

[0020] The first three layers of the ResNet101 network (used to extract low-level and intermediate-level features, such as edges, textures, basic shapes, and object parts) extract mid-level features from the query image and support images. For example, extracting features from a query image where the instance category is cow and features from multiple support images where the instance category is sheep. FFA distills the support feature prototypes into the query branch, while the fourth layer of the backbone network is used to extract high-level features and retain high-resolution features, generating Region of Interest (RoI) features and category prototypes (i.e., the fifth and sixth feature maps) respectively. Step 3: Input the fifth and sixth feature maps into the feature alignment module and the prototype fusion module in sequence to obtain the prototype fusion feature. Input the prototype fusion feature into the detection head and output the category and location of the query image. like Figure 2 As shown, the semantic similarity between candidate regions and categories is crucial to detection accuracy; the similarity matrix on the bi-branch features is calculated through matrix multiplication. And use foreground attention to locate foreground targets; The feature alignment module includes: Given a pair of input features (Fifth Feature Map) and (Sixth Feature Map) , All include Each convolutional grid feature has an embedding dimension of 1. ; Calculate the similarity matrix Each element is the dot product of two embedded features, as shown in the following formula: (1) in, Representing candidate box features In spatial location Above and category prototypes In spatial location The similarity of embedded features is used to capture dense feature relationships; T This is a transpose.

[0021] Category prototypes through Guided grid feature weighted averaging is used to generate features, where higher similarity scores amplify the contribution of the corresponding features; this process ensures accurate alignment and extracts discriminative features crucial for small-sample object detection; then, Perform softmax normalization along the feature dimension to obtain the matrix. Category Prototype With normalized similarity matrix Multiplying them yields a category prototype aligned with the candidate regions. The formula is as follows: (2) Foreground Attention Module: For new classes, the region proposal network generates candidate regions of lower quality than the base class, thus including unwanted background regions and misclassified regions. Foreground attention masks are then used to address this. Highlight the relevant target areas and supplement the deficiencies in this part; more specifically, for query features... Each spatial location in Using a relation matrix To calculate With class prototype features The sum of similarities at each location is then normalized to class probabilities using the sigmoid function, as shown in the following formula: (3) The mask This represents the semantic information corresponding to each spatial location, where the higher value... Representing spatial location and class prototype The higher similarity between them implies similar attribute features; conversely, the background region, due to its significant difference from the prototype features, corresponds to a mask value Low; utilize this feature to mask attention. Aligned class prototype Multiplication selectively enhances foreground-related activations while suppressing background noise through spatially adaptive feature reweighting, as shown in the following formula: (4) Note that this invention fully considers the contribution of different features to the model and introduces learnable parameters. and All are initialized to 0, with values ​​ranging from 0 to 1; these parameters are respectively and Multiply the features and then add the result to the original features to achieve stable training.

[0022] like Figure 3 For the prototype fusion module, in meta-learning methods, the alignment of high-level semantic information before final prediction is crucial to model performance. Existing methods fuse RoI features and class prototypes through element-wise multiplication, which can capture inter-class similarity but ignores dissimilar features, potentially harming classification accuracy. This invention introduces a non-linear fusion network and improves it into a prototype fusion module based on adaptive weights. Specifically, the prototype fusion module independently processes element-wise multiplication, subtraction, and concatenation of features to learn inter-class relationships. Dynamic parameters weight these operations, thereby enhancing feature complementarity and improving detection accuracy, as shown in the following formula: (5) in, , , and This represents an independent fully connected layer, and each fully connected layer is followed by a ReLU activation function. This indicates a fully connected layer. , and This represents the learnable parameters, initialized to 1, with a value range of 0 to 1. The prototype fusion module can effectively help the model consider both intra-class similarity and inter-class differences, and dynamically adjust parameters based on contributions to improve model performance. It not only provides a stronger ability to thoroughly explore the relationships between high-level features, but also includes the original RoI features to capture information from the original region of interest, reducing noise from random prototypes and can be used to regress target locations.

[0023] The detection head employs a fusion of ROI classification loss, regression loss, and meta-classification loss. The ROI classification loss is used to classify candidate regions into multiple categories. The ROI regression loss is used to fine-tune the position and size of the detection boxes. The meta-classification loss is used to train the model to quickly use new categories, simulating a few-shot learning scenario where the model encounters a base class during training and a new class during testing, thus enabling the model to possess the meta-learning ability of "how to learn." The formula is as follows: (6) in, This represents the multi-class cross-entropy loss, which includes all categories and the background class. This indicates that multi-class cross-entropy loss is used during the fine-tuning phase to calculate the probability of the small sample class. This indicates the regression loss using smooth L1.

[0024] Algorithm accuracy comparison: The detection results on the PASCAL VOC dataset are shown in Table 1. The evaluation metric used is the new class detection accuracy (nAP50). The methods compared can be divided into two categories: transfer learning-based methods and meta-learning-based methods. Experimental results show that the method of this invention achieves optimal or competitive performance under various settings. First, in terms of overall performance, the average nAP50 of the method of this invention reaches 54.6%, which is significantly better than other comparative methods. Second, with a sample setting of K={2,3,5}, the detection performance on new class set 2 is the highest; while in new class set 3, the method of this invention achieves the best results with a sample setting of K={1,2,3,5}. These results show that the method can still maintain high detection accuracy in scenarios with extremely low sample sizes, demonstrating its applicability in practical application scenarios with scarce samples. In addition, in the 10-sample setting, the method of this invention remains highly competitive and achieves the highest performance of 68.0% on new class set 1. Although the performance in the 10-sample settings of new class sets 2 and 3 is slightly lower than Meta-DETR and FPD. While the proposed method still ranks second or third, it demonstrates stable performance across different sample sizes. Notably, the method maintains stable performance across all three new class settings without significant fluctuations, showcasing strong robustness to different class combinations. Compared to the baseline method FPD... In comparison, the method of this invention improves the average performance by 1.0% and achieves significant improvements in multiple low-sample settings, verifying the enhancement effect of the designed FC and PF modules on feature representation capability and inter-class discrimination.

[0025] Table 1 shows the FSOD results (nAP50 metric) across the three partitions of the Pascal VOC dataset.

[0026] illustrate: The numbers indicate the reproduced results; Ours is the baseline model that combines fine-grained feature coupling (FC) and prototype fusion (PF); bold and underline represent the best and second-best results, respectively.

[0027] Table 2 shows the detection results on the MS COCO dataset. Compared to PASCAL VOC, the COCO dataset contains more categories, larger-scale images, and more complex scenes. These factors lead to the performance of small-sample object detection models on COCO generally being lower than that on PASCAL VOC. Under the most challenging 1-sample setting, the model proposed in this study achieves the best performance on the AP, AP50, and AP75 metrics, reaching 9.5%, 22.5%, and 7.1%, respectively. Compared with the baseline method FPD... In comparison, these results represent relative improvements of 33.8%, 48.0%, and 24.6%, respectively, indicating that the method maintains strong feature learning capabilities even in scenarios with extremely scarce data. In the 2-sample setting, the model continues to maintain leading performance, ranking first with an AP value of 11.5% and an AP50 value of 26.7%. In the 3-sample setting, the model of this invention remains competitive with an AP50 value of 27.4%. Therefore, the method of this invention with the integrated PF module achieves performance improvements in all K={1,2,3} sample settings. It is worth noting that the model with the FC module performs poorly on the COCO dataset, so the model architecture with the integrated PF module was ultimately chosen.

[0028] Table 2 shows the FSOD results (nAP metric) on the MS COCO dataset.

[0029] illustrate: The result indicates the reproduced result; Ours (PF) is the baseline model using only prototype fusion; bold and underline indicate the best and second best results, respectively.

[0030] To verify the necessity of the improved module in small sample target detection, an ablation experiment was conducted on the PASCAL VOC Split3 data partition with a sample setting of K={1,2,3}. As shown in Table 3, starting from the baseline model, the feature coupling module FC and the prototype fusion module PF were gradually integrated. The experimental results show that introducing the FC module alone improves the performance of the K={1,2,3} sample setting by approximately 2.7%, 2.4%, and 1.0% respectively compared to the baseline model, proving that the module can effectively improve the feature alignment between the support set and the query set. Using the PF module alone improves the 3-sample detection performance by approximately 1.9% while maintaining the competitiveness of the K={1,2} sample setting. When the FC and PF modules are integrated simultaneously, the model achieves performance gains of approximately 3.2%, 2.8%, and 1.5% respectively under the three sample settings, fully verifying the necessity and synergistic effect of the two modules.

[0031] Table 3 Ablation Experiments with Different Module Combinations

[0032] Note: FC uses only the baseline model with fine-grained feature coupling; PF uses only the baseline model with prototype fusion.

[0033] Based on the above-described preferred embodiments of the present invention, and through the foregoing description, those skilled in the art can make various changes and modifications without departing from the inventive concept. The technical scope of this invention is not limited to the contents of the specification, but must be determined according to the scope of the claims.

Claims

1. A small-sample target detection method based on fine-grained feature extraction and spatial alignment, characterized in that, Includes the following steps: Step 1: Obtain the query image and several supporting images respectively; Step 2: Input the query image and supporting images into a twin ResNet101 network with a shared backbone, and output candidate box feature maps and category feature maps; Step 3: Input the candidate box feature map and the category feature map into the feature alignment module and the prototype fusion module in sequence to obtain the prototype fusion feature. Input the prototype fusion feature into the detection head and output the category and location of the query image.

2. The small-sample target detection method based on fine-grained feature extraction and spatial alignment according to claim 1, characterized in that, Step two specifically includes: The query image and the support image are respectively input into the two-branch ResNet101 network of the first to third layers, and the first feature map and the second feature map are output. The first feature map and the second feature map are input into FFA and RPN and fused to obtain the third feature map. The third feature map and the second feature map are respectively output into the two-branch ResNet101 network of the fourth layer to produce the fifth feature map and the sixth feature map.

3. The small-sample target detection method based on fine-grained feature extraction and spatial alignment according to claim 1, characterized in that, The feature alignment module includes: Calculate the similarity matrix of embedded features , , These are the candidate bounding box feature map and the category feature map, respectively. Will Softmax normalization is performed along the feature dimension to obtain the feature matrix. A ’ ; Calculate the category prototype for candidate region alignment ; Computing foreground attention mask ; The feature map of the candidate bounding box output by the feature alignment module. and category feature map , and These are learnable parameters.

4. The small-sample target detection method based on fine-grained feature extraction and spatial alignment according to claim 3, characterized in that, The formula for the prototype fusion module is: in, , , and This indicates an independent fully connected layer; It is the dot product.

5. The small-sample target detection method based on fine-grained feature extraction and spatial alignment according to claim 2, characterized in that, The first to fourth layers of the ResNet101 network consist of 3, 4, 23, and 3 Bottleneck residual blocks, respectively.

6. The small-sample target detection method based on fine-grained feature extraction and spatial alignment according to claim 5, characterized in that, Bottleneck residual blocks include: 1x1 convolutions, 3x3 convolutions for feature extraction, and 1x1 convolutions that fuse the output and input through addition.

7. The small-sample target detection method based on fine-grained feature extraction and spatial alignment according to claim 1, characterized in that, The nAP50 metric was used to evaluate the model based on the Siamese ResNet101 network, the feature alignment module, the prototype fusion module, and the detection head.

8. The small-sample target detection method based on fine-grained feature extraction and spatial alignment according to claim 1, characterized in that, The detection head uses a fusion of ROI classification loss, regression loss, and meta-classification loss.

9. A small-sample target detection system based on fine-grained feature extraction and spatial alignment, characterized in that, include: Memory is used to store instructions that can be executed by the processor; A processor for executing instructions to implement the small sample target detection method based on fine-grained feature purification and spatial alignment as described in any one of claims 1-8.

10. A computer-readable medium storing computer program code, characterized in that, The computer program code, when executed by a processor, implements the small sample target detection method based on fine-grained feature purification and spatial alignment as described in any one of claims 1-8.

Images