A few-shot open-set object detection method based on regional epicenter guidance and proxy class boundary enhancement

Abstract

This invention discloses a few-shot open-set target detection method based on regional epicenter guidance and surrogate class boundary enhancement, comprising: extracting image features, using a dual-branch region proposal network to obtain a target score and an epicenter score based on axis alignment and diagonal geometric symmetry, and using the geometric mean of the two as the candidate region quality score; extracting visual features from high-quality candidate regions, combining them with a text encoder to obtain known category text features, and weighted aggregation to construct category visual prototypes; generating surrogate class text and visual prototypes based on text semantic nearest neighbor relationships; jointly optimizing the detection loss, surrogate class alignment loss, and class boundary enhancement loss during training, the latter including parent class balancing, surrogate center constraint, and boundary repulsion loss to stabilize the classifier boundary; during testing, if the maximum known class matching probability of a candidate region is lower than a threshold, it is judged as an unknown class. This invention can simultaneously improve the accuracy of known class detection, new class transfer, and the stability of unknown class rejection.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and intelligent perception technology, and in particular to a method for detecting targets in a few open sets based on regional epicenter guidance and proxy-class boundary enhancement. Background Technology

[0002] Traditional object detection methods are typically based on the closed-set assumption, meaning that the same set of categories is shared during training and testing. While these methods can achieve high detection accuracy when there are sufficient samples and a fixed category space, they often encounter situations in real-world environments where unseen categories, out-of-distribution samples, and complex backgrounds coexist. This can easily lead to misclassifying unknown targets as known categories or directly including unknown targets as background, resulting in both false positives and false negatives.

[0003] In recent years, open-set object detection and open-world object detection have received widespread attention. Meanwhile, vision-language pre-trained models have provided a new technical path for generalizing to unseen categories through cross-modal semantic alignment. However, most existing works assume sufficient known class samples, making it difficult to adapt to the long-tailed distribution and sample scarcity conditions in real-world scenarios. For open-set object detection tasks with few samples, the model must not only form stable category representations on a small number of known class samples but also reliably reject unknown targets not encountered during training during the testing phase. Therefore, it faces the dual challenges of insufficient learning of known classes and difficulty in recognizing unknown classes.

[0004] In a two-stage detection framework, candidate regions are typically generated by a region proposal network. Under limited sample conditions, this network tends to over-bind foreground concepts to appearance patterns of a limited number of known classes, leading to insufficient coverage of unknown targets and weakening the ability of subsequent detection heads to perceive them. On the other hand, due to the limited number of samples for each known class, the classifier struggles to achieve a tight intra-class distribution and clear inter-class boundaries. When an unknown target appears during the testing phase, the model tends to attach it to a semantically similar known class region.

[0005] While existing methods for enhancing centrality or objectivity can improve candidate region quality to some extent, most rely primarily on horizontal and vertical positional relationships, failing to adequately characterize the overall geometric symmetry of points within the bounding box. Furthermore, simply relying on visual samples to construct class boundaries is difficult to maintain stability under limited sample conditions. Therefore, how to simultaneously improve the quality of candidate region generation, enhance the stability of class prototypes, and explicitly reinforce fragile boundaries between known classes remains a pressing technical challenge in the field of limited-sample open-set object detection. Summary of the Invention

[0006] To address the problems of insufficient coverage of unknown targets in the candidate region generation stage, fragile boundaries between known and unknown classes in the classification stage, and unstable class representation under few-sample conditions in existing few-sample open-set target detection methods, this invention provides a few-sample open-set target detection method based on regional epicenter guidance and surrogate class boundary enhancement.

[0007] A few-sample open-set target detection method based on regional epicentrality guidance and surrogate boundary enhancement includes: The input image is acquired and a shared feature map is obtained through a backbone feature extraction network; The shared feature map is input into a dual-branch region proposal network, which includes a target prediction branch, an epicenter prediction branch, and a bounding box regression branch. The epicenter prediction branch constructs an epicenter target value based on the axial alignment symmetry and diagonal symmetry of the sampling points relative to the target box to supervise the prediction of the epicenter score, thereby obtaining the target score, epicenter score, and bounding box regression value of the candidate region. The geometric mean of the target score and the epicenter score is calculated as the candidate region quality score. High-quality candidate regions are selected based on the quality score, and region of interest features are extracted and semantic mapping is performed on the high-quality candidate regions to obtain the region visual features. The text encoder is used to obtain the text features of the known categories based on the text prompts of the known categories, and the visual features of the high-quality candidate regions are weighted and aggregated according to the quality score to obtain the visual prototypes of each known category. Based on the semantic similarity between known category text features, the semantic nearest neighbor category of each known category is determined. By interpolating the text features and visual prototypes of adjacent known categories, a proxy class text prototype and a proxy class visual prototype are generated. The proxy class is only used for boundary modeling during the training phase and is not used as the output category during the testing phase. During the training phase, the detection model is trained by jointly optimizing the total loss function. The total loss function includes at least the region proposal network loss, the image-text alignment and regression loss of the detection branch, the proxy class alignment loss, and the class boundary enhancement loss. The class boundary enhancement loss is used to constrain the proxy class to maintain a balanced affiliation in the semantic space of the two semantically adjacent parent classes and to be located in the middle region, while pushing the real class samples away from the boundary region corresponding to the proxy class. During the testing phase, the matching probability of candidate regions with the text features of each known category is calculated. When the maximum matching probability is lower than the preset open set judgment threshold, the candidate region is identified as an unknown category; otherwise, the detection result of the corresponding known category is output.

[0008] In one embodiment of the present invention, the process of constructing the epicenter target value of the epicenter prediction branch includes: For a positive sample location located inside the target box, calculate the distance from that location to the left, right, top, and bottom boundaries of the target box; Based on the distance, an axis-aligned symmetry metric and a diagonal symmetry metric are constructed respectively. The axis-aligned symmetry metric reflects the positional balance of the sampling points in the horizontal and vertical directions, and the diagonal symmetry metric reflects the positional balance of the sampling points in the diagonal direction. The geometric mean of the axis-aligned symmetry measure and the diagonal symmetry measure is used as the epicentrality target value.

[0009] In one embodiment of the present invention, the region proposal network loss includes target-oriented branch loss, bounding box regression branch loss, and epicenter branch loss; wherein, the epicenter branch loss is the average of the absolute values ​​of the difference between the predicted epicenter value and the corresponding target epicenter value at each positive sample location.

[0010] In one embodiment of the present invention, the visual prototypes of each known category are constructed as follows: the visual features of each high-quality candidate region belonging to the same category are weighted and summed with weights formed by normalization of their respective quality scores, and then the weighted summation result is normalized to obtain the visual prototype of the category in the joint visual-text semantic space.

[0011] In one embodiment of the present invention, the proxy text prototype and proxy visual prototype are generated as follows: for any known category, another known category with the highest cosine similarity to its text features is selected as the semantic nearest neighbor category; interpolation coefficients sampled from a preset distribution are introduced, and the text features of this category and the text features of the semantic nearest neighbor category are linearly interpolated and normalized using the interpolation coefficients to obtain the proxy text prototype; the visual prototype of this category and the visual prototype of the semantic nearest neighbor category are linearly interpolated and normalized using the same interpolation coefficients to obtain the proxy visual prototype.

[0012] In one embodiment of the present invention, the proxy class alignment loss is constructed by minimizing the L2 distance between the visual prototype of the proxy class and the corresponding text prototype of the proxy class, and is used to constrain the consistency between the visual prototype and the text prototype in the joint semantic space.

[0013] In one embodiment of the present invention, the class boundary enhancement loss includes: The parent class balance loss is constructed based on the normalized probability distribution of the matching scores between the visual prototype of the proxy class and the text prototypes of the two parent classes, and is used to promote the probability distribution to tend to be balanced, thereby constraining the proxy class from being overly biased towards either parent class in the semantic space. The proxy center constraint loss is constructed based on the difference between the average matching score between the proxy visual prototype and the two parent text prototypes, and the matching score between the proxy visual prototype and its own proxy text prototype. It is used to encourage the proxy visual prototype to move closer to the semantic position of its proxy text prototype. Boundary repulsion loss is constructed based on the matching score between the visual features of the real class candidate region and the corresponding proxy class text prototype, as well as the difference between the matching score of the visual feature and the real class text prototype. It is used to push the visual feature away from the boundary region corresponding to the proxy class and closer to the region where the real class prototype is located.

[0014] In one embodiment of the present invention, the image-text alignment and regression loss of the detection branch includes: The image-text alignment loss is constructed based on the matching score between the visual features of the candidate region and the text features of each known category, with the annotation of whether the candidate box belongs to the corresponding known category as the supervision signal. Bounding box regression loss is used to constrain the consistency between the regression results of positive candidate boxes and the true target boxes.

[0015] In one embodiment of the present invention, the total loss function is composed of the region proposal network loss, the image-text alignment and regression loss of the detection branch, the proxy class alignment loss, and the weighted class boundary enhancement loss; wherein, the contribution of the class boundary enhancement loss in the training process is adjusted by a preset boundary enhancement weight hyperparameter.

[0016] In one embodiment of the present invention, during the testing phase, only the main detection branch is retained for inference, and the calculation of the seismic score, the calculation of the proxy class generation, and the calculation of the class boundary enhancement loss of the seismic prediction branch are not performed; the determination of the unknown category depends only on the comparison result between the maximum matching probability and the preset open set determination threshold.

[0017] Compared with the prior art, the present invention has at least the following beneficial effects: (1) By introducing a regional epicenter guidance mechanism, the coverage quality of candidate regions for real target subjects under few-sample conditions is significantly improved. Existing target enhancement methods mostly rely on the positional relationship in the horizontal and vertical directions, which does not adequately characterize the overall geometric symmetry of the sampling points located inside the target box. Under few-sample conditions, it is easy to over-bind the foreground concept to the appearance pattern of a limited number of known classes, resulting in insufficient quality of candidate regions generated for unknown targets. This invention adds an epicenter prediction branch to the region proposal network, and uses the axial alignment symmetry and diagonal symmetry of the sampling points and the target box to jointly construct the epicenter target value, which can comprehensively measure the centrality and geometric balance of the sampling points located inside the target subject from multiple directions; furthermore, the geometric mean of the target score and the epicenter score is used as the basis for evaluating the quality of candidate regions, so that the candidate region selection process takes into account both foreground attributes and geometric center attributes. This design is a class-independent geometric prior and does not rely on specific class appearance information. Therefore, under few-sample conditions, it can effectively improve the coverage ability of candidate regions for known and unknown class targets, and provide higher quality candidate input for subsequent detection and rejection.

[0018] (2) By constructing proxy classes in the joint visual-text semantic space and applying boundary enhancement constraints, the fragile boundaries between known classes are explicitly reinforced, reducing the risk of unknown targets being incorrectly absorbed by semantically similar known classes. Under few-sample conditions, due to the limited number of samples in each known category, the classifier is difficult to form a tight intra-class distribution and clear inter-class boundaries. Unknown targets are easily absorbed into a semantically similar known category during the testing phase. Based on the semantic nearest neighbor relationship between the text features of known categories, this invention generates proxy class text prototypes and proxy class visual prototypes between adjacent known classes, using the proxy class as an explicit semantic anchor point for the boundary transition region; and designs a class boundary enhancement training objective that includes parent class balance loss, proxy center constraint loss, and boundary repulsion loss, constraining the proxy class to maintain a balanced affiliation in the semantic space and be located in the middle region from multiple perspectives, while pushing real category samples away from the boundary region. The above design enables the classifier to form a stable transition structure between known classes. Even when facing unknown category targets not seen in training during the testing phase, their visual features are more likely to fall into the boundary region between known classes rather than being classified into a certain known class, thereby reliably achieving unknown class rejection through a preset threshold. The proxy class only participates in boundary modeling during the training phase and is not used as an output class during the testing phase, thus avoiding any impact on the inference process.

[0019] (3) By jointly optimizing the hierarchical training objectives of region proposal loss, detection branch loss, surrogate class alignment loss, and class boundary enhancement loss, the synergistic improvement of known class detection, few-sample new class transfer, and unknown class rejection capability is achieved. This invention unifies the epicenter learning in the candidate region generation stage, the image-text semantic alignment in the detection stage, the cross-modal consistency constraint of surrogate classes, and the class boundary shaping constraint into a single training framework. Each loss term has a clear division of labor and works synergistically: the region proposal network loss is responsible for improving the quality of candidate regions, the detection branch loss is responsible for ensuring the semantic alignment and localization accuracy of known categories, the surrogate class alignment loss ensures the consistency of surrogate classes between visual and textual modalities, and the class boundary enhancement loss is responsible for strengthening the boundary shaping and unknown class rejection capability. This hierarchical training objective design enables the model to simultaneously achieve known class detection performance, new class generalization capability, and unknown class discrimination capability with a small number of known category samples, overcoming the problem that it is difficult to achieve all three in the prior art.

[0020] (4) The inference stage relies solely on the main detection branch and a single threshold for determining unknown classes, requiring no additional network branches or complex post-processing. This results in high inference efficiency and strong feasibility for engineering deployment. In the testing phase, this invention removes the additional computation, proxy class generation, and boundary enhancement modules from the epicenter prediction branch, retaining only the main detection branch for image-text matching inference. The determination of unknown categories is based solely on the comparison between the maximum known class matching probability and a preset threshold, without introducing additional hyperparameters or complex post-processing logic. Computational resource comparison experiments show that this invention's training time is basically on par with existing representative methods, and its inference time is within the same order of magnitude. The overall computational overhead is controllable, indicating that this invention has good real-time performance and usability in actual deployment.

[0021] (5) Experimental results show that the present invention achieves excellent performance on multiple few-shot open set detection benchmarks, verifying the effectiveness and advancement of the technical solution. Based on experimental data, the recall rate of the unknown class in the VOC-COCO dataset under 10-shot and 30-shot conditions reached 38.92% and 40.36%, respectively; in the VOC10-5-5 dataset under 1-shot, 3-shot, 5-shot, and 10-shot conditions, the performance of new class detection was improved by 4.76, 1.14, 2.79, and 4.79 percentage points, respectively, compared with the strong baseline method. At the same time, the AOSE index reached the lowest value in the entire table under the 5-shot condition. Qualitative visualization results show that the method of the present invention can generate a more concentrated and complete target response distribution in complex scenes, and the discrimination of unknown targets is more stable and reliable. The above results fully demonstrate the effectiveness and engineering application value of the present invention in simultaneously improving the performance of known class detection, the ability to transfer new classes in few-shot environments, and the stability of unknown class rejection in a few-shot open environment. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a schematic diagram of the overall process of a few-sample open set target detection method based on regional epicentral guidance and proxy-class boundary enhancement provided in an embodiment of the present invention.

[0024] Figure 2 This is a schematic diagram of the category prototype construction provided in an embodiment of the present invention, which compares the differences between uniform weighting and quality weighting.

[0025] Figure 3 This is an example diagram showing a visual comparison of detection results provided in an embodiment of the present invention.

[0026] Figure 4 This is a CAM heatmap visualization comparison example provided in an embodiment of the present invention. Detailed Implementation

[0027] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but the scope of protection of the present invention is not limited to the following embodiments. For those skilled in the art, equivalent substitutions, combinations or conventional modifications made without departing from the concept of the present invention should all fall within the scope of protection of the present invention.

[0028] This invention provides a few-sample open-set target detection method based on regional epicentral guidance and surrogate boundary enhancement, comprising: The input image is acquired and a shared feature map is obtained through a backbone feature extraction network; The shared feature map is input into a dual-branch region proposal network, which includes a target prediction branch, an epicenter prediction branch, and a bounding box regression branch. The epicenter prediction branch constructs an epicenter target value based on the axial alignment symmetry and diagonal symmetry of the sampling points relative to the target box to supervise the prediction of the epicenter score, thereby obtaining the target score, epicenter score, and bounding box regression value of the candidate region. The geometric mean of the target score and the epicenter score is calculated as the candidate region quality score. High-quality candidate regions are selected based on the quality score, and region of interest features are extracted and semantic mapping is performed on the high-quality candidate regions to obtain the region visual features. The text encoder is used to obtain the text features of the known categories based on the text prompts of the known categories, and the visual features of the high-quality candidate regions are weighted and aggregated according to the quality score to obtain the visual prototypes of each known category. Based on the semantic similarity between known category text features, the semantic nearest neighbor category of each known category is determined. By interpolating the text features and visual prototypes of adjacent known categories, a proxy class text prototype and a proxy class visual prototype are generated. The proxy class is only used for boundary modeling during the training phase and is not used as the output category during the testing phase. During the training phase, the detection model is trained by jointly optimizing the total loss function. The total loss function includes at least the region proposal network loss, the image-text alignment and regression loss of the detection branch, the proxy class alignment loss, and the class boundary enhancement loss. The class boundary enhancement loss is used to constrain the proxy class to maintain a balanced affiliation in the semantic space of the two semantically adjacent parent classes and to be located in the middle region, while pushing the real class samples away from the boundary region corresponding to the proxy class. During the testing phase, the matching probability of candidate regions with the text features of each known category is calculated. When the maximum matching probability is lower than the preset open set judgment threshold, the candidate region is identified as an unknown category; otherwise, the detection result of the corresponding known category is output.

[0029] This embodiment provides a method for target detection in a few open sets based on regional epicenter guidance and surrogate boundary enhancement. For example... Figure 1 As shown, this method is based on a two-stage detection framework, which mainly includes: a backbone feature extraction network, a dual-branch region proposal network, an RoI feature extraction module, a semantic mapping module, a text encoding module, a category prototype construction module, a proxy class generation module, and a boundary enhancement module.

[0030] The training set is denoted as D. train The test set is denoted as D. test The training phase utilizes only the known set of classes C. K Supervised by labeled samples; during the testing phase, a known set of categories C exists simultaneously. K and the set of unknown categories C U .

[0031] For each input image, a feature map F is first extracted through the backbone network, and then the feature map is fed into a dual-branch region proposal network to obtain the target score, epicenter score, and bounding box regression of the candidate region. Based on the target score and epicenter score, the candidate region quality score is calculated. After screening high-quality candidate regions, the region visual features are obtained through RoI feature extraction and semantic mapping. Simultaneously, text features of known categories are obtained using text cue encoding. Based on the quality scores of high-quality candidate regions, a weighted aggregation is performed to construct a visual prototype for each known category. According to the semantic nearest neighbor relationship between categories, proxy text prototypes and proxy visual prototypes for semantically adjacent categories are generated in the joint visual-text semantic space. During the training phase, the model is optimized using proxy class alignment loss and class boundary enhancement loss. During the testing phase, only the main detection branch is retained to calculate the matching probability between the candidate region and the text features of each known category, and unknown class rejection is performed based on a preset threshold.

[0032] In this embodiment, the backbone feature extraction network preferably adopts the ResNet-50 model; the text encoding module preferably adopts the CLIP text encoder; the overall training adopts a two-stage strategy of "base class training + few-sample fine-tuning", and the learning rate of the base class training stage is preferably 2×10. -4 The optimal learning rate for the few-sample fine-tuning stage is 1×10⁻⁶. -4 The optimizer is preferably SGD, the momentum is preferably 0.9, and the weight decay is preferably 5 × 10⁻⁶. -5 .

[0033] The region proposal network is responsible for generating candidate regions that may contain the target. This invention adds an epicenter prediction branch to the traditional target prediction branch and bounding box regression branch, forming a two-branch region proposal network.

[0034] The epicenter prediction branch aims to learn a class-independent geometric prior to measure the geometric center tendency and multi-directional equilibrium of a sample point located inside the real target body.

[0035] For a positive sample position (x, y) located inside the ground truth bounding box, let the coordinates of the ground truth bounding box be (x0, y0, x1, y1), then its distance to the four boundaries is defined as follows:

[0036] in, , , and These represent the distances from the position to the left, right, top, and bottom edges of the actual bounding box, respectively.

[0037] Based on the above distances, axis-aligned symmetry is constructed respectively. Symmetry along the diagonal direction :

[0038] These two symmetries characterize the geometrical balance of the sampling points within the target box in the horizontal and vertical directions, as well as the diagonal direction, respectively. Furthermore, the epicentral target value... and candidate region quality score Further definitions are as follows:

[0039] in, Predict scores for the target branch. This represents the predicted score for the epicenter branch. (Using...) Ranking candidate regions allows proposal quality evaluation to simultaneously consider foreground and geometric center attributes, effectively improving the coverage quality of candidate boxes when facing unknown targets. This quality score provides a class-independent signal under conditions of few samples, effectively enhancing the coverage of candidate regions with the real target and reducing interference from low-quality regions or background noise.

[0040] The proposed total network loss for the dual-branch region is defined as follows:

[0041] in, The target prediction branch loss is used to constrain foreground or background classification; binary cross-entropy loss is preferred. The bounding box regression branch loss is used to constrain the consistency between the positive sample anchor box regression results and the true box offset; the SmoothL1 loss is preferred. The epicentral branch loss is defined as follows:

[0042] in, Indicates the number of positive sample anchor frames. Indicates the first The epicenter prediction value for a positive sample. Indicates the first The epicenter target value of a positive sample. The smaller the value, the more accurately the epicenter branch reflects the degree to which the candidate location is centered within the actual target body.

[0043] like Figure 2 As shown, the construction methods of category prototypes can be divided into two cases: uniform weighting and quality weighting. Figure 2In the left-hand portion of the model, all region features are assigned the same weight during prototype construction; therefore, the category prototype is obtained by a simple averaging of multiple region features. This method is relatively straightforward, but it is easily affected by low-quality region features or noisy samples. Figure 2 In the right-hand side of the graph, different regional features are assigned different weights based on their quality scores. For example, high-quality regions receive larger weights, while low-quality regions account for only a small proportion. Through this weighted aggregation method, the class prototype can retain more discriminative information of the target subject region, reduce the influence of noisy and offset regions on the prototype representation, thereby improving the stability and representativeness of the prototype representation and providing a more reliable feature foundation for subsequent class alignment and open set discrimination.

[0044] For each known category Construct text prompts based on category names And obtain text features through a text encoder. For the first The candidate bounding boxes are used to obtain visual features based on RoIAlign and the semantic mapping module. The definition is as follows:

[0045] The matching score between visual features and text features is defined as the cosine similarity after scaling. Adjusted value: The matching score between the candidate box and the category text is defined as follows:

[0046] in, The default scaling factor in the pre-trained CLIP model is preferably used. Based on the matching score, the image-text alignment loss is defined as follows:

[0047] in, Indicates the number of candidate boxes. Indicates the number of known categories. Indicates the first Does the candidate box belong to the first...? There are 10 known categories. The smaller the value, the more stable the match between the actual candidate box and the corresponding text semantics.

[0048] After obtaining high-quality candidate boxes, a weighted aggregation method is used to construct a prototype category region:

[0049] in, For category The set of positive candidate boxes, For the first The quality score of each candidate box. This is a prototype normalization operation. Because... It is directly related to the candidate box quality score, so high-quality proposals account for a higher proportion in category prototype construction.

[0050] To address the issues of ambiguous boundaries between known categories and the easy absorption of unknown targets into adjacent known categories in open set scenarios with few samples, this invention explicitly constructs proxy classes between semantically adjacent categories in a joint semantic space.

[0051] For category First, based on the similarity between text features, the semantic nearest neighbor category is selected, defined as follows:

[0052] in, These are the category interpolation coefficients. Preferably, they are used... Construct a proxy class text prototype between two parent classes. and proxy class visual prototype :

[0053] To ensure that the visual representation and textual representation of the proxy class remain consistent in a unified semantic space, the proxy class alignment loss is defined as follows:

[0054] The smaller the value, the more consistent the visual prototype and textual prototype of the proxy class are, and the more the proxy class can serve as a stable semantic anchor point for the boundary region.

[0055] To avoid the proxy class from being overly biased towards a certain parent class, and to prevent real class samples from spreading to the boundary region, this invention designs a class boundary enhancement loss composed of parent class balance loss, proxy center constraint loss, and boundary repulsion loss.

[0056] Define the matching score of the proxy class relative to the text prototypes of the two parent classes and the proxy text prototype:

[0057] Based on the above definition, the parent class balance loss and the agency center to constrain losses They are defined as follows:

[0058] in, This is used to ensure that the proxy class maintains a balanced affiliation between the two parent classes, and to prevent the proxy class from being "pulled off" by a single category; This is used to encourage the proxy class to be closer to its own proxy text prototype, rather than just to be near the average similarity of the two parent classes.

[0059] For any true candidate box Let its true category be The corresponding proxy text prototype is The boundary repulsion loss is defined as follows:

[0060] The purpose of this is to increase the loss when the real candidate box is closer to the boundary proxy prototype than the real class prototype, thereby pushing the real samples away from the unstable boundary region.

[0061] Summing the three sub-losses over all known categories yields the total boundary augmentation loss:

[0062] The smaller the value, the smoother and more stable the class boundary, and the lower the probability of an unknown target being mistakenly absorbed into a known class. This loss constrains the geometric structure and semantic stability of the classification boundary from multiple perspectives, enabling the classifier to form a stable transition structure in the boundary region between known classes, significantly reducing the risk of unknown targets being mistakenly absorbed.

[0063] On the ground truth candidate boxes, the main detection branch performs image-text alignment and bounding box regression tasks, with the loss defined as follows:

[0064] in, To detect the regression loss of the head bounding box, which is used to constrain the regression consistency between positive candidate boxes and the ground truth boxes, the SmoothL1 loss is preferred.

[0065] Therefore, the overall training objective of this invention is defined as follows:

[0066] in, To enhance the loss weight at the boundary, The weight hyperparameters for the boundary enhancement loss are used to adjust the contribution of the boundary enhancement module during training. Used to improve proposal quality Semantic alignment and regression for real candidate boxes Used for consistency between text and images in proxy classes. Used for boundary shaping and identification of unknown classes. The above loss terms have clear division of labor and work together to form a complete training objective for few-sample open set detection.

[0067] During the testing phase, only the main detection branch is retained, and it no longer participates in proxy class generation and boundary enhancement modules to ensure inference efficiency. For any candidate box, its sum with all other boxes is calculated first. K The matching probabilities between known class text features are used, and the maximum value is taken as the maximum known class probability.

[0068] The rule for determining the unknown class is: when the probability of the largest known class is lower than the preset open set determination threshold. At that time, that is

[0069] The candidate region is classified as an unknown class; otherwise, the detection result of the known class with the highest probability is output.

[0070] This strategy only compares thresholds, without needing to explicitly output proxy classes during the testing phase. Instead, it directly utilizes the stable boundary shaped during the training phase to achieve unknown class rejection. It has a small number of additional hyperparameters and high feasibility for engineering deployment.

[0071] To verify the effectiveness of the present invention, the visual semantic representation branch adopts the ResNet-50 image encoder pre-trained by RegionCLIP, the candidate region generation network adopts the ResNet-50 pre-trained on ImageNet, the training batch size is preferably 1, the background class loss weight is preferably 0.2, and the focal scaling parameter is preferably 0.5.

[0072] As shown in Table 1, in the VOC-COCO dataset experiments, the R-value of this invention under 10-shot and 30-shot conditions is... U The scores reached 38.92 and 40.36 respectively. In terms of AOSE index, the present invention outperformed the reproducible baseline under multiple settings, indicating that it can maintain a strong ability to distinguish unknown classes while improving the detection capabilities of known and new classes.

[0073] Table 1. Experimental results of FS-OSOD under different shot settings on the VOC-COCO dataset.

[0074]

[0075] As shown in Table 2, in the VOC10-5-5 dataset experiments, the mAP of this invention under 1-shot, 3-shot, 5-shot, and 10-shot conditions is as follows. NCompared to the CED-FOOD baseline, the improvements were 4.76, 1.14, 2.79, and 4.79 percentage points, respectively. Under 5-shot conditions, AOSE achieved the lowest value across the entire table, indicating that the present invention can more effectively suppress the misidentification of unknown targets as known categories under specific settings.

[0076] Table 2. Experimental results of FS-OSOD under different shot settings in the VOC10-5-5 dataset.

[0077]

[0078] Table 3 presents a comparison of computational resources for different methods on the VOC10-5-5 dataset. It can be seen that the training time of the method presented in this invention is basically consistent with CED-FOOD, with a single iteration training time of 0.1957 s. This is very small compared to several representative methods, indicating that the introduced dual-branch RPN and Proxy enhancement modules do not significantly increase training overhead. During the inference phase, the time consumption of the method presented in this invention is 0.03991 s, which is on the same order of magnitude as methods such as ORE, OpenDet*, and CED-FOOD. The overall computational overhead remains controllable, indicating that the method presented in this invention still has good real-time performance and availability in practical deployments.

[0079] Table 3 Comparison of computational resources on the VOC10-5-5 dataset

[0080] Table 4 presents the module ablation experiment results under a 30-shot setting. All experimental results were repeated under 10 different random seeds and averaged to ensure the stability and reliability of the results. The best results in the table are marked in bold. This table is mainly used to analyze the independent roles of the two key components, the dual-branch RPN and the proxy enhancement, as well as their combined effect, thereby verifying the contribution of each module to the overall performance improvement.

[0081] Table 4. Ablation experimental results of each component module under the 30-shot setting.

[0082] Figure 3This paper presents a qualitative comparison of different methods in the open-set object detection task with few samples. It can be observed that the traditional closed-set detection method, FR-CNN, suffers from a significant closed-set bias problem. When the input image contains categories not seen during training, the model tends to incorrectly classify these objects into a known category rather than identifying them as unknown categories. For example, in an open scene containing both giraffes and zebras, FR-CNN misclassifies the giraffe as a "bird" and the zebra as a "chair." This phenomenon indicates that under few-sample conditions, the class discrimination boundaries learned by the model are still not accurate enough, causing unknown categories to be incorrectly included in the known category space.

[0083] In contrast, while OpenDet and CED-FOOD can identify unknown categories to some extent, they still suffer from insufficient stability in complex scenes, such as weak response to small-scale targets and the easy omission of unknown regions. The method of this invention exhibits greater stability across various scenarios, not only more accurately identifying unknown targets but also effectively separating unknown regions while maintaining the detection performance of known categories. This demonstrates that the proposed method has advantages in improving candidate region quality and optimizing category boundaries, thereby enhancing the discriminative power and overall reliability in few-shot open set detection.

[0084] Figure 4 The paper presents a comparison of response heatmaps for different methods. FR-CNN exhibits a relatively dispersed response distribution, tending to concentrate on the background or salient regions, and showing insufficient attention to unknown targets. OpenDet and CED-FOOD improve this issue to some extent, but their responses to small or secondary targets remain inadequate. In contrast, the method proposed in this invention can form a more concentrated and complete response distribution across various scenarios, effectively covering the target area and suppressing background interference. The results demonstrate that the proposed method helps improve the quality of candidate regions and the stability of unknown target recognition.

[0085] This invention is applicable to practical application scenarios such as the perception of rare obstacles in autonomous driving scenarios, object recognition in open environments by robot vision systems, detection of abnormal and suspicious targets in security patrols, and discovery of unknown defect types in industrial quality inspections. It can balance the recognition accuracy of known classes with the ability to reject unknown classes under the condition of providing only a small number of known class labeled samples, thus greatly improving the generalization and security of the perception system.

[0086] In summary, this invention extracts input image features based on a backbone network; in the region proposal stage, it sets up a target branch, an epicenter branch, and a bounding box regression branch, and constructs an epicenter metric using the geometric symmetry of the axis alignment and diagonal direction between the sampling points and the target box, and uses the geometric mean of the target score and the epicenter score as the basis for evaluating the quality of candidate regions; it performs RoI feature extraction and semantic mapping on high-quality candidate regions, and obtains known class text features by combining text prompt encoding; it constructs class prototypes according to the quality weights of candidate regions, and generates surrogate class text prototypes and region prototypes based on the semantic nearest neighbor relationship of the text; in the training stage, it optimizes the model by jointly using detection loss, surrogate class alignment loss, and class boundary enhancement loss, where the class boundary enhancement loss includes parent class balance constraint, surrogate center constraint, and boundary repulsion constraint; in the testing stage, it retains only the main detection branch, and judges the matching probability of candidate regions with all known class text features, and outputs an unknown class when the maximum known class probability is lower than a set threshold. This invention can simultaneously improve the detection performance of known classes, the transfer capability of new classes with few samples, and the stability of unknown class rejection in open environments with few samples. It is applicable to scenarios such as autonomous driving, robot perception, security inspection, and industrial quality inspection.

[0087] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for detecting open-set targets with few samples based on regional epicentrality guidance and surrogate boundary enhancement, characterized in that, include: The input image is acquired and a shared feature map is obtained through a backbone feature extraction network; The shared feature map is input into a dual-branch region proposal network, which includes a target prediction branch, an epicenter prediction branch, and a bounding box regression branch. The epicenter prediction branch constructs an epicenter target value based on the axial alignment symmetry and diagonal symmetry of the sampling points relative to the target box to supervise the prediction of the epicenter score, thereby obtaining the target score, epicenter score, and bounding box regression value of the candidate region. The geometric mean of the target score and the epicenter score is calculated as the candidate region quality score. High-quality candidate regions are selected based on the quality score, and region of interest features are extracted and semantic mapping is performed on the high-quality candidate regions to obtain the region visual features. The text encoder is used to obtain the text features of the known categories based on the text prompts of the known categories, and the visual features of the high-quality candidate regions are weighted and aggregated according to the quality score to obtain the visual prototypes of each known category. Based on the semantic similarity between known category text features, the semantic nearest neighbor category of each known category is determined. By interpolating the text features and visual prototypes of adjacent known categories, a proxy class text prototype and a proxy class visual prototype are generated. The proxy class is only used for boundary modeling during the training phase and is not used as the output category during the testing phase. During the training phase, the detection model is trained by jointly optimizing the total loss function. The total loss function includes at least the region proposal network loss, the image-text alignment and regression loss of the detection branch, the proxy class alignment loss, and the class boundary enhancement loss. The class boundary enhancement loss is used to constrain the proxy class to maintain a balanced affiliation in the semantic space of the two semantically adjacent parent classes and to be located in the middle region, while pushing the real class samples away from the boundary region corresponding to the proxy class. During the testing phase, the matching probability of candidate regions with the text features of each known category is calculated. When the maximum matching probability is lower than the preset open set judgment threshold, the candidate region is identified as an unknown category; otherwise, the detection result of the corresponding known category is output.

2. The method of claim 1, wherein, The process of constructing the target value of the epicenter prediction branch includes: For a positive sample location located inside the target box, calculate the distance from that location to the left, right, top, and bottom boundaries of the target box; Based on the distance, an axis-aligned symmetry metric and a diagonal symmetry metric are constructed respectively. The axis-aligned symmetry metric reflects the positional balance of the sampling points in the horizontal and vertical directions, and the diagonal symmetry metric reflects the positional balance of the sampling points in the diagonal direction. The geometric mean of the axis-aligned symmetry measure and the diagonal symmetry measure is used as the epicentrality target value.

3. The method of claim 1, wherein, The region proposal network loss includes target branch loss, bounding box regression branch loss, and epicenter branch loss; wherein, the epicenter branch loss is the average of the absolute values ​​of the difference between the predicted epicenter value and the corresponding target epicenter value at each positive sample location.

4. The method of claim 1, wherein, The visual prototypes of each known category are constructed as follows: the visual features of each high-quality candidate region belonging to the same category are weighted and summed using the weights formed by normalization of their respective quality scores, and then the weighted sum is normalized to obtain the visual prototype of the category in the joint visual-text semantic space.

5. The method of claim 1, wherein, The generation method of the proxy text prototype and proxy visual prototype is as follows: For any known category, select another known category with the highest cosine similarity to its text features as the semantic nearest neighbor category; introduce interpolation coefficients sampled from a preset distribution, and use the interpolation coefficients to perform linear interpolation and normalization between the text features of this category and the text features of the semantic nearest neighbor category to obtain the proxy text prototype; use the same interpolation coefficients to perform linear interpolation and normalization between the visual prototype of this category and the visual prototype of the semantic nearest neighbor category to obtain the proxy visual prototype.

6. The method according to claim 5, characterized in that, The proxy class alignment loss is constructed by minimizing the L2 distance between the visual prototype of the proxy class and the corresponding text prototype of the proxy class, and is used to constrain the consistency between the visual prototype and the text prototype in the joint semantic space.

7. The method of claim 1, wherein, The class boundary enhancement loss includes: The parent class balance loss is constructed based on the normalized probability distribution of the matching scores between the visual prototype of the proxy class and the text prototypes of the two parent classes, and is used to promote the probability distribution to tend to be balanced, thereby constraining the proxy class from being overly biased towards either parent class in the semantic space. The proxy center constraint loss is constructed based on the difference between the average matching score between the proxy visual prototype and the two parent text prototypes, and the matching score between the proxy visual prototype and its own proxy text prototype. It is used to encourage the proxy visual prototype to move closer to the semantic position of its proxy text prototype. Boundary repulsion loss is constructed based on the matching score between the visual features of the real class candidate region and the corresponding proxy class text prototype, as well as the difference between the matching score of the visual feature and the real class text prototype. It is used to push the visual feature away from the boundary region corresponding to the proxy class and closer to the region where the real class prototype is located.

8. The method of claim 1, wherein, The image-text alignment and regression loss of the detection branch include: The image-text alignment loss is constructed based on the matching score between the visual features of the candidate region and the text features of each known category, with the annotation of whether the candidate box belongs to the corresponding known category as the supervision signal. Bounding box regression loss is used to constrain the consistency between the regression results of positive candidate boxes and the true target boxes.

9. The method of claim 1, wherein, The total loss function is composed of the region proposal network loss, the image-text alignment and regression loss of the detection branch, the proxy class alignment loss, and the weighted class boundary enhancement loss, which are weighted sums. The contribution of the class boundary enhancement loss during training is adjusted by a preset boundary enhancement weight hyperparameter.

10. The method of claim 1, wherein, During the testing phase, only the main detection branch is retained for inference, and the calculation of the seismic score, proxy class generation, and class boundary enhancement loss related to the seismic prediction branch are not performed; the determination of the unknown category depends only on the comparison result between the maximum matching probability and the preset open set determination threshold.

Images