A few-shot object detection method fusing semantic guided contrastive learning and cross-modal calibration

Abstract

The present application belongs to the field of computer vision target detection, and in particular to a few-shot target detection method fusing semantic guided contrast learning and cross-modal calibration, comprising: relying on DCFS two-stage decoupling detection architecture to jointly construct a unified semantic guidance framework with CLIP visual language model. During training, the semantic guided contrast learning module generates a sharpened semantic weight matrix, and combines the memory prototype queue and the sample filtering strategy to construct a weighted contrast loss to stabilize the class features; the text guided feature enhancement module optimizes the RoI features of the classification branch with the help of sparse attention and gating mechanism. In the inference stage, the cross-modal anchor semantic calibration module introduces negative semantic anchors to expand the semantic set, and the detection confidence is corrected through probability competition and score fusion. The present application realizes full-process semantic enhancement, alleviates the problems of class confusion, prototype fluctuation and high-confidence false detection in few-shot detection, and adapts to few-shot detection scenes such as industry and remote sensing.

Description

Technical Field

[0001] This invention belongs to the field of computer vision object detection, and in particular refers to a few-shot object detection method that integrates semantic-guided contrastive learning and cross-modal calibration. Background Technology

[0002] Few-shot object detection (FSOD) aims to detect new object categories using a very small number of labeled samples, effectively addressing the pain points of scarce and high-cost labeled data in scenarios such as autonomous driving, remote sensing monitoring, industrial inspection, and medical imaging, and has significant application value. Most current mainstream FSOD technologies employ a two-stage training framework of "base class pre-training + new class fine-tuning." In recent years, visual language pre-trained models such as CLIP, with their excellent cross-modal semantic alignment capabilities, have been gradually introduced into few-shot detection tasks to assist in optimizing category feature representation and model generalization ability.

[0003] Currently, existing few-shot object detection methods can be mainly divided into four categories: fine-tuning optimization methods, contrastive learning methods, visual-language cross-modal fusion methods, and classification score calibration methods. Fine-tuning optimization methods alleviate overfitting and feature shift problems by constraining fine-tuning parameters and decoupling gradients and loss functions, but they rely solely on pure visual feature learning, resulting in limited feature representation capabilities under few-shot conditions and a tendency for class confusion. Contrastive learning methods optimize feature distribution through positive and negative sample constraints, but often employ a uniform negative sample constraint strategy, failing to consider semantic similarity differences between categories and exhibiting poor ability to distinguish similar categories. Visual-language fusion methods introduce textual semantics to assist modeling, but are mostly adapted to open-vocabulary detection scenarios and only utilize semantic information in the classification stage, failing to fully leverage the guiding role of cross-modal knowledge in feature learning. Classification score calibration methods optimize detection results through post-processing, but rely solely on visual features and classification probability calibration, lacking explicit semantic constraints and struggling to suppress high-confidence false detections.

[0004] Overall, existing few-sample target detection technologies still have many core shortcomings, as follows: First, the contrastive learning constraint mechanism is unreasonable. Existing methods only construct positive and negative sample constraints based on category labels, and apply a uniform constraint strategy to all negative samples. They do not consider the semantic similarity differences between different categories, and cannot effectively widen the feature gap between similar categories. The problem of feature confusion between similar categories is serious, and the model's fine-grained discrimination ability is insufficient.

[0005] Second, the stability and discriminativeness of new category features are poor. Existing methods rely on only a small number of visual samples to learn new category features. The scarcity of samples leads to insufficient feature learning and a chaotic feature distribution, making it difficult to form stable and highly discriminative category features, thus limiting the model's generalization ability.

[0006] Third, cross-modal semantic information is not fully utilized. Existing visual-language fusion schemes only introduce textual semantics in the classification stage, without integrating cross-modal semantic priors into core processes such as target feature learning and feature enhancement. As a result, the utilization rate of semantic knowledge is low, and it is impossible to fundamentally improve the quality of features from a small number of samples.

[0007] Fourth, the category prototype estimation has low accuracy and poor stability. Existing methods rely on a small number of random training samples to construct category prototypes. Due to the small number of samples and the strong randomness of sampling, the category center estimation bias is large, resulting in poor feature alignment and seriously affecting the classification and detection accuracy.

[0008] Fifth, the detection and calibration mechanism has flaws, leading to frequent high-confidence false detections. Existing calibration methods rely solely on visual features and classification results for post-processing, lacking semantic constraints on background and non-target interference regions. This makes it difficult to effectively suppress high-confidence false detections caused by interference, resulting in low reliability of detection results.

[0009] Sixth, there is a lack of a unified semantic guidance mechanism, resulting in poor module collaboration. Existing technologies optimize feature learning, category discrimination, and result calibration independently, without relying on visual language priors to build a unified semantic guidance system. This makes it impossible to systematically improve the performance of few-sample detection, and performance improvement faces significant bottlenecks.

[0010] In summary, existing few-shot target detection methods generally suffer from technical problems such as weak feature discrimination ability, insufficient semantic utilization, inaccurate prototype estimation, high false detection rate, and poor module coordination, making it difficult to meet the requirements of high-precision and high-reliability new category detection in few-shot scenarios. Therefore, it is urgent to propose a new few-shot target detection scheme to address the above-mentioned technical shortcomings. Summary of the Invention

[0011] Therefore, the technical problem to be solved by this invention is to overcome the problems of category feature confusion, insufficient expression of new category features, unstable category prototype estimation, insufficient utilization of cross-modal semantic information, and high-confidence false detection in the background region caused by the scarcity of labeled samples in the few-sample target detection scenario. Thus, it provides a few-sample target detection method that integrates semantically guided contrastive learning and cross-modal calibration, and constructs a unified semantic guidance framework to realize semantic enhancement of the entire process of feature learning, category discrimination, and result calibration.

[0012] According to one aspect of the present invention, a few-shot target detection method integrating semantically guided contrastive learning and cross-modal calibration is provided, comprising: Obtain a few-sample object detection training dataset, which includes sample images and their corresponding class labels and ground truth bounding boxes; A few-shot target detection framework based on a visual language model is constructed. The detection framework includes a feature extraction network, a text-guided feature enhancement module, a semantic-guided contrastive learning module, and a cross-modal anchoring semantic calibration module. During the training phase, the feature extraction network extracts training images and their corresponding category labels and ground truth bounding boxes from the training dataset, and inputs the training images into the feature extraction network to obtain the original RoI features of multiple regions of interest; For each original RoI feature, the first and second processes are executed in parallel, and the first and second processes share the original RoI features. The first process is as follows: using the text-guided feature enhancement module, based on the semantic information of each category of text generated by the visual language model, the original RoI features are semantically modulated to obtain enhanced RoI features, and the enhanced RoI features are input into the classification branch for category prediction, and the original RoI features without semantic modulation are input into the regression branch for bounding box prediction to obtain classification prediction results and regression prediction results. The second process is as follows: using the semantic-guided contrastive learning module, constructing a semantic similarity matrix between categories based on the text embeddings of each category provided by the visual language model, applying weighted constraints to the negative samples in the contrastive learning based on the semantic similarity matrix, applying contrastive learning loss to the original RoI features, and updating the dynamic prototype memory queue of each category using the original RoI features that meet the preset conditions. The detection framework is updated based on the differences between the classification prediction results and the category labels, the differences between the regression prediction results and the ground truth bounding boxes, and the contrastive learning loss. During the inference phase, candidate detection boxes are obtained based on the classification prediction results and the regression prediction results, and the visual features of each candidate detection box are extracted using the image encoder of the visual language model. Using the cross-modal anchoring semantic calibration module, an extended semantic set is constructed, which includes text embeddings of each category and predefined negative semantic anchor text embeddings. The visual features of the candidate detection box are semantically matched with the extended semantic set to obtain the calibrated category confidence. The original classification scores of the candidate detection boxes are corrected based on the calibrated category confidence, and the final detection results are output.

[0013] As an optional technical solution, constructing a semantic similarity matrix between categories includes: The category names are input into the text encoder of the visual language model using a unified prompt template to obtain the text embedding vectors corresponding to each category; Calculate the cosine similarity between each text embedding vector to obtain the initial similarity matrix; A nonlinear sharpening operation is performed on the initial similarity matrix to obtain the semantic similarity matrix between categories, whose matrix elements are... satisfy: , in, This represents the initial similarity matrix between the text embedding vectors of class i and class j. This refers to the sharpening index.

[0014] As an optional technical solution, the dynamic prototype memory queues for various categories are updated using the original RoI features that meet preset conditions, including: Obtain the category label and intersection-union ratio (IU) of the candidate bounding boxes corresponding to each original RoI feature, wherein the IU is the intersection-union ratio between the candidate bounding box and its corresponding ground truth bounding box; If the category label is a non-background category and the cross-union ratio is greater than the preset cross-union ratio threshold, then the original RoI feature is added to the dynamic prototype memory queue of its category. The dynamic prototype memory queue adopts a first-in-first-out (FIFO) update strategy. Each category corresponds to one memory queue, and the capacity of each memory queue is set according to the number of labeled samples of that category in the training set. When the number of features in the queue reaches the preset capacity limit, the feature at the head of the queue is removed first, and then the current feature is stored. The current prototype for each category is obtained by averaging all the original RoI features stored in the dynamic prototype memory queue for each category.

[0015] As an optional technical solution, a contrastive learning loss is applied to the original RoI features, including: A semantically weighted contrastive loss function is constructed. The similarity between the current original RoI feature and its corresponding positive prototype is calculated using a first weight, and the similarity between the current original RoI feature and each negative prototype is calculated using the corresponding weights in the inter-category semantic similarity matrix. The semantically weighted contrastive loss function is expressed as follows: , in, The original RoI feature set that meets the preset quality conditions, The number of samples in the original RoI feature set. For the set of positive prototype indices of category i, For the set of negative prototype indices of category i, For positive sample similarity, This represents the semantically weighted similarity of negative samples.

[0016] As an optional technical solution, the enhanced RoI features include: The original RoI features are mapped to the text embedding space of the visual language model through a linear projection layer and then subjected to L2 normalization to obtain the projected features. Calculate the similarity between the projected features and the text embeddings of each category, select the top k categories of text embeddings with the highest similarity, and perform weighted aggregation on the selected text embeddings using the Softmax normalization function with temperature parameter to obtain aggregated text features; The aggregated text features are input into the first multilayer perceptron and the second multilayer perceptron respectively to generate channel-level scaling factors and channel-level bias terms; The channel-level scaling factor and channel-level bias term are combined with the learnable channel-level gating vector to perform affine modulation on the original RoI features, resulting in enhanced RoI features.

[0017] As an optional technical solution, the calibrated category confidence is obtained, including: The predefined negative semantic anchor text embedding is obtained by encoding a set of text prompts describing non-target regions, wherein the text prompts include text descriptions of blurred images, random textures, and background regions; The text embeddings of each category and the text embeddings of each negative semantic anchor are concatenated along the feature dimension to form an extended semantic set; Calculate the dot product similarity between the visual features of the candidate detection box and each embedding vector in the extended semantic set; The dot product similarity is converted into a semantically calibrated category probability distribution using a Softmax function with a temperature parameter. The semantically calibrated category probability distribution is used as the calibrated category confidence.

[0018] As an optional technical solution, the expression for the semantically calibrated category probability distribution is as follows: , in, represents the visual feature vector of the candidate detection box. To expand the semantic set The text embedding vector of the y-th category in the text. To expand the semantic set The text embedding vector of the l-th category, The total number of categories, The number of negative semantic anchor text embeddings. For temperature parameters, This represents the probability of belonging to category y after calibration.

[0019] As an optional technical solution, the original classification score of the candidate detection box is corrected based on the calibrated category confidence, including: Determine whether the original classification score of the candidate detection box falls within the preset confidence interval: If the score falls within the confidence range, the calibrated category confidence score and the original classification score are weighted and fused according to a preset fusion coefficient to obtain the final calibration score. If the score does not fall within the confidence interval, the original classification score is retained unchanged.

[0020] According to another aspect of the present invention, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the few-sample target detection method that integrates semantic-guided contrastive learning and cross-modal calibration as described above.

[0021] According to another aspect of the present invention, a non-transitory computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the few-sample target detection method that integrates semantically guided contrastive learning and cross-modal calibration as described above.

[0022] Compared with the prior art, the above-described technical solution of the present invention has the following advantages: This invention relies on an integrated technical solution of cross-modal semantic probability competition, negative semantic anchor probability diversion, interval selective calibration, and multi-source score weighted fusion, achieving outstanding and non-obvious technical advantages: First, this invention relies on the global cross-modal semantic prior provided by the CLIP pre-trained model to construct a joint probability competition mechanism of target category embedding and negative semantic anchor embedding, so that the probability weights corresponding to the background and low-quality candidate boxes are diverted to the negative semantic anchor, thereby achieving active suppression of high-confidence false detection samples in the semantic dimension.

[0023] Second, compared with existing techniques that rely solely on visual feature similarity for calibration, this invention upgrades confidence calibration to a cross-modal semantic probability competition level, making up for the lack of explicit semantic discrimination constraints in traditional calibration schemes and improving the semantic interpretability of the confidence correction process.

[0024] Third, this invention innovatively introduces negative semantic anchor text embedding for typical false detection scenarios, constructs independent probability diversion channels for non-target and noise regions, and achieves natural dilution of background candidate box classification confidence from the probability calculation level, filling the technical gap of lacking background semantic modeling in existing few-sample detection confidence calibration schemes.

[0025] Fourth, the cross-modal anchoring semantic calibration module designed in this invention is a parameterless post-processing component on the inference side, requiring no additional fine-tuning of network parameters. It is pluggable and compatible with various two-stage few-sample detection baseline architectures, and the incremental inference computing power is controllable. At the same time, it is combined with an interval selective calibration strategy and a dual-score weighted fusion mechanism to differentiate between high-reliability predictions and fuzzy interval samples, avoiding the problem of correct target confidence shift caused by global semantic correction. It has good generalization robustness and detection stability in multiple scenarios and multiple datasets. Attached Figure Description

[0026] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0027] Figure 1 This is a schematic diagram of a few-shot target detection framework based on the CLIP visual language model, according to an embodiment of the present invention. Figure 2 This is a schematic diagram of the structure of the semantic-guided contrastive learning module according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of the text guidance feature enhancement module according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the cross-modal anchoring semantic calibration module according to an embodiment of the present invention; Figure 5 This is a comparison of the nAP performance results of various algorithms under different sample ratios on the Pascal Voc dataset; Figure 6 This is a comparison of the nAP performance metrics of various algorithms under different few-sample annotation conditions on the MS COCO dataset. Detailed Implementation

[0028] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0029] Reference Figures 1-4 As shown, this invention provides a few-shot target detection method that integrates semantically guided contrastive learning and cross-modal calibration, comprising: Obtain a few-sample object detection training dataset, which includes sample images and their corresponding class labels and ground truth bounding boxes; Based on the DCFS two-stage decoupled classification loss detection model as the backbone, a few-shot target detection framework based on the CLIP visual language model is constructed. The detection framework includes a feature extraction network, a text-guided feature enhancement module (abbreviated as "TFE" in the figure), a semantic-guided contrastive learning module (abbreviated as "CGCL" in the figure), and a cross-modal anchored semantic calibration module (abbreviated as "CASC" in the figure).

[0030] During the training phase, the feature extraction network extracts training images and their corresponding category labels and ground truth bounding boxes from the training dataset, and inputs the training images into the feature extraction network to obtain the original RoI features of multiple regions of interest; For each original RoI feature, the first and second processes are executed in parallel, and the first and second processes share the original RoI features. The first process is as follows: using the TFE module, based on the semantic information of each category of text generated by the CLIP visual language model, the original RoI features are subjected to cross-modal alignment, Top-k sparse semantic aggregation and gated FiLM channel modulation operations to obtain enhanced RoI features. The enhanced RoI features are then input into the classification branch for category prediction, and the original RoI features without semantic modulation are input into the regression branch for bounding box prediction to obtain classification prediction results and regression prediction results. The second process is as follows: using the CGCL module, constructing a semantic similarity matrix between categories based on the text embeddings of each category provided by the visual language model, applying weighted constraints to negative samples in contrastive learning based on the semantic similarity matrix, applying contrastive learning loss to the original RoI features, and updating the dynamic prototype memory queues of each category using the original RoI features that meet the preset conditions. The detection framework is updated based on the differences between the classification prediction results and the category labels, the differences between the regression prediction results and the ground truth bounding boxes, and the contrastive learning loss. During the inference phase, candidate detection boxes are obtained based on the classification prediction results and the regression prediction results, and the visual features of each candidate detection box are extracted using the image encoder of the visual language model. Using the CASC module, an extended semantic set is constructed that includes text embeddings of each category and predefined negative semantic anchor text embeddings. The visual features of the candidate detection boxes are semantically matched with the extended semantic set to obtain the calibrated category confidence. The original classification scores of the candidate detection boxes are corrected based on the calibrated category confidence, and the final detection results are output.

[0031] Furthermore, before executing the few-shot target detection method that integrates semantic-guided contrastive learning and cross-modal calibration, the acquired complete target detection dataset is divided into two categories: base class and new class. All samples containing only base class target images and annotations are summarized into the base class dataset, and all a small number of samples containing only new class target images and annotations are summarized into the new class few-shot fine-tuning subset. The base class dataset and the new class few-shot fine-tuning subset are stored independently with no overlap between samples.

[0032] A two-stage decoupled classification loss basic detection skeleton of DCFS is built, a feature extraction network is connected in series, a CLIP visual language model is connected, and the TFE module and CGCL module are fully assembled.

[0033] The DCFS basic network was fully pre-trained using a large number of labeled samples in the base class dataset. After pre-training, the underlying backbone parameters of the feature extraction network were frozen, and only the trainable parameters in the subsequent fine-tuning stage were opened to obtain the pre-trained basic detection model.

[0034] During the fine-tuning phase, the few-shot object detection fine-tuning dataset used in this method is a new class of few-shot fine-tuning subset. All training images within the new class of few-shot fine-tuning subset are read and input into a pre-trained feature extraction network. The network extracts candidate regions from the images and outputs the original RoI visual features corresponding to all candidate boxes. All original RoI features are simultaneously fed into the parallel TFE and CGCL modules. The TFE module outputs enhanced RoI features, which are only input into the classification branch, while the original RoI features are directly input into the regression branch, avoiding semantic modulation interference with bounding box localization accuracy.

[0035] In few-shot detection scenarios, due to the scarcity of single-class labeled samples, the model struggles to learn a compact distribution of similar features, and visually similar different classes are prone to aliasing in the feature space. Furthermore, the class representation vectors obtained solely from a small batch of samples fluctuate wildly and cannot be stably used as a classification benchmark. To address these two shortcomings, this invention designs the CGCL module, the structure of which is as follows: Figure 2 As shown, the overall module design consists of two parts: First, the CLIP text encoder extracts all category text embeddings, constructs a semantic similarity matrix that can distinguish categories, and sets differentiated repulsion strengths for different negative samples based on the matrix values, applying stronger separation constraints to semantically similar dissimilar categories; Second, a dynamic first-in-first-out prototype storage queue is introduced to continuously accumulate multiple batches of ground truth bounding box (RoI) features to obtain the category mean representation, thereby improving the stability of the category prototype throughout the training process. The category prototype is defined as a representative feature vector of a single-class target, generated by aggregating the mean of the RoI features corresponding to all ground truth bounding boxes of that class.

[0036] Specifically, the CGCL module pre-constructs a fixed semantic prior, namely a semantic similarity matrix between categories. The specific execution steps are as follows: Set a uniform text prompt template "a photo of {category name}", and input each category name sequentially into the text encoder of the CLIP visual language model for encoding, obtaining the text embedding vector corresponding to each category, forming a complete text embedding set. Where C represents the total number of target categories in the dataset. Let be the text embedding vector for the i-th category.

[0037] Based on the vector cosine similarity formula, calculate the embeddings of any two types of text. and The similarity values ​​between them are used to generate an initial similarity matrix. The element at any position in its matrix The cosine similarity between the text embeddings of class i and class j is defined as: ; The original similarity values ​​obtained using uniform template encoding have a narrow distribution range and low class discrimination. Directly using them as weights would reduce the effectiveness of contrastive learning constraints. Therefore, the initial similarity matrix... A nonlinear power-law sharpening transformation is performed to obtain the semantic similarity matrix between categories. Matrix elements satisfy: , in, For the sharpening index, this embodiment takes... . When the two categories represent different targets, the original similarity is amplified exponentially. When the time represents the same category, the weight is fixed at 1.

[0038] The power-sharpened matrix W remains constant throughout the entire training cycle of the model and is used as a fixed semantic prior in the contrastive loss calculation of the CGCL module. This transformation can widen the weight gap between high-similarity and low-similarity categories, preserving the semantic association between similar categories while eliminating weight noise from irrelevant categories.

[0039] As an optional technical solution, the dynamic prototype memory queues for various categories are updated using the original RoI features that meet preset conditions, including: Obtain candidate bounding boxes corresponding to each original RoI feature. Category tags The intersection-union ratio between the actual bounding boxes and the corresponding true bounding boxes ; The category label The category is not the background category, and the cross-union ratio (CUNR) is greater than the preset CUNR threshold. The original RoI features were determined to be qualified features and included in the valid feature set. Its definition is: , in , This indicates the background category. This filtering mechanism excludes both background candidate boxes and samples with low localization quality, ensuring that contrastive learning is built upon reliable positive and negative sample pairs.

[0040] Configure a separate first-in-first-out (FIFO) memory queue for each target category in the dataset. Each category corresponds to a memory queue, and the capacity of each memory queue is set according to the number of labeled samples of that category in the training set. The formula for calculating the queue capacity is as follows: , Adjust the number of labeled samples in the subset for the corresponding category. In this embodiment, the queue capacity coefficient is used. ; The qualified original RoI features are compressed and mapped into compact feature vectors of a preset dimension (set to 128 in this embodiment). The feature vector is then sent to the FIFO queue of its category. If the queue reaches its preset capacity limit, the feature at the front of the queue is removed first, and then a new, compact feature vector is stored. ; Compact feature vectors stored in a single-class memory queue Calculate the arithmetic mean and generate a real-time category prototype for that category. The calculation formula is: , This is the memory queue corresponding to category c.

[0041] The aforementioned dynamic update mechanism enables the prototype to gradually stabilize as the training process progresses, thereby providing a more reliable reference point for comparative learning.

[0042] In contrastive learning, not all candidate boxes are suitable for participation in constraints. The CGCL contrastive loss is constructed only based on the filtered effective RoI features, real-time prototypes of each category, and semantic similarity matrices. The specific steps are as follows: Traverse the set of valid candidate boxes valid candidate boxes within The compressed feature vector corresponding to this candidate box is The memory bank stores all combinations of class prototypes. Where M represents the total number of category prototypes in the memory bank. Let j be the feature vector of the j-th prototype. This is the category label corresponding to the j-th prototype.

[0043] Based on the current valid candidate boxes Matching real category labels Divide the prototype set: group prototypes whose labels match the candidate boxes into the positive prototype index set. Prototypes with different labels are all grouped into the negative prototype index set. ,Right now: , ; Set temperature coefficient Controlling the similarity distribution, calculate the scaling similarity of the positive and negative prototypes separately: Positive prototype similarity Negative prototype combined with category semantic weights Weighted negative similarity is obtained. ; Based on the positive prototype similarity and the weighted negative similarity, a semantically weighted contrastive loss function for the CGCL module is constructed: , in, The number of samples in the effective RoI feature set. This represents the natural exponent operation. Semantic weights are used during loss calculation. Differentiated constraint weights are applied to different negative prototypes: if the category to which the negative prototype belongs has a high semantic similarity to the current sample category, the weight value is larger, and the corresponding exclusion constraint is stronger; if the semantic difference between the two is large, the weight value is smaller, weakening the meaningless exclusion constraint.

[0044] In low-sample detection scenarios, the number of labeled samples is scarce, making it difficult to generate sufficiently discriminative RoI features by training models solely based on visual information. Furthermore, new categories may suffer from significant sample bias due to insufficient sample size. To address this deficiency, this invention designs a TFE module that leverages the category semantic prior output by the CLIP text encoder to perform channel-dimensional semantic enhancement processing only on the RoI features of the classification branch.

[0045] It should be noted that this module does not perform any semantic modulation on the regression branch: the bounding box regression task relies solely on category-independent spatial localization features. Introducing semantic information would disrupt the spatial offset patterns learned by the model, reducing the accuracy of the bounding box localization. A schematic diagram of the TFE module is shown below. Figure 3 As shown.

[0046] Furthermore, the TFE module performs semantic modulation on the original RoI features based on the semantic information of each category of text generated by the CLIP visual language model, to obtain enhanced RoI features, including: Let the original visual feature matrix corresponding to N RoI features in the batch be... , The original RoI visual feature dimension; the CLIP text embedding set for all categories is denoted as . Single category text embedding , For text embedding dimensions; set a learnable alignment projection matrix. The original RoI visual features are mapped to the CLIP text embedding space through linear projection, and L2 normalization is performed to obtain the projected feature matrix. The calculation formula is as follows: , Among them, the projection feature matrix elements in Let be the projection feature corresponding to the i-th RoI.

[0047] Calculate the projection features Embedded with all categories of text The similarity is calculated using the following formula: Where T is a learnable temperature parameter; the top k category text embeddings with the highest similarity to the current RoI are selected, and after performing Softmax normalization on the selected similarity values, the corresponding text embeddings are weighted and summed with normalized weights to obtain the aggregated text features of the current RoI. : ; Aggregate text features Input the first multilayer perceptron and the second multilayer perceptron respectively to generate channel-level scaling factors. and channel-level bias terms ; Set a learnable channel-gated vector whose dimensions match the original RoI features. This is used to control the injection intensity of semantic information into each channel. The learnable channel gating vector is utilized. Combined with the channel-level scaling factor and channel-level bias terms Original RoI features Channel-by-channel affine modulation is performed to obtain RoI features after text semantic enhancement. : , in, This represents the element-wise multiplication operation of vectors.

[0048] After fine-tuning with a small number of new class samples, few-shot detectors generally suffer from inaccurate confidence calibration. This includes the model often assigning unreasonably high classification scores to background regions, blurred edges, or low-quality candidate boxes, resulting in a large number of high-confidence false detections during the inference stage. Existing calibration methods (such as PCB) mainly rely on the similarity between visual features to adjust scores, but due to the lack of explicit semantic discrimination criteria, their suppression effect in complex scenes remains limited.

[0049] To address this issue, a cross-modal anchoring semantic calibration module (CASC) was designed, the core idea of ​​which is as follows: Figure 4 As shown, the CASC module is designed with two main goals: first, to recalibrate the detection scores using cross-modal semantic information provided by CLIP; and second, to introduce an additional set of negative semantic anchors as a "semantic absorption pool" for background regions and low-quality candidate boxes, thereby enabling the calibration process to have explicit semantic discriminative capabilities. It is worth noting that this module is designed to be training-independent and can be used directly and plug-and-play during the inference phase without any additional training costs.

[0050] Furthermore, the CASC module constructs an extended semantic set containing text embeddings for each category and predefined negative semantic anchor text embeddings. The visual features of the candidate detection boxes are semantically matched with this extended semantic set to obtain calibrated category confidence scores, including: The CLIP image encoder is used to extract the visual features corresponding to a single candidate detection box. Among them, feature dimension ; Multiple sets of text prompts were manually written to represent typical false detection scenarios. The text content included three types of descriptions: blurry targetless images, random texture noise, and pure background regions. All text prompts were input into the CLIP text encoder for encoding to generate M sets of negative semantic anchor text embeddings. The text embeddings of all C target categories in the dataset are concatenated with the text embeddings of M negative semantic anchors along the feature dimension to form an extended semantic set. ; Calculate the visual features of candidate detection boxes With extended semantic set The dot product similarity of each embedding vector within the vector; Introducing calibration temperature parameters The dot product similarity is scaled and then normalized using the Softmax function to output the semantic category probability distribution; this semantic probability distribution is then directly used as the calibrated category confidence.

[0051] As an optional technical solution, the expression for the semantically calibrated category probability distribution is as follows: , in, represents the visual feature vector of the candidate detection box. To expand the semantic set The text embedding vector of the y-th category in the text. To expand the semantic set The text embedding vector of the l-th category, The total number of categories, The number of negative semantic anchor text embeddings. For temperature parameters; This is the semantic calibration probability that the current candidate detection box is identified as a target of class y, i.e., the calibrated class confidence.

[0052] As an optional technical solution, the original classification score of the candidate detection box is corrected based on the calibrated category confidence, including: Pre-set the confidence uncertainty interval and read the raw classification scores of the candidate detection boxes directly output by the detector. Determine whether the original classification score falls within the confidence uncertainty interval: If the original classification score falls within the confidence interval, then the calibrated class confidence score is compared with the original classification score according to a preset fusion coefficient. Weighted fusion is performed to obtain the final calibration score. : , In the formula, α is a fixed fusion coefficient, and in this embodiment, α = 0.7. This strategy retains the discrimination ability of the original detector trained based on rich visual features, and introduces the cross-modal semantic correction function of CLIP. The two complement each other and jointly improve the reliability and robustness of the final detection results.

[0053] If the original classification score does not fall within the confidence interval, it is considered a high-confidence reliable prediction, and the original classification score is retained. constant.

[0054] To verify the comprehensive performance of the proposed solution on standard few-shot target detection benchmark tasks, a comparative experiment was conducted using the PascalVOC public dataset. The experiment strictly followed the industry-standard base class and new class partitioning specifications, setting 15 base classes and 5 new classes, and configuring three non-overlapping new class partitioning subsets for multiple group comparisons. nAP50 (average accuracy under the condition of an intersection-union threshold of 0.5) was selected as the quantitative evaluation index, and the proposed solution was compared with the existing mainstream few-shot detection algorithms in terms of performance.

[0055] Experimental results show that this invention demonstrates superior performance across all three new class subsets and sample annotation configurations. Taking Novel Set1 as an example, the nAP50 scores of this scheme reach 64.1% and 72.4% under 1-shot and 10-shot annotation configurations, respectively, representing improvements of 18.3 and 4.4 percentage points compared to the DCFS baseline algorithm. Under 2-shot and 3-shot annotation scenarios, the nAP50 scores are 65.3% and 69.5%, respectively, also significantly outperforming comparable algorithms. On Novel Set2 and Novel Set3 subsets, this scheme maintains stable performance gains. In Novel Set3, the 10-shot configuration achieves a detection accuracy of 66.1%, demonstrating good generalization stability across different class distribution scenarios. In few-shot target detection, K-shot (where "shot" represents an instance, i.e., K-instance) is defined as the number of labeled support sample images provided for each new class, denoted as the hyperparameter K. In this embodiment of the invention: when K=1, only a single labeled sample is provided for each new category, and the model must rely on cross-category prior knowledge for instantaneous category mapping, which is an extreme scenario with extremely scarce data; when K=10, ten labeled samples are provided for each new category, and the model can use limited samples for rapid fine-tuning or distribution calibration, significantly improving detection accuracy. Other settings such as K=2, 3, 5, etc., follow the same principle. The larger the K value, the more sufficient the intra-class variance information the model can obtain, and the more stable the detection performance tends to be. However, the above conditions all fall within the small sample range, which is fundamentally different from the supervised settings of thousands of samples per class in conventional object detection.

[0056] Under the extremely sparse 1-shot annotation condition, the proposed solution achieves an nAP50 of 64.1%, surpassing the performance of state-of-the-art algorithms RSF and PoE-FSOD by 4.0 and 5.4 percentage points, respectively. This verifies that the CGCL semantic-guided contrastive learning and TFE text feature enhancement modules can effectively improve class feature discrimination in scenarios with extremely scarce labeled samples. As the number of labeled samples per class increases to 5-shot and 10-shot, the model's detection accuracy steadily increases while performance gains remain stable. This experimental phenomenon confirms that the dynamic prototype memory queue can build highly stable class representations based on accumulated samples from multiple batches, allowing semantically weighted contrastive constraints to fully leverage the feature space partitioning function. In summary, the quantitative experimental data from each group fully demonstrate that the present invention exhibits excellent performance under the Pascal Voc standard for few-shot detection benchmarks. The relevant experimental values ​​are as follows: Figure 5 As shown in the figure, N1, N2, and N3 represent three different data partitions.

[0057] To further verify the generalization performance of the algorithm in multi-class, large-scale, and complex detection scenarios, an extended control experiment was conducted using the MSCOCO dataset. This dataset contains 80 target samples and features diverse sample distributions, significant differences in appearance among similar targets, and a large range of target scales, posing higher testing requirements for the feature representation and discrimination capabilities of the few-shot detection model. This experiment used the standard nAP as a quantitative evaluation metric, which calculates the mean accuracy by integrating multiple sets of intersection-union thresholds. Model performance was evaluated under six few-shot annotation scenarios: 1-shot, 2-shot, 3-shot, 5-shot, 10-shot, and 30-shot, and compared horizontally with several existing mainstream algorithms.

[0058] Quantitative experimental results demonstrate that the proposed solution exhibits superior detection performance and competitive advantages on the MS COCO benchmark dataset: Under 10-shot annotation conditions, the model's nAP reaches 18.1%, significantly higher than the 17.7% and 17.2% of the leading algorithms PoE-FSOD and GCSTG, respectively; under 30-shot conditions, the nAP reaches 25.1%, a 1.4 percentage point improvement over the best-performing comparison algorithm PoE-FSOD and a 3.2 percentage point improvement over the DCFS baseline algorithm; in the 1-shot extremely sparse annotation scenario, the proposed solution achieves an nAP of 9.6%, significantly outperforming traditional few-shot detection baseline algorithms such as DeFRCN and DCFS; under 2-shot, 3-shot, and 5-shot configurations, the model's nAP remains consistently at 15.1%, 16.3%, and 18.1%, respectively, with all metrics ranking among the top of comparable algorithms.

[0059] Compared to the Pascal Voc dataset with only 20 target classes, the MS COCO dataset increases the total number of classes to 80, significantly increasing the risk of semantic overlap and feature confusion between classes. However, this invention maintains stable performance gains even in this high-confusion scenario, demonstrating that the cross-modal semantic-guided contrastive learning mechanism and text-driven feature enhancement module can continuously strengthen the class separability of the feature space in multi-class, complex semantic association environments. Simultaneously, the confidence calibration strategy based on negative semantic anchors can continuously suppress false detections in complex background images. Multiple sets of experimental data comprehensively confirm that the algorithm of this invention possesses excellent scene versatility and detection robustness. The corresponding experimental data are as follows: Figure 6 As shown.

[0060] According to another aspect of the present invention, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the few-sample target detection method that integrates semantic-guided contrastive learning and cross-modal calibration as described above.

[0061] According to another aspect of the present invention, a non-transitory computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the few-sample target detection method that integrates semantically guided contrastive learning and cross-modal calibration as described above.

[0062] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0063] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0064] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0065] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0066] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A few-shot target detection method integrating semantic-guided contrastive learning and cross-modal calibration, characterized in that, include: Obtain a few-sample object detection training dataset, which includes sample images and their corresponding class labels and ground truth bounding boxes; A few-shot target detection framework based on a visual language model is constructed. The detection framework includes a feature extraction network, a text-guided feature enhancement module, a semantic-guided contrastive learning module, and a cross-modal anchoring semantic calibration module. During the training phase, the feature extraction network extracts training images and their corresponding category labels and ground truth bounding boxes from the training dataset, and inputs the training images into the feature extraction network to obtain the original RoI features of multiple regions of interest; For each original RoI feature, the first and second processes are executed in parallel, and the first and second processes share the original RoI features. The first process is as follows: using the text-guided feature enhancement module, based on the semantic information of each category of text generated by the visual language model, the original RoI features are semantically modulated to obtain enhanced RoI features, and the enhanced RoI features are input into the classification branch for category prediction, and the original RoI features without semantic modulation are input into the regression branch for bounding box prediction to obtain classification prediction results and regression prediction results. The second process is as follows: using the semantic-guided contrastive learning module, constructing a semantic similarity matrix between categories based on the text embeddings of each category provided by the visual language model, applying weighted constraints to the negative samples in the contrastive learning based on the semantic similarity matrix, applying contrastive learning loss to the original RoI features, and updating the dynamic prototype memory queue of each category using the original RoI features that meet the preset conditions. The detection framework is updated based on the differences between the classification prediction results and the category labels, the differences between the regression prediction results and the ground truth bounding boxes, and the contrastive learning loss. During the inference phase, candidate detection boxes are obtained based on the classification prediction results and the regression prediction results, and the visual features of each candidate detection box are extracted using the image encoder of the visual language model. Using the cross-modal anchoring semantic calibration module, an extended semantic set is constructed, which includes text embeddings of each category and predefined negative semantic anchor text embeddings. The visual features of the candidate detection box are semantically matched with the extended semantic set to obtain the calibrated category confidence. The original classification scores of the candidate detection boxes are corrected based on the calibrated category confidence, and the final detection results are output. The enhanced RoI features obtained include: The original RoI features are mapped to the text embedding space of the visual language model through a linear projection layer and then subjected to L2 normalization to obtain the projected features. Calculate the similarity between the projected features and the text embeddings of each category, select the top k categories of text embeddings with the highest similarity, and perform weighted aggregation on the selected text embeddings using the Softmax normalization function with temperature parameter to obtain aggregated text features; The aggregated text features are input into the first multilayer perceptron and the second multilayer perceptron respectively to generate channel-level scaling factors and channel-level bias terms; The channel-level scaling factor and channel-level bias term are combined with the learnable channel-level gating vector to perform affine modulation on the original RoI features, resulting in enhanced RoI features. Applying a contrastive learning loss to the original RoI features includes: A semantically weighted contrastive loss function is constructed. The similarity between the current original RoI feature and its corresponding positive prototype is calculated using a first weight, and the similarity between the current original RoI feature and each negative prototype is calculated using the corresponding weights in the inter-category semantic similarity matrix. The semantically weighted contrastive loss function is expressed as follows: ， in, The original RoI feature set that meets the preset quality conditions, The number of samples in the original RoI feature set. For the set of positive prototype indices of category i, For the set of negative prototype indices of category i, For positive sample similarity, This represents the semantically weighted similarity of negative samples.

2. The few-shot target detection method integrating semantic-guided contrastive learning and cross-modal calibration according to claim 1, characterized in that: Construct a semantic similarity matrix between categories, including: The category names are input into the text encoder of the visual language model using a unified prompt template to obtain the text embedding vectors corresponding to each category; Calculate the cosine similarity between each text embedding vector to obtain the initial similarity matrix; A nonlinear sharpening operation is performed on the initial similarity matrix to obtain the semantic similarity matrix between categories, whose matrix elements are... satisfy: ， in, This represents the initial similarity matrix between the text embedding vectors of class i and class j. This refers to the sharpening index.

3. The few-shot target detection method integrating semantic-guided contrastive learning and cross-modal calibration according to claim 1, characterized in that: Update the dynamic prototype memory queues for each category using the original RoI features that meet preset conditions, including: Obtain the category label and intersection-union ratio (IU) of the candidate bounding boxes corresponding to each original RoI feature, wherein the IU is the intersection-union ratio between the candidate bounding box and its corresponding ground truth bounding box; If the category label is a non-background category and the cross-union ratio is greater than the preset cross-union ratio threshold, then the original RoI feature is added to the dynamic prototype memory queue of its category. The dynamic prototype memory queue adopts a first-in-first-out (FIFO) update strategy. Each category corresponds to one memory queue, and the capacity of each memory queue is set according to the number of labeled samples of that category in the training set. When the number of features in the queue reaches the preset capacity limit, the feature at the head of the queue is removed first, and then the current feature is stored. The current prototype for each category is obtained by averaging all the original RoI features stored in the dynamic prototype memory queue for each category.

4. The few-shot target detection method integrating semantically guided contrastive learning and cross-modal calibration according to claim 1, characterized in that: The calibrated category confidence scores are obtained, including: The predefined negative semantic anchor text embedding is obtained by encoding a set of text prompts describing non-target regions, wherein the text prompts include text descriptions of blurred images, random textures, and background regions; The text embeddings of each category and the text embeddings of each negative semantic anchor are concatenated along the feature dimension to form an extended semantic set; Calculate the dot product similarity between the visual features of the candidate detection box and each embedding vector in the extended semantic set; The dot product similarity is converted into a semantically calibrated category probability distribution using a Softmax function with a temperature parameter. The semantically calibrated category probability distribution is used as the calibrated category confidence.

5. The few-shot target detection method integrating semantically guided contrastive learning and cross-modal calibration according to claim 4, characterized in that: The expression for the semantically calibrated category probability distribution is as follows: ， in, represents the visual feature vector of the candidate detection box. To expand the semantic set The text embedding vector of the y-th category in the text. To expand the semantic set The Middle l Text embedding vectors of each category, The total number of categories, The number of negative semantic anchor text embeddings. For temperature parameters, This represents the probability of belonging to category y after calibration.

6. The few-shot target detection method integrating semantic-guided contrastive learning and cross-modal calibration according to claim 1, characterized in that: The original classification scores of candidate detection boxes are corrected based on the calibrated category confidence, including: Determine whether the original classification score of the candidate detection box falls within the preset confidence interval: If the score falls within the confidence range, the calibrated category confidence score and the original classification score are weighted and fused according to a preset fusion coefficient to obtain the final calibration score. If the score does not fall within the confidence interval, the original classification score is retained unchanged.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the few-shot target detection method that integrates semantic-guided contrastive learning and cross-modal calibration as described in any one of claims 1 to 6.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the few-sample target detection method that integrates semantic-guided contrastive learning and cross-modal calibration as described in any one of claims 1 to 6.

Images