A supervised contrast fine-tuning method and device for active few-shot learning

Abstract

The application provides a supervised contrast fine-tuning method and device for active few-shot learning, and relates to the technical field of text classification in natural language processing. Active few-shot learning (AFSL) is an effective paradigm to improve the performance of large language models under limited annotation budget. In view of the problem that the traditional fine-tuning target is low in efficiency in AFSL, the application proposes a supervised contrast fine-tuning framework specially for the text classification task of natural language processing. The framework combines supervised contrast learning and hard negative sample mining, optimizes the embedding space by using an enhanced hybrid loss function, and thus improves the utilization efficiency of labeled samples. A large number of experiments on five benchmark datasets show that, under the fixed SOTA query strategy, the text classification performance of the method of the application is superior to that of the baseline model. These findings prove that the improvement of the learning target to optimize "how to learn" provides a new direction complementary to the existing query strategy for promoting AFSL research.

Description

Technical Field

[0001] This invention relates to the field of text classification technology in natural language processing (NLP), and more particularly to a supervised contrastive fine-tuning method and apparatus for active few-shot learning. Background Technology

[0002] Pre-trained language models (PLMs) have greatly advanced natural language processing (NLP), but they still rely heavily on labeled data, posing significant challenges in low-resource scenarios. Few-Shot Learning (FSL) and its active learning variants address this issue by selecting the most informative samples for labeling under a strict labeling budget. This invention focuses on Active Few-Shot Learning (AFSL) in text classification tasks, where robust learning from a very small number of labeled samples is crucial.

[0003] AFSL improves model accuracy with a limited labeling budget by iteratively selecting the most informative unlabeled samples for labeling and updating the model. Previous research has mainly focused on optimizing the query strategy (i.e., the "what to learn" stage) (Long Z, Killick G, Mccreadie R, et al. When hard negative sampling meets supervised contrastive learning[J]. 2023.). However, the learning objective after obtaining samples—i.e., "how to learn"—has received little attention. Most AFSL processes default to using cross-entropy (CE) as the loss function. CE focuses on the classification accuracy of individual samples, but it is difficult to provide a good metric structure in incremental small-sample updates. This choice often leads to suboptimal embedding structures in small-sample updates, limiting the effectiveness of each labeled sample and forming a practical bottleneck. Summary of the Invention

[0004] This invention argues that the inefficient sample utilization problem caused by the suboptimality resulting from the learning objective has become a key bottleneck restricting the performance of existing active few-shot learning frameworks. To address this bottleneck, this invention shifts the research focus from "sample selection" to "sample utilization" and proposes a supervised contrastive fine-tuning method specifically designed for active few-shot learning. This invention believes that in the iterative framework of AFSL, the model needs to adapt to a dynamically growing support set in each round; therefore, an ideal learning objective must be able to stably optimize the entire embedding space driven by a very small number of newly added samples.

[0005] In a first aspect, the present invention provides a supervised contrastive fine-tuning method for active few-shot learning, comprising:

[0006] Step 1: Obtain the unlabeled text data pool;

[0007] Step 2: Select M text samples from the unlabeled text data pool using a preset query strategy, add the labeled text samples to the support set, and delete them from the unlabeled text data pool; M is an integer greater than 1.

[0008] Step 3: Input the updated support set into the pre-trained language model, calculate the mixed objective loss between the model output and the labeled true class labels, and optimize the parameters of the pre-trained language model based on the mixed objective loss; the mixed objective loss consists of cross-entropy loss and supervised contrastive loss based on hard negative sample mining;

[0009] Step 4: Repeat steps 2 and 3 until the preset stopping condition is met to obtain the optimal pre-trained language model for text classification.

[0010] Within the AFSL framework, the SCL-HNM proposed in this invention can impose explicit structural constraints on the learning objective of the embedding space, and can utilize each labeled sample more efficiently than the traditional cross-entropy loss.

[0011] Furthermore, the formula for calculating the supervised contrastive loss based on hard negative sample mining is as follows:

[0012]

[0013]

[0014] in, All in the current batch A set of indexes for each text sample; for any anchor point within a batch. Its sentence embedding vector go through Normalized representation ; It is in the batch and the anchor point A set of positive samples belonging to the same class; It is a set of hard negative examples; This is a temperature coefficient used to adjust the distribution of similarity scores. Represents the cosine similarity function. For a single anchor point The monitoring comparison loss, This refers to the loss in batch monitoring and comparison.

[0015] Furthermore, the formula for calculating the hybrid target loss is as follows:

[0016]

[0017] in, For mixed target loss, For cross-entropy loss, This is a hyperparameter.

[0018] Furthermore, the query strategy adopts either a state-of-the-art (SOTA) strategy or a random sampling strategy.

[0019] Furthermore, before inputting the updated support set into the pre-trained language model, the process includes: selecting a pre-trained language model, removing the non-linear projection head located after the encoder in the model, and selecting to directly optimize the core semantic representation output by the encoder.

[0020] Secondly, the present invention provides a supervised contrastive fine-tuning device for active few-shot learning, comprising:

[0021] The data acquisition module is used to acquire an unlabeled text data pool;

[0022] The text query module is used to select M text samples from the unlabeled text data pool for annotation using a preset query strategy, add the annotated text samples to the support set, and delete them from the unlabeled text data pool; M is an integer greater than 1;

[0023] The model fine-tuning module is used to input the updated support set into the pre-trained language model, calculate the mixed objective loss between the model output and the labeled true class labels, and iteratively optimize the parameters of the pre-trained language model based on the mixed objective loss until the optimal pre-trained language model for text classification is obtained; the mixed objective loss consists of cross-entropy loss and supervised contrastive loss based on hard negative sample mining.

[0024] Thirdly, the present invention provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the method as described in the first aspect.

[0025] Fourthly, the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method described in the first aspect.

[0026] The beneficial effects of this invention are as follows:

[0027] (1) This invention believes that current AFSL research focuses too much on “sample selection” and neglects “sample utilization”. The low sample utilization efficiency caused by suboptimal learning objectives (such as cross-entropy loss) is the key bottleneck restricting its performance. To solve this bottleneck, this invention adapts SCL-HNM to the AFSL framework for the first time. It systematically adapts and introduces supervised contrastive learning (SCL-HNM) integrating hard negative sample mining into the AFSL iterative fine-tuning framework in the NLP field, aiming to maximize the utilization efficiency of each valuable labeled sample.

[0028] (2) Extensive experimental results on five benchmark datasets show that, under a fixed state-of-the-art (SOTA) query strategy, the SCL-HNM method proposed in this invention stably and significantly outperforms the baseline model, especially in the "cold start" stage where labeled data is extremely sparse. This verifies the effectiveness and generalization ability of the method in this invention. Attached Figure Description

[0029] Figure 1 A flowchart illustrating a supervised contrastive fine-tuning method for active few-shot learning provided in an embodiment of the present invention;

[0030] Figure 2 The following is a visualization of t-SNE of sentence embeddings generated by different methods on the Amazon Reviews test set (K=100) provided for embodiments of the present invention: (a) The embedding space generated by the SOTA baseline (CE Loss) is entangled; (b) The embedding space generated by the SCL-HNM method of the present invention exhibits excellent intra-class compactness and inter-class separation.

[0031] Figure 3 This is a schematic diagram of the structure of a supervised contrastive fine-tuning device for active few-shot learning provided in an embodiment of the present invention;

[0032] Figure 4 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of the embodiments of this invention will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0034] SCL: Supervised Contrastive Learning;

[0035] HNM: Hard Negative Mining;

[0036] The core idea of ​​this invention is to no longer treat samples as isolated individuals, but to maximize the value of each labeled sample by explicitly optimizing the geometry of the embedding space. To this end, this invention is the first to adapt and introduce a supervised contrastive learning objective (SCL-HNM) that integrates hard negative sample mining into the AFSL framework. The method of this invention contains two key components: (1) Supervised contrastive learning: Based on supervised contrastive learning, this invention forces the model to "bring closer" to samples of the same class (intra-class compactness) and "push away" samples of different classes (inter-class separation) in the embedding space through a loss function, thereby shaping a highly structured and more discriminative representation space. (2) Hard negative sample mining: This invention recognizes that not all negative examples are equally important. Those "hard negative examples" that are easily confused with anchors in the feature space are crucial for learning accurate decision boundaries. Therefore, this invention draws on and integrates the HNM mechanism to focus the model's learning resources on distinguishing these most challenging sample pairs, thereby achieving "gradient focusing" and maximizing learning efficiency when the data is extremely sparse.

[0037] SCL (Strictly Comparative Learning) can make similar samples more compact in the representation space while separating samples from different classes, thus making the representation geometry more compatible with the incremental learning mechanism of AFSL (Automatic Feedback Learning). Currently, adapting SCL and HNM to the iterative settings of AFSL still faces the following challenges: (i) each iteration provides only a small number of new labeled samples, requiring the objective function to balance stability and adaptability; (ii) pre-trained language models (PLMs) bring strong prior knowledge but also lead to anisotropy in the embedding space; (iii) language semantics introduce high intra-class variance, making the definition of hard negative examples more nuanced. To address these challenges, this invention integrates supervised contrastive learning combined with hard negative sample mining (SCL-HNM) into AFSL, aiming to bridge the gap between representation structure and active sample selection, and to provide a unified perspective on "how to learn" under limited supervision.

[0038] To address the bottleneck of low sample utilization efficiency in active few-shot learning, and unlike previous studies that focused on sample selection strategies, the core objective of this invention's novel supervised contrastive fine-tuning method is to optimize the learning process itself, i.e., answering the question of "how to learn more efficiently." It aims to explicitly optimize the geometry of the embedding space by designing an advanced learning objective, thereby enhancing its discriminative power.

[0039] like Figure 1 As shown, the supervised contrastive fine-tuning method for active few-shot learning provided in this embodiment of the invention follows the standard active learning protocol and mainly includes the following steps:

[0040] S101: Obtain the unlabeled text data pool;

[0041] S102: Select M text samples from the unlabeled text data pool using a preset query strategy, add the labeled text samples to the support set, and delete them from the unlabeled text data pool; M is an integer greater than 1;

[0042] Specifically, the query strategy can be a state-of-the-art (SOTA) strategy or a random sampling strategy.

[0043] S103: Input the updated support set into the pre-trained language model, calculate the mixed objective loss between the model output and the labeled true class label, and optimize the parameters of the pre-trained language model based on the mixed objective loss; the mixed objective loss consists of cross-entropy loss and supervised contrastive loss based on hard negative sample mining;

[0044] Specifically, the pre-trained language model google / flan-t5-base was selected as the base model. The reason for choosing this model is that, through large-scale instruction fine-tuning, it has demonstrated strong few-shot learning capabilities on various natural language understanding tasks, providing a highly competitive starting point for this research. To obtain high-quality sentence-level semantic representations, following standard practice, its encoder part is utilized. For any input text sequence... The hidden states of all words in the last layer of the encoder are extracted, and a fixed-dimensional sentence embedding vector is generated through average pooling. Considering that average pooling can aggregate the semantic information of all words in a sequence, thus providing a smoother and more global representation of the sentence, this has been widely proven to be robust and effective in text classification and semantic matching tasks. This vector Used as input sentence It is a comprehensive semantic encoding and serves as the basic unit for all subsequent learning objectives.

[0045] S104: Repeat steps S102 and S103 until the preset stopping condition is met to obtain the optimal pre-trained language model for text classification.

[0046] In one embodiment, an iterative evaluation protocol is employed to accurately measure the effectiveness of the learning objective on labeled data of different scales. Its formal definition is as follows: Given a large-scale unlabeled data pool... A pre-trained model and a query strategy The research objective is to... Round iteration, each round from Search Each sample was added to the support set. ,when Once the preset scale is reached, the model is fine-tuned from scratch to evaluate its optimal performance in downstream tasks. The specific process is shown in Algorithm 1.

[0047]

[0048] In the aforementioned protocol, the state-of-the-art (SOTA) strategy Rep(En)-ClUn(En) is employed as the sample selector. This strategy has been proven to be efficient in combining embedding space representativeness with model uncertainty. Therefore, the core contribution of this invention is not to propose a new query strategy, but rather to focus on the "model update" stage in the algorithm flow, aiming to demonstrate that a better learning objective can bring significant performance improvements without changing the sample selection strategy.

[0049] To enhance the robustness and efficiency of the method in small-sample scenarios, the following key design choices were made in the aforementioned protocol: A rigorous weight resetting strategy was adopted to ensure strict comparability of performance evaluations across different support set sizes (K=10, 20, ..., 100). In each round of active learning iteration... Before starting, model weights All have been restored to their original pre-training checkpoints. Subsequently, only on the currently accumulated support set Fine-tuning is then performed. This protocol ensures that the improvement in model performance is entirely attributable to changes in the quantity and quality of the support set samples, rather than the cumulative effect of continuous training, thus enabling a pure evaluation of the effectiveness of the learning method of this invention.

[0050] In one embodiment, the present invention further restructures the original pre-trained language model, using the adjusted pre-trained language model as the base model. In existing contrastive learning work, a nonlinear projection head is typically added after the encoder output representation to generate features for calculating the contrastive loss. The core advantage of this design is that it decouples the information-rich representation space required by the downstream task from the contrastive learning task itself, preventing the representation space from losing generality to accommodate the contrastive loss. However, in the active few-shot fine-tuning scenario focused on by this invention, this invention intentionally omits this module and chooses to directly optimize the core semantic representation output by the encoder, including directly generating features for calculating the supervised contrastive loss based on the core semantic representation output by the encoder. The motivation for this design is: 1) Parameter efficiency: In parameter-sensitive few-shot scenarios, avoiding the introduction of additional trainable parameters is crucial to preventing model overfitting; 2) Direct optimization: The goal of this invention is to directly optimize the semantic representation space itself used by the downstream classifier, prioritizing model simplicity and optimization directness in low-resource environments. This design represents a trade-off between parameter efficiency and representation decoupling. This invention assumes that, in the case of extremely sparse AFSL data, avoiding overfitting takes precedence over the potential benefits of representation decoupling.

[0051] In one embodiment, to efficiently utilize samples and shape a structured embedding space, the design concept of the hybrid target loss in this invention is to combine the traditional classification task loss with a powerful representation space regularization term.

[0052] First, the standard cross-entropy loss is retained as the basic classification objective. Its definition is as follows:

[0053]

[0054] in, The model is for the input Predict its true label is The probability of this loss. This loss directly drives the model to learn and complete the final classification task.

[0055] Next, relying solely on This approach is suboptimal in small sample scenarios. Therefore, to explicitly shape a compact embedding space within classes and separate embedding spaces between classes, this invention proposes a supervised contrastive loss based on hard negative sample mining. The core of this loss function lies in... Based on the loss, only a selected set of hard negative examples is used for comparison calculation. To clearly define this loss, the following notation is first defined:

[0056] make All in the current batch The index set of each sample; for any anchor point within a batch. Its sentence embedding vector go through Normalized representation ; It is in the batch and the anchor point The set of positive samples belonging to the same class. It is a set of hard negative examples. First, define the set of negative examples. Then, from Selected from The highest cosine similarity Each sample constitutes . This is a temperature coefficient used to adjust the distribution of similarity scores. This represents the cosine similarity function.

[0057] Based on the above definition, a single anchor point Supervisory comparison loss Defined as:

[0058]

[0059] Total batch loss This is the average of all losses within the batch:

[0060]

[0061] A key difference between the loss function design of this invention and the standard SCL lies in the composition of the denominator. In the standard SCL, the denominator includes all negative examples within the batch. However, the loss function designed in this invention only includes the set of hard negative examples. The core motivation behind this design lies in gradient focus: 1) Learning efficiency: In contrastive learning, a large number of "simple" negative examples that are far from the anchor point in the feature space contribute almost zero to the loss function, resulting in invalid gradients. This not only wastes computational resources but, more importantly, dilutes the effective gradient signals from hard negative examples. In the AFSL scenario where labeled samples are extremely valuable, every model update is crucial. Therefore, we believe that focusing limited model capacity and computational budget on optimizing the most challenging decision boundaries is a more efficient learning strategy. 2) Computational efficiency: This design significantly reduces the computational terms in the denominator of the loss function, improving the computational efficiency of single-step training while maintaining the core contrastive idea.

[0062] However, this focused strategy also brings a theoretical trade-off. Focusing solely on the hardest negatives may have two potential risks: first, it may overlook information provided by "semi-hard negatives" that is equally useful for outlining the overall category profile; second, in some cases, it may lead to training instability due to overemphasis on a few hard samples. Despite these risks, we believe that the benefits of this strategy far outweigh the costs in the "cold start" phase of AFSL. The reason is that when labeled data is extremely scarce, the primary task is to quickly establish a "usable" rather than a "perfect" decision boundary using the strongest signal. Hard negatives provide precisely this strongest signal. At the same time, a robust pre-trained model itself provides a very solid initialization point, which largely mitigates the learning instability that may occur when training from scratch.

[0063] Finally, the two loss functions are weighted and combined to form the complete training objective of this invention. :

[0064]

[0065] Within this framework, and Their respective functions are as follows: Ensure the model learns the predictive ability to perform the final classification task; and This serves as a powerful structured regularization term, imposing explicit constraints on the embedding space and guiding the model to learn more discriminative representations. Hyperparameters Used to balance the weights between these two objectives.

[0066] To verify the advantages of the present invention, the present invention also provides the following experimental data.

[0067] (a) Dataset

[0068] To comprehensively evaluate the effectiveness and generalization ability of the proposed method, five benchmark datasets were carefully selected to cover diverse text classification challenges. Detailed statistical information for each dataset is shown in Table 1, and specific task input and output examples are shown in Table 2.

[0069] Table 1. Detailed statistical information for each dataset

[0070]

[0071] Table 1 The columns show the uniformity of the label distribution for each dataset. ,in It is a collection of tags. It is a tag The relative frequency. It indicates a completely uniform distribution.

[0072] Table 2 Examples of task inputs and outputs for each dataset

[0073]

[0074] MPQA Opinion Corpus is a fine-grained opinion mining corpus. For comprehensive evaluation, it was constructed as three independent classification tasks: opinion type (multi-label classification), sentiment polarity (binary classification), and sentiment intensity (pentagram classification).

[0075] AG News is a standard news topic classification dataset. Since the official dataset does not provide a validation set, following standard practice, 1200 samples were randomly sampled from its training set to serve as a fixed validation set.

[0076] Amazon Reviews is a multilingual review corpus. This experiment uses the English portion of this corpus to predict ratings from one to five stars. To avoid overfitting in the small sample stage, a moderately sized validation set was constructed by randomly downsampling 1200 samples from the original validation set.

[0077] The selection of datasets for this experiment was mainly based on the following considerations: (1) Diversity of task types: covering binary classification, multi-class classification, and multi-label classification. (2) Differences in semantic discriminability: including AG News with clear themes, as well as Amazon Reviews and MPQA Intensity with more fine-grained semantics. This verifies that the SCL-HNM proposed in this invention will be more advantageous in such fine-grained tasks. (3) Challenges in data distribution: the non-uniformity (U%) of the label distribution of each dataset is different, which allows us to evaluate the robustness of the method in the face of class imbalance problems.

[0078] (ii) Baseline Model

[0079] To comprehensively evaluate the effectiveness of the method proposed in this invention, the following two core baseline models were established for comparison:

[0080] Random + CE: This baseline employs a random sampling strategy to select samples and uses standard cross-entropy (CE) loss for model fine-tuning. It represents a naive approach that does not rely on intelligent sample selection or advanced learning objectives, forming the lower bound of our experimental performance.

[0081] State-of-the-art Query + CE: This baseline is our core comparison target. It employs the state-of-the-art query strategy Rep(En)-ClUn(En) for sample selection. This strategy efficiently selects samples by combining the representativeness of the embedding space (selecting samples that represent the data distribution) with model uncertainty (selecting the most perplexing samples). By comparing with this baseline, we can accurately isolate and evaluate the pure performance gain of the new learning objective proposed in this invention without changing the state-of-the-art sample selection strategy.

[0082] The complete method of this invention is referred to as SOTA Query + SCL-HNM, specifically as follows: based on the SOTA query strategy, a designed... The loss function is fine-tuned.

[0083] (III) Fine-tuning Experiment

[0084] All experiments followed a standard Active Few-Shot Learning (AFSL) iterative protocol to accurately evaluate the effectiveness of the learning objective under different support set sizes. To ensure fairness and independence in the evaluation, an evaluation process including weight resetting was designed. The specific steps of this protocol are as follows:

[0085] Initialization: The labeled support set at the start of the experiment. Empty; all training data are located in the unlabeled pool. middle.

[0086] Iterative Queries and Augmentation: The experiment involved multiple iterations. In each iteration, a given query strategy (such as random sampling or a state-of-the-art strategy) was used to query... Select These samples are considered labeled and added to the support set. In the middle, at the same time from Removed from the middle.

[0087] Model evaluation: When the support set size (Right now Reaching the preset key node, i.e. At that time, a complete model performance evaluation should be conducted.

[0088] Weight Resetting and Fine-tuning: To ensure the fairness and independence of the evaluation, the model's weights are always reset from the original pre-training checkpoints before each evaluation. Reload. Subsequently, the model will only run at its current size. Support set Fine-tuning is then performed. The fine-tuning process follows the hyperparameter settings given below and employs an early stopping strategy based on validation set performance.

[0089] Performance Report: After model fine-tuning, its performance will be evaluated on a fixed test set completely separate from the training process. We use the Micro-F1 score as the primary performance metric.

[0090] The core design of this protocol (especially the weight resetting) ensures that the performance gains measured at different K values ​​are entirely attributable to changes in the quantity and quality of the support set samples, rather than the cumulative effect of continuous training, thus achieving a pure evaluation of the effectiveness of the learning objective. To ensure the robustness of the results, all AFSL experiments were repeated on five different random seeds, and their mean and standard deviation are reported.

[0091] (iv) Experimental details

[0092] Hardware and software environment: All experiments were performed on a single NVIDIA RTX 4090 (24GB) GPU. The PyTorch deep learning framework was used, and the google / flan-t5-base model was loaded and fine-tuned using the Hugging Face Transformers library.

[0093] Hyperparameter settings: The key hyperparameters used for model fine-tuning are shown in Table 3. Among them, the core hyperparameters related to contrastive learning (including loss weights) Temperature coefficient and the number of hard negative examples The parameters were determined through a systematic sensitivity analysis on the validation set of the MPQA Polarity dataset. This analysis aimed to verify the robustness of the proposed method and select a set of stable and generalizable parameter configurations for all subsequent experiments. The AdamW optimizer was used, and an early stopping strategy (patience=20) based on the validation set performance was configured to prevent model overfitting.

[0094] Table 3 Key hyperparameters used for model fine-tuning

[0095]

[0096] To ensure the complete reproducibility of all experimental results, all relevant random seeds (including PyTorch, NumPy, and data partitioning) were fixed in the experimental code.

[0097] (V) Experimental Results and Analysis

[0098] (1) Main results

[0099] This section aims to comprehensively evaluate the effectiveness and generalization ability of the proposed SCL-HNM fine-tuning method under different data scales. A series of experiments were conducted on five benchmark datasets, and the method of this invention was rigorously compared with multiple baselines. Detailed experimental results are summarized in Table 4, and the results are reported as "mean ± standard deviation" of five different randomized seed experiments. The symbol "-" indicates that the configuration is not applicable.

[0100] Table 4. Comparison of Micro-F1 scores (%) of different methods on five benchmark datasets

[0101]

[0102] Note: The query strategy selects samples from the unlabeled pool. Therefore, this strategy is only applicable to active learning processes (i.e., scenarios where K < FULL) and not applicable to settings using the full training set.

[0103] Performance ceiling analysis under sufficient data scenarios. First, we evaluate the performance ceiling of SCL-HNM as a general learning objective using the full training data (the "FULL" column in Table 4). Experimental results show that on all five datasets, simply replacing the learning objective from cross-entropy (CE) with SCL-HNM results in a comprehensive performance improvement. For example, on the MPQA Aolarity and MPQA Intensity datasets, SCL-HNM brings absolute performance improvements of 1.6% (95.8% vs 94.2%) and as high as 1.3% (51.3% vs 50.0%), respectively. This demonstrates that even within the standard supervised learning framework, by imposing explicit structural constraints on the embedding space, SCL-HNM remains a superior fine-tuning strategy compared to cross-entropy. Next, we analyze the core results within the AFSL framework, mainly covering the following two aspects:

[0104] Independent contribution of the learning objective. To isolate the evaluation of the pure effect of the learning objective, the performance of SCL-HNM and CE loss was first compared under a random sampling strategy. The results show that SCL-HNM outperforms CE in most cases, especially when the sample size is large (e.g., K=100) (e.g., performance improved from 88.4% to 90.3% on MPQA Polarit).

[0105] Synergistic Effects with State-of-the-Art (SOTA) Query Strategies. The core of this research is to verify whether SCL-HNM can further improve performance based on top-tier query strategies. To this end, the SOTA Query + CE baseline was compared with the complete method of this invention, Ours (SOTA + SCL-HNM). The method of this invention demonstrated superior performance in the vast majority of experimental settings. Specifically, on all datasets except AG News, the performance of the method of this invention was consistently better than or equal to the SOTA baseline. A notable exception was in the cold start phase of AG News (K=10, 20), where the baseline model performed better. This may be related to the characteristics of this news classification task, indicating that the advantage of SCL-HNM is not absolute under all conditions. Overall, these results strongly demonstrate that optimizing "how to learn" (learning objective) and optimizing "what to learn" (query strategy) are two complementary and equally important dimensions.

[0106] The advantages are significant during the "cold start" phase. Notably, the method of this invention exhibits particularly strong performance gains during the "cold start" phase, when the sample size is most sparse (e.g., K=10, 20). This phenomenon has been validated on multiple datasets. For example, on the MPQA Polarity task (K=20), the method of this invention achieves an absolute performance improvement of 2.6% (84.1% vs. 81.5%); similarly, on the Amazon Reviews task (K=20), the performance improvement reaches 1.8% (53.8% vs. 52.0%). This characteristic is of significant value in real-world applications because it means that users can reach the usable performance threshold of the model faster with less initial annotation cost.

[0107] Strong task generalization. The effectiveness of SCL-HNM has been widely validated on different types of tasks, including binary classification (MPQA Polarity), multi-class classification (MPQA Intensity, AG News, Amazon Reviews), and multi-label classification (MPQA Type), demonstrating its strong generalization ability as a general embedding space regularization strategy.

[0108] In summary, the experimental results presented in this section systematically verify the core hypothesis: optimizing the learning objective is key to improving AFSL performance, and it forms an orthogonal and complementary dimension with optimizing the query strategy. By introducing SCL-HNM, this invention can significantly improve the sample utilization efficiency and final performance of the AFSL framework without changing the existing state-of-the-art query strategy.

[0109] (2) Ablation test

[0110] To accurately analyze the independent contributions of each core component in the proposed SCL-HNM method, this section presents ablation experiments to compare the performance of different components in the baseline model and the proposed method. Using a fixed state-of-the-art (SOTA) query strategy and a support set size of K=100, comparative experiments were conducted on two representative datasets: MPQA Intensity and MPQA Polarity. The experiments only changed the learning objective, aiming to isolate and quantify the pure effectiveness of each component. The experimental results are shown in Table 5.

[0111] Table 5 Ablation Experiment Results (Micro-F1, %)

[0112]

[0113] The experimental results shown in Table 5 clearly reveal the successful internal synergistic effect mechanism of the method of the present invention:

[0114] By comparing the results of model (1) and model (2), it can be observed that introducing only the standard supervised contrastive loss (SCL) on the basis of the SOTA Query + CE baseline brings significant and consistent performance gains. Specifically, the performance is improved by 0.7% on the MPQA-Intensity dataset and by 1.2% on the MPQA-Polarity dataset. This result shows that in the AFSL framework, using supervised contrastive learning as the learning objective is significantly better than the traditional cross-entropy loss. The fundamental reason is that imposing explicit structured constraints (intra-class aggregation, inter-class separation) on the embedding space is a much more efficient way to utilize samples than optimizing each sample independently, especially in scenarios with sparse data.

[0115] Building upon SCL, further comparison of models (2) and (3) reveals that integrating the Hard Negative Sample Mining (HNM) mechanism significantly improves model performance. For example, performance on MPQA-Intensity increases from 40.4% to 41.6%, and a 0.5% gain is achieved on MPQA-Polarity. This finding demonstrates that integrating the HNM mechanism can bring additional significant performance gains to SCL. This clearly shows that HNM is not a dispensable add-on, but a key catalyst for pushing model performance to new heights. It validates our core insight: in small-sample scenarios, forcing the model to focus its limited learning resources on distinguishing the most challenging "hard" sample pairs is the most efficient way to shape accurate decision boundaries.

[0116] In summary, this ablation experiment systematically reveals the internal working mechanism of the SCL-HNM method: SCL lays the foundation for superior performance compared to cross-entropy by constructing a structured representation space; while HNM, on top of this, achieves a further leap in performance by focusing on fine-grained optimization of difficult samples. The two complement each other, together constituting the core advantage of the method of this invention.

[0117] (3) Embedded spatial analysis

[0118] The preceding sections have verified the effectiveness of the method of this invention through performance indicators. To further reveal the source of its performance advantages at the mechanistic level, this section will conduct an in-depth analysis combining qualitative and quantitative methods from the perspective of the geometric structure of the embedded space.

[0119] (3.1) Visualization of spatial structure

[0120] Using t-SNE dimensionality reduction technology, we perform two-dimensional visualization of the sentence embeddings generated on the Amazon Reviews test set (K=100) by the SOTA baseline (CE Loss) and the SCL-HNM method of this invention. For example... Figure 2 As shown, the representation spaces shaped by the two methods exhibit significant differences:

[0121] SOTA baseline (CE Loss, such as...) Figure 2 As shown in (a) in the figure: the representation space learned by this method exhibits a typical "entangled state". There is a large area of ​​overlap between samples of different categories (distinguished by color), and the category boundaries are blurred. At the same time, the distribution of each category itself is also relatively loose, failing to form effective intra-class clusters, which poses a huge challenge to downstream classifiers.

[0122] Ours (SCL-HNM Loss, such as Figure 2 As shown in (b) of this paper: In contrast, the embedding space shaped by the method of this invention exhibits superior structure. Its most significant feature is the high degree of "intra-class compactness" and "inter-class separability". Samples of the same class are tightly clustered into dense clusters, while the margin between clusters of different classes is significantly increased. This clear, highly structured spatial distribution provides a nearly linearly separable ideal input for downstream classifiers, thus explaining the source of its superior performance.

[0123] (3.2) Embedded quality quantitative assessment

[0124] To provide mathematical evidence for the above visual observations, two standard metrics, alignment and uniformity, are introduced to quantitatively evaluate the quality of the embedding space.

[0125] Alignment: Calculates the expected value of the embedding vector distance between pairs of samples within the same class. The lower the value, the better the alignment of samples within the class, and the more compact the clusters within the class.

[0126] Uniformity: Measures how evenly all samples are embedded in the hypersphere. The lower the value, the more uniform the sample distribution, which better preserves information entropy and avoids representation degradation.

[0127] Table 6. Quantitative evaluation of embedding space quality on the Amazon Reviews test set (K=100)

[0128]

[0129] As shown in Table 6, the quantitative evaluation results are highly consistent with the visualization findings, providing strong data support for the working mechanism of SCL-HNM: the model of this invention significantly reduced the alignment score from 0.6009 to 0.4131, data-drivenly confirming the success of the "pulling positive examples closer" mechanism. Simultaneously, the uniformity score improved from -1.2812 to -1.5695, indicating that the mechanism of "pushing negative examples away" (especially hard negative examples) effectively avoids the collapse of the embedding space, making the feature distribution more uniform and preserving richer information.

[0130] In summary, combining qualitative visualization and quantitative indicators, we can conclude that the performance advantage of the SCL-HNM method lies in its successful transformation of a chaotic and entangled embedding space into a highly structured representation space that is compact within classes, separable between classes, and rich in information.

[0131] Based on the same inventive concept, such as Figure 3 As shown, this embodiment of the invention also provides a supervised contrastive fine-tuning device for active few-shot learning, including: a data acquisition module, a text query module, and a model fine-tuning module;

[0132] The data acquisition module is used to acquire an unlabeled text data pool; the text query module is used to select M text samples from the unlabeled text data pool for simulated annotation using a preset query strategy, add the annotated text samples to the support set, and delete them from the unlabeled text data pool; M is an integer greater than 1; the model fine-tuning module is used to input the updated support set into the pre-trained language model, calculate the mixed objective loss between the model output and the real labels, and iteratively optimize the parameters of the pre-trained language model based on the mixed objective loss until the optimal pre-trained language model for text classification is obtained; the mixed objective loss consists of cross-entropy loss and supervised contrastive loss based on hard negative sample mining.

[0133] It should be noted that the supervised contrastive fine-tuning device for active few-shot learning provided in this embodiment of the invention is for implementing the above method. Its specific functions can be referred to in the above method embodiments, and will not be repeated here.

[0134] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4 As shown, the electronic device may include: a processor 401, a communication interface 402, a memory 403, and a communication bus 404, wherein the processor 401, the communication interface 402, and the memory 403 communicate with each other through the communication bus 404. Processor 401 can call logic instructions in memory 403 to execute a supervised contrastive fine-tuning method for active few-shot learning. This method includes: Step 1: Obtaining an unlabeled text data pool; Step 2: Selecting M text samples from the unlabeled text data pool using a preset query strategy for simulated annotation, adding the annotated text samples to the support set, and simultaneously deleting them from the unlabeled text data pool; M is an integer greater than 1; Step 3: Inputting the updated support set into a pre-trained language model, calculating the mixed target loss between the model output and the real labels, and optimizing the parameters of the pre-trained language model based on the mixed target loss; the mixed target loss consists of cross-entropy loss and supervised contrastive loss based on hard negative sample mining; Step 4: Repeating steps 2 and 3 until a preset stopping condition is reached to obtain the optimal pre-trained language model for text classification.

[0135] Furthermore, when the logical instructions in the aforementioned memory 403 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0136] This invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions, and when the program instructions are executed by a computer, the computer can execute a supervised contrastive fine-tuning method for active few-shot learning provided in the above-described method embodiments.

[0137] This invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a supervised contrastive fine-tuning method for active few-shot learning provided in the above-described method embodiments.

[0138] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0139] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A supervised contrastive fine-tuning method for active few-shot learning, characterized in that, include: Step 1: Obtain the unlabeled text data pool; Step 2: Select M text samples from the unlabeled text data pool using a preset query strategy, add the labeled text samples to the support set, and delete them from the unlabeled text data pool; M is an integer greater than 1. Step 3: Input the updated support set into the pre-trained language model, calculate the mixed objective loss between the model output and the labeled true class labels, and optimize the parameters of the pre-trained language model based on the mixed objective loss; the mixed objective loss consists of cross-entropy loss and supervised contrastive loss based on hard negative sample mining; Step 4: Repeat steps 2 and 3 until the preset stopping condition is met to obtain the optimal pre-trained language model for text classification.

2. The supervised contrastive fine-tuning method for active few-shot learning according to claim 1, characterized in that, The formula for calculating the supervised contrastive loss based on hard negative sample mining is as follows: in, All in the current batch A set of indexes for each text sample; for any anchor point within a batch. Its sentence embedding vector go through Normalized representation ; It is in the batch and the anchor point A set of positive samples belonging to the same class; It is a set of hard negative examples; This is a temperature coefficient used to adjust the distribution of similarity scores. Represents the cosine similarity function. For a single anchor point The monitoring comparison loss, This refers to the loss from batch-by-batch monitoring and comparison.

3. The supervised contrastive fine-tuning method for active few-shot learning according to claim 2, characterized in that, The formula for calculating the hybrid target loss is as follows: in, For mixed target loss, For cross-entropy loss, This is a hyperparameter.

4. The adaptive dynamic receptive field infrared target detection method for multi-view scenarios of unmanned aerial vehicles (UAVs) according to any one of claims 1-3, characterized in that, The query strategy employs either a state-of-the-art (SOTA) strategy or a random sampling strategy.

5. The adaptive dynamic receptive field infrared target detection method for multi-view scenarios of unmanned aerial vehicles according to claim 4, characterized in that, Before inputting the updated support set into the pre-trained language model, the process also includes: selecting the pre-trained language model, removing the non-linear projection head located after the encoder in the model, and selecting to directly optimize the core semantic representation output by the encoder.

6. A supervised contrastive fine-tuning device for active few-shot learning, characterized in that, include: The data acquisition module is used to acquire an unlabeled text data pool; The text query module is used to select M text samples from the unlabeled text data pool for annotation using a preset query strategy, add the annotated text samples to the support set, and delete them from the unlabeled text data pool; M is an integer greater than 1; The model fine-tuning module is used to input the updated support set into the pre-trained language model, calculate the mixed objective loss between the model output and the labeled true class labels, and iteratively optimize the parameters of the pre-trained language model based on the mixed objective loss until the optimal pre-trained language model for text classification is obtained; the mixed objective loss consists of cross-entropy loss and supervised contrastive loss based on hard negative sample mining.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 5.

Images