Few-sample text classification method based on BERT model and KNN semi-supervised learning
By using the BERT model and the KNN semi-supervised learning framework, the semantic space is reconstructed and pseudo-labels are assigned, which solves the problem of performance limitation of deep learning models under few sample conditions and achieves efficient utilization of a small amount of labeled data and optimization of model performance.
Patent Information
- Application Number
- CN202510681573.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-26
- Publication Date
- 2025-11-07
AI Technical Summary
Existing deep learning models have limited performance under conditions of few samples, especially in specialized fields or low-resource languages, where annotation costs are high, sample distribution is uneven, and small sample problems are prominent, making it difficult to effectively utilize a small amount of labeled data.
We employ the BERT model and the KNN semi-supervised learning framework. By reconstructing the semantic space, we extract high-attention tokens using the BERT attention mechanism, assign pseudo-labels to unlabeled samples using the KNN algorithm, and optimize model performance through the KNN voting mechanism.
It improves the accuracy of pseudo-labels and the stability of the model, reduces the dependence on the quality of the initial labeled samples, and enhances the model's generalization ability and engineering applicability.
Smart Images
Figure CN120910259A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of text sentiment semantic opinion tendency classification, and particularly relates to a method for improving semi-supervised semantic classification learning efficiency. BACKGROUND
[0002] Text classification, as a core task in natural language processing, is widely used in sentiment analysis, news classification, and question-answering systems, etc. For example, in e-commerce platforms, sentiment classification is used to automatically identify the emotional tendency (positive, negative, neutral) of user reviews to help businesses improve products and services; in the financial industry, text classification can assist in identifying risk signals in financial news to support decision analysis; and in social media platforms, classification technology can be used to filter bad content and identify hot topics.
[0003] Although current deep learning models have made significant progress in this task, their performance often depends on large amounts of high-quality labeled data. However, the manual labeling process is costly and time-consuming, and especially under the condition of few samples, the model performance is often severely constrained. In practical scenarios, especially in professional fields (such as law, medicine, finance) or low-resource languages, obtaining sufficient labeled data often faces the following difficulties:
[0004] 1. High labeling cost: Text content understanding often requires the participation of domain experts, and labeling consumes manpower and time;
[0005] 2. Uneven sample distribution: Many real-world application scenarios have class imbalance, leading to model learning bias;
[0006] 3. Small sample problem is prominent: In emerging fields or unpopular tasks, there are very few available labeled samples, making it difficult for the model to generalize.
[0007] Therefore, how to construct an efficient semi-supervised learning method that can make full use of a small amount of labeled data and a large amount of unlabeled data has become an important challenge in current research. In view of the above, the present application provides a few-shot text classification method based on BERT model and KNN semi-supervised learning to solve the above problems. SUMMARY
[0008] In view of the above, the present application provides a few-shot text classification method based on BERT model and KNN semi-supervised learning to solve the above problems.
[0009] A few-shot text classification method based on BERT model and KNN semi-supervised learning, characterized in that it comprises the following steps:
[0010] S1: supervised training of the BertATT model using the original labeled dataset, and after completing the training of the labeled dataset, the BertATT model is used to predict the unlabeled dataset;
[0011] S2: based on the prediction result of the BertATT model, the current labeled dataset and the unlabeled dataset are extracted according to the attention distribution of the BertATT model to reconstruct the semantic space;
[0012] S3: based on the reconstructed semantic space, introduce the KNN model, calculate the distance between each unlabeled dataset sample and all labeled dataset samples, and through the KNN voting mechanism, assign a pseudo label to the high confidence unlabeled dataset sample, and add it to the training set for subsequent training;
[0013] Thus, while improving the accuracy of pseudo-labeling, the model performance is gradually optimized, and efficient use of the small sample labeled dataset is achieved.
[0014] The above technical solution has the following beneficial effects:
[0015] (1) The present application reconstructs the semantic feature space of the dataset, classifies the unlabeled samples through the KNN voting mechanism, assigns a pseudo label to the unlabeled sample, and adds it to the training set for subsequent training, thereby improving the accuracy of the pseudo label when it is assigned, optimizing the performance of the model, and achieving efficient use of the small amount of labeled data;
[0016] (2) The present application introduces the BERT attention mechanism to model the key information of the input sample, which can accurately extract representative semantic features when the sample quantity is limited or unevenly distributed, thereby alleviating the performance fluctuation problem caused by low-quality labeled samples, improving the stability and generalization ability of the model training, and reducing the dependence on the quality of the initial labeled samples;
[0017] (3) The semi-supervised learning framework proposed in the present application is flexible in structure, suitable for BERT and its variant models, and also compatible with other classification models based on vector space or graph structure, easy to deploy in different task scenarios and data scales, and has strong engineering applicability and cross-domain expandability. BRIEF DESCRIPTION OF DRAWINGS
[0018] Figure 1 The figure is a schematic diagram of the overall framework module structure of the present application. DETAILED DESCRIPTION
[0019] The foregoing and other technical contents, features and effects of the present application will be clearly presented in the following detailed description of embodiments with reference to the accompanying drawings. The content mentioned in the following embodiments is all referred to the drawings.
[0020] As shown in Embodiment 1, Figure 1 The overall framework of the method is divided into three modules: basic model training and prediction, sample space reconstruction, and pseudo-label generation. A semi-supervised learning framework is proposed by combining the BERT attention mechanism and the K-Nearest Neighbor algorithm (KNN). First, the initial labeled data is used to train the BertATT model, and the high attention tokens of the labeled and unlabeled samples are extracted according to the attention distribution of the model, so as to reconstruct the semantic space and enhance the sample representation ability. Subsequently, in the pseudo-label generation stage, the KNN algorithm is introduced, and the distance between the unlabeled samples and the labeled samples in the reconstructed sample space is calculated. Through the KNN voting mechanism, pseudo-labels are assigned to high-confidence unlabeled samples, and they are added to the training set to participate in subsequent training. This method improves the accuracy of pseudo-labeling while gradually optimizing the model performance, achieving efficient use of a small amount of labeled data.
[0021] In the basic model training and prediction module, a small amount of labeled data set is used to supervise the training of the BertATT model. Specifically, the labeled data is first input into the BertATT model. The model uses the multi-head self-attention mechanism (Multi-Head Self-Attention) of BERT to model the input sequence globally and generate semantic feature representation and attention weight for each token. Through the attention mechanism, the model can identify the most important context information for the classification task. Then, the BertATT model performs sentence-level feature aggregation on the input sequence, usually through a pooling operation to extract global feature vectors. These feature vectors are passed through a fully connected layer to generate a class distribution prediction result. In the training process, the cross-entropy loss function as shown in formula (1) is used to optimize the model parameters to minimize the difference between the predicted distribution and the true class label.
[0022]
[0023] wherein, is the true label of sample i, is the probability distribution predicted by the model.
[0024] After the training of the labeled dataset is completed, the BertATT model is used to make predictions on the unlabeled data. During the prediction process, the model not only outputs the class prediction distribution of each sample, but also generates an attention weight matrix to quantify the importance of different positions in the input text. These prediction results and attention weights provide semantic features and class information for subsequent modules.
[0025] In embodiment 3, the sample semantic feature space is reconstructed. The core goal of the sample space reconstruction module is to dynamically reconstruct the representation space of the labeled dataset and the unlabeled dataset based on the prediction results of the model. First, the attention mechanism of the BertATT model is used to analyze the input text and extract tokens with high attention weights. According to the preset threshold, a set of high-weight tokens is selected to construct a semantic representation that focuses on key information. By retaining only tokens with high attention distribution, the sample dimension is effectively compressed, and redundant and noisy features are removed, thereby improving the discriminability of the representation. Finally, the sample is mapped to a simplified feature space composed of high-attention tokens, and serves as the basis for distance measurement and pseudo-label generation in the subsequent KNN module.
[0026] In embodiment 4, pseudo labels are generated. The pseudo label generation module aims to select high-confidence unlabeled samples based on the reconstructed feature space to expand the labeled dataset, thereby improving the quality of the model's supervision signal and generalization ability. Specifically, the module first calculates the similarity between each unlabeled sample and the labeled sample in the newly constructed semantic representation space. The cosine similarity (Cosine Similarity) is used as the measurement index, as shown in equation (2):
[0027]
[0028] where x i and x j represent the vector representation of two samples in the reconstructed feature space. Through this measurement method, the system can identify neighborhood samples that are more semantically similar, providing a reliable basis for subsequent KNN voting.
[0029] After the similarity calculation is completed, for each unlabeled sample, its K nearest neighbor samples are selected, and a majority vote is performed according to their respective classes to determine the temporary label t i . Subsequently, the similarity between the neighbor samples and the target sample is combined to calculate the weighted confidence C i of the temporary label t i , which serves as a measure of the effectiveness of the pseudo label. Only when the sample confidence exceeds the set threshold, will its pseudo label be accepted and added to the expanded training set.
[0030] To enhance the stability and robustness of the pseudo-label generation process, the voting information of the previous iteration is introduced to update the current label and confidence. The update rule is as follows:
[0031] (1) The label is consistent (the voting results of the current round and the last round are the same), which is specifically manifested as the voting results of the current round and the last round are the same, i.e., the temporary label t i of the current round is not changed, and the weighted confidence C i is updated to formula (3) as shown:
[0032]
[0033] In the formula, is the temporary label t i of the current round of sample i, and the corresponding confidence is C is the temporary label t i of the sample i of the last round, and the corresponding weighted confidence is C is a weight parameter of α;
[0034] (2) When the label is inconsistent, it is specifically manifested as the voting results of the current round and the last round are different, i.e., the temporary label t i of the current round is changed, and only when formula (4) is met as shown below:
[0035]
[0036] When the temporary label of the sample and the corresponding confidence are retained, the temporary label t i of the current round is retained, in the formula, is the temporary label t i of the current round of sample i, and the corresponding confidence is C is the temporary label t i of the sample i of the last round, and the corresponding weighted confidence is C is a preset threshold value for judging whether the confidence C is increased enough; otherwise, the temporary label t i and the corresponding weighted confidence C i of the last round are still retained.
[0037] That is, only when the temporary label of the sample i is consistent in the two iterations, we introduce the weight α to calculate the weighted confidence of the two rounds; if the temporary label of the sample i is inconsistent in the two iterations, only when the confidence corresponding to the predicted temporary label of the current round is large enough, we determine the temporary label and the confidence of the sample i as the temporary label and the confidence obtained by the KNN voting of the current round, otherwise, the final temporary label and the confidence of the sample i of the last round are retained to prevent too much fluctuation in different iterations.
[0038] The module ensures that the generated pseudo-labels have good semantic consistency and distribution reliability by introducing a KNN strategy in the semantic reconstruction space and a multi-round confidence fusion mechanism. During the iteration process, the mechanism continuously improves the quality of pseudo-labels, providing stronger learning signals for the model, thereby achieving effective modeling of unlabeled samples and enhancing generalization ability.
[0039] In Example 5, four public datasets, IMDB review, AG News, Yahoo! Answers, and DBpedia, are used to evaluate the performance of the method proposed in this study on different domain datasets. 1% of the original dataset is randomly selected as the labeled dataset, and the remaining 99% is used as the unlabeled dataset. For the new labeled dataset, 85% is used as the training set, and the remaining 15% is used as the development set. The distribution of the datasets is shown in Table 1:
[0040] Table 1 Performance of the method on different public datasets
[0041]
[0042] Note: In Table 1, Dataset is the dataset, Type is the label type, Classes is the number of dataset categories, Unlabeled is the number of unlabeled data samples, Train is the number of training set samples, Dev is the number of validation set samples, Test is the number of test set samples; Review Sentiment represents review sentiment, News Topic represents news topic, QA Topic represents question and answer topic, and Wikipedia Topic represents Wikipedia topic.
[0043] To verify the efficiency of the model, accuracy (Accuracy), average F1 value (Macro F1), and are used as evaluation indicators. The experimental results are shown in Table 2:
[0044] Table 2 Actual operation of each index of the method
[0045]
[0046] The IMDB review dataset is a sentiment analysis dataset containing a large number of movie review data, and the task is to judge the sentiment tendency (positive or negative) of the review. On this dataset, the accuracy of the model is 83.472%, that is, the model correctly classifies about 83.47% of the reviews. The macro F1 score is 83.471%, which takes into account the F1 score of each class and reflects the overall performance of the model on all classes. The numerical value indicates that the model has high stability and performance on this dataset.
[0047] The AG News dataset is a news classification dataset, and the task is to classify news text into one of four topic categories. The model performs slightly better on the AG News dataset than the IMDB review dataset, with an accuracy of 85.237%, which is about 1.77% higher than the accuracy of the IMDB dataset. The macro F1 score is 85.197%, which is also higher than the IMDB review dataset, showing better performance of the model in multi-class classification tasks.
[0048] From this, we can preliminarily conclude that the semi-supervised few-shot classification framework based on BertATT proposed in this paper is more suitable for few-shot classification tasks.
Claims
1. A few-shot text classification method based on a BERT model and KNN semi-supervised learning, characterized in that, The method comprises the following steps: S1: supervised training of the BertATT model using the original labeled dataset, and after the training of the labeled dataset is completed, the BertATT model is used to predict the unlabeled dataset; S2: based on the prediction result of the BertATT model, the current labeled dataset and the unlabeled dataset are extracted according to the attention distribution of the BertATT model, so as to reconstruct the semantic space; S3: based on the reconstructed semantic space, a KNN model is introduced, the distance between each unlabeled dataset sample and all labeled dataset samples is calculated, the KNN voting mechanism is used to assign a pseudo label to the high-confidence unlabeled dataset sample, and the sample is added to the training set for subsequent training; Thus, while improving the accuracy of pseudo labeling, the model performance is gradually optimized, and the efficient use of the few-sample labeled dataset is realized.
2. The few-sample text classification method based on BERT model and KNN semi-supervised learning according to claim 1, characterized in that: The supervised training in step S1 uses a cross-entropy loss function to optimize the parameters of the BertATT model; The prediction process includes that the BertATT model not only outputs the class prediction distribution of each sample of the unlabeled dataset, but also generates an attention weight matrix for quantifying the importance of different positions in the input text. These class prediction distribution results and attention weight matrices provide semantic features and class information for subsequent steps.
3. The few-shot text classification method based on BERT model and KNN semi-supervised learning according to claim 2, characterized in that, Step S2 specifically includes: S2-1: using the attention mechanism of the BertATT model to analyze the labeled dataset and the unlabeled dataset, and extracting tokens with high attention weights therefrom; S2-2: according to the pre-set threshold, the token set with high weights is screened out to construct a semantic representation focusing on key information; S2-3: finally, the dataset is mapped to a simplified feature semantic space reconstructed by the token set with high weights, and is used as a basic representation for distance calculation and pseudo label assignment in the subsequent KNN model.
4. The few-shot text classification method based on the BERT model and KNN semi-supervised learning according to claim 3, characterized in that, Step S3 specifically includes: S3-1: first, in the reconstructed simplified feature semantic space, the similarity between each unlabeled dataset sample and all labeled dataset samples is calculated, and the cosine similarity (Cosine Similarity) is used as a measurement index, as shown in formula (1): where x i and x j respectively represent the vector representation of two samples in the reconstructed reduced feature semantic space in the dataset, through the measurement method, the system can identify the neighborhood samples that are more close in semantics, and provide reliable basis for subsequent KNN voting; S3-2: After completing the similarity calculation, for each unlabeled sample in the data set, i.e., the target sample, select its K nearest neighbor samples, and determine the temporary label t according to the category to which the nearest neighbor samples belong i Then, combined with the similarity between the nearest neighbor samples and the target sample, calculate the temporary label t i The corresponding weighted confidence C i As an effectiveness measure standard of the pseudo label, i.e., the temporary label t i Only when the target sample weighted confidence C i Exceeds the preset threshold, the pseudo label of the target sample will be adopted, and the target sample will be added to the subsequent training set.
5. The few-shot text classification method based on BERT model and KNN semi-supervised learning according to claim 4, in order to enhance the stability and robustness of the pseudo-label generation process, when performing majority voting, the voting information of the previous round of iteration is introduced to update the current temporary label t i and the weighted confidence is fused and updated, characterized in that, The update rule is as follows: The current round is the same as the previous round of voting, that is, the temporary label t i When it is not changed, we will update the weighted confidence C i The introduction of the voting information of the previous round of iteration can effectively reduce the influence of noise, wherein the weighted confidence C i The update is shown in formula (2): wherein is the temporary label t of the current round sample i i is the corresponding confidence; is the temporary label t of the previous round sample i i is the corresponding weighted confidence; a is is the weight parameter; The current round and the previous round of voting results are not the same, that is, the temporary label t i When the change occurs, only when the following formula (3) is satisfied: The sample temporary label and its corresponding confidence reserve the temporary label t of the current round i and its corresponding confidence wherein, is the temporary label t of the sample i of the current round i corresponding confidence; is the temporary label t of the sample i of the last round i corresponding weighted confidence; β is the preset threshold value for judging whether the increase is large enough; otherwise, the temporary label t of the last round is still reserved and its corresponding weighted confidence C i and its corresponding weighted confidence C i to prevent too much fluctuation in different iterations.