Integrated active learning scientific text classification method and system based on prompt tuning
Through an integrated active learning method based on prompt tuning, pre-trained models and hard templates are used for text classification, the problems of insufficient performance and sampling strategy deviation when applied on large data sets are solved, and higher classification accuracy and model reliance are achieved Great.
Patent Information
- Application Number
- CN202411498523.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-25
- Publication Date
- 2025-02-21
AI Technical Summary
When the existing text classification method is applied on large data sets, it is suggested that the advantages of tuning technology are not obvious, sampling strategies may lead to model overfitting, and a single sampling strategy performs inconsistently in different data sets and tasks.
An integrated active learning method based on prompt tuning is adopted to predict through pre-trained models and hard templates, calculating the integrated confidence screening sample, marking and data updates, and training the model iteratively round by round.
In the case of scarce sample data, improve classification accuracy, overcome the bias and diversity of sampling strategies, and enhance the learning efficiency and robustness of the model.
Smart Images
Figure CN119494013A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of text classification, and in particular to an integrated active learning scientific text classification method and system based on prompt tuning. Background Art
[0002] Text classification is a core task in natural language processing (NLP), which aims to automatically classify texts into corresponding categories by analyzing the subject and content of the text according to preset classification standards. This task has wide applications in information retrieval, public opinion monitoring, content filtering and other fields.
[0003] Early text classification mainly relied on machine learning models, such as Naive Bayes, support vector machines, decision trees, and random forests. These methods usually require manual feature engineering, including building bag-of-words models and calculating TF-IDF weights. The process is cumbersome and highly dependent on feature extraction. With the rise of deep learning, deep learning architectures such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), and long short-term memory networks (LSTMs) have begun to be widely used in text classification. These models can automatically learn feature representations without manual feature extraction, and can process long texts more flexibly. They significantly improve the performance of text classification by capturing sequence information in text.
[0004] In recent years, pre-trained deep learning models, such as BERT, GPT, and XLNet, have become important tools for text classification tasks. These models are pre-trained on large-scale text data and have learned rich language representation capabilities. On specific tasks, by fine-tuning the pre-trained models, impressive performance can be achieved. Figure 1 As shown in the figure, the input representation of the BERT pre-trained model includes token embedding (embedding of the word itself), segment embedding (distinguishing different sentences) and position embedding (indicating the position of the word), which can fully understand the text information. As a variant of the BERT architecture, SciBERT has optimized the vocabulary for scientific corpora and built a new WordPiece vocabulary (SCIVOCAB) using the SentencePiece library, thus showing better performance than BERT in scientific text tasks.
[0005] Although fine-tuning of pre-trained models has achieved great success, there is a large gap between pre-training and fine-tuning of models. In order to alleviate this problem, the concept of prompt tuning came into being. The Pattern Exploiting Training (PET) method proposed by Schick converts the text classification task into a cloze task by constructing a natural language hard template, thereby using BERT's MLM model for prediction. This method effectively narrows the target gap between pre-training and fine-tuning, and further improves model performance. In summary, the existing text classification methods still have the following problems:
[0006] First, the limitations of the hint tuning technique: on few-sample tasks, the hint tuning technique outperforms traditional fine-tuning. However, when the number of training instances increases, its advantage is not obvious. This limits the application of the hint tuning technique on large datasets.
[0007] Second, the bias of sampling strategy: The sampling strategy in active learning may be biased towards selecting samples of a certain category, causing the model to overfit on these categories. This bias will accumulate as the number of iterations increases, affecting the overall performance of the model.
[0008] Third, the diversity of sampling strategies: The most effective sampling strategies may be different in different datasets and tasks. A single sampling strategy may perform well on one task but poorly on another. This requires that in practical applications, appropriate sampling strategies should be selected according to specific circumstances. Summary of the invention
[0009] To solve the above technical problems, the present invention provides an integrated active learning scientific text classification method based on prompt tuning, which includes the following steps:
[0010] S1: Obtain a first labeled text data set, an unlabeled second text data set, and a third labeled text data set, and use the first text data sets to pre-train a text classification model constructed in advance to obtain a pre-trained text classification model;
[0011] S2: Based on the pre-trained text classification model, use a hard template including prediction points to predict the second text data set to obtain the confidence of the prediction label of each sample data;
[0012] S3: According to the confidence level, sample data with a confidence level lower than a preset threshold is selected from the second text data set as a fourth text data set, and the fourth text data set is marked to obtain a marked fourth text data set;
[0013] S4: using the marked fourth text data set to update the first text data set to obtain an updated first text data set;
[0014] S5: Using the updated first text data set to train the next round of text classification model, in each iterative training, determine whether the set number of iterations is reached or the evaluation index value does not change:
[0015] If not, return to step S2;
[0016] If yes, a trained text classification model is obtained, and a classification result is obtained for the text data to be classified using the trained text classification model.
[0017] In one embodiment of the present invention, in S2, the method for predicting the second text data set using a hard template including prediction points is as follows:
[0018] The hard template is spliced to each original input text in the second text data set, and the text classification task is converted into a cloze task to form a new input text.
[0019] In one embodiment of the present invention, in S2, the method for obtaining the confidence of the predicted label of each sample data is:
[0020] Ec(x)=L(x)+a*H(x)+b*[1-M(x)]
[0021] Among them, Ec(x) is the integrated confidence, L(x) is the minimum confidence, H(x) is the entropy sampling value, M(x) is the edge sampling value, and a and b are both trainable parameters.
[0022] In one embodiment of the present invention, the minimum confidence L(x):
[0023]
[0024] in, is the most likely category label, P is the probability of the sample output by the discriminant model, and x is the sample input to the model.
[0025] In one embodiment of the present invention, the entropy sampling value H(x):
[0026] H(x)=-∑P(y i |x)·lnP(y i |x)
[0027] Among them, y i is the label of the i-th category.
[0028] In one embodiment of the present invention, the edge sampling value Mx is:
[0029] M(x)=P(y^1|x)-P(y2|x)
[0030] Here, y^1 and y^2 are the first and second most likely class labels.
[0031] In one embodiment of the present invention, the labels of the first text data set include background text, method text and conclusion text.
[0032] Based on the same inventive concept, the present invention also provides an integrated active learning scientific text classification system based on prompt tuning, including:
[0033] A model pre-training module, used to obtain a first labeled text data set, an unlabeled second text data set, and a third labeled text data set, and use the first text data set to pre-train a text classification model constructed in advance to obtain a pre-trained text classification model;
[0034] A confidence calculation module, used to predict the second text data set based on the pre-trained text classification model using a hard template including prediction points to obtain the confidence of the prediction label of each sample data;
[0035] A data screening module, configured to screen out sample data with a confidence level lower than a preset threshold from the second text data set according to the confidence level as a fourth text data set, and mark the fourth text data set to obtain a marked fourth text data set;
[0036] a label data updating module, configured to update the first text data set using the labeled fourth text data set to obtain an updated first text data set;
[0037] The text classification module is used to train the next round of text classification model using the updated first text data set. In each iterative training, if the set number of iterations is reached or the evaluation index value no longer changes, a trained text classification model is obtained. For the text data to be classified, the trained text classification model is used to obtain the classification result.
[0038] The present invention also provides a text classification device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned integrated active learning scientific text classification method based on prompt tuning when executing the program.
[0039] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the instructions of the above-mentioned integrated active learning scientific text classification method based on prompt tuning.
[0040] The above technical solution of the present invention has the following advantages compared with the prior art:
[0041] The present invention uses the prompt tuning model as a discriminator for the active learning process to more accurately select "high-value" samples. This method makes full use of the unique advantages of the prompt tuning model in processing a small number of sample classification tasks, and can achieve higher classification accuracy when the amount of data is limited. In addition, in order to overcome the limitations of the traditional single sampling strategy, the present invention proposes to construct an integrated confidence scoring mechanism, which effectively responds to the challenges brought by sample imbalance, thereby improving the processing capabilities of multi-classification tasks. This series of improvement measures not only optimizes the learning efficiency of the model, but also enhances its robustness and accuracy in complex application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below according to specific embodiments of the present invention in conjunction with the accompanying drawings, wherein:
[0043] Figure 1 This is a schematic diagram of the structure of the BERT model;
[0044] Figure 2 It is a flow chart of an integrated active learning scientific text classification method based on prompt tuning provided in the first embodiment of the present invention;
[0045] Figure 3 is a schematic diagram of prompt tuning, where (a) represents a classifier based on a PET hard template, and (b) represents an active learning discriminator or active learning sampler based on a PET hard template, which is used for “high-value” sample screening;
[0046] Figure 4 This is a structural diagram of an integrated active learning scientific text classification system based on prompt tuning provided in the second embodiment of the present invention;
[0047] Explanation of the reference numerals in the specification: 100, model pre-training module; 200, confidence calculation module; 300, data screening module; 400, label data updating module; 500, text classification module. DETAILED DESCRIPTION
[0048] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.
[0049] Embodiment 1
[0050] Reference Figure 2As shown, the present invention provides an integrated active learning scientific text classification method based on prompt tuning, which includes the following steps:
[0051] S1: Obtain a first labeled text dataset, an unlabeled second text dataset, and a third labeled text dataset, and use the first text dataset to pre-train a text classification model (SciBERT model) constructed in advance to obtain a pre-trained text classification model;
[0052] S2: Based on the pre-trained text classification model, use a hard template including prediction points to predict the second text data set to obtain the confidence of the prediction label of each sample data;
[0053] S3: According to the confidence level, sample data with a confidence level lower than a preset threshold is selected from the second text data set as a fourth text data set, and the fourth text data set is marked to obtain a marked fourth text data set;
[0054] S4: using the marked fourth text data set to update the first text data set to obtain an updated first text data set;
[0055] S5: Using the updated first text data set to train the next round of text classification model, in each iterative training, determine whether the set number of iterations is reached or the evaluation index value (F1 score) no longer changes:
[0056] If not, return to step S2;
[0057] If yes, a trained text classification model is obtained, and the classification result is obtained for the text data to be classified using the trained text classification model.
[0058] The F1 score is obtained by testing the trained text classification model using the third text dataset as a test set.
[0059] It can be seen from the above technical solutions that the present invention can make full use of all available information when sample data is scarce, and identify the most valuable data for manual labeling through active learning, thereby improving the efficiency of model training; using the pre-trained SciBERT model and adding newly labeled data through continuous iteration, the generalization ability and accuracy of the model can be gradually enhanced. With each iteration, the model will more accurately identify the features in the text and reduce the risk of overfitting.
[0060] Further, in this embodiment, in S2, the method for predicting the second text data set using a hard template including prediction points is as follows:
[0061] The hard template is spliced to each original input text in the second text data set, and the text classification task is converted into a cloze task to form a new input text.
[0062] For example, design a hard template "The article about [MASK] speaks:", where [MASK] is the position that the SciBERT model needs to predict; splice the hard template before each original input text to form a new input text; convert the original text classification task into a cloze task, that is, the SciBERT model needs to predict the label candidate word that is most likely to fill in the [MASK] position.
[0063] In step S2, based on the pre-trained text classification model, the second text data set is predicted using a hard template including prediction points, and three active learning uncertainty sampling strategies are used to obtain the integrated confidence Ec(x) of the prediction label of each sample data. The specific calculation method is:
[0064] Ec(x)=L(x)+a*H(x)+b*[1-M(x)]
[0065] Among them, L(x) is the minimum confidence, H(x) is the entropy sampling value, M(x) is the edge sampling value, and a and b are both trainable parameters. Through the two parameters a and b, the weights of the three sampling methods are adjusted so that the final integrated confidence can help the model better screen out more valuable data.
[0066] The minimum confidence L(x) is used to select the sample with the largest minimum confidence, and its calculation expression is:
[0067]
[0068] in, is the most likely category label, P is the probability of the sample output by the discriminant model, and x is the sample input to the model.
[0069] The entropy sampling value H(x) is used to select the sample with the maximum information entropy, and its calculation expression is:
[0070] H(x)=-∑P(y i |x)·lnP(y i |x)
[0071] Among them, y i is the label of the i-th category.
[0072] The edge sampling value M(x) is used to select the sample with the minimum marginal effect, and its calculation expression is:
[0073] M(x)=P(y^1|x)-P(y2|x)
[0074] Here, y^1 and y^2 are the first and second most likely class labels.
[0075] Furthermore, the labels of the first text dataset include background text, method text and result text. The MLM model of the SciBERT model is used as the final classification model and the discriminator of active learning, as shown in the schematic diagram Figure 3 shown.
[0076] In order to verify the effectiveness of the method described in the present invention, the experimental parameters are set: the training round epoch is 10, the batch size is set to 4, the maximum sentence length is 512, the optimizer uses AdamW, the learning rate is set to 2e-5, and the dataset used is the public citation intent classification dataset SciCite. The dataset composition and examples are shown in Tables 1 and 2.
[0077] Table 1: SciCite data composition
[0078]
[0079] Table 2: SciCite dataset examples
[0080]
[0081] Specific implementation steps:
[0082] 1. Compare the performance of traditional fine-tuning and hinted tuning as classifiers on the SciBERT model using randomly sampled data.
[0083] 2. Compare the results of using different sampling strategies when using traditional fine-tuning methods and hinted tuning methods as active learning discriminators.
[0084] 3. Build integrated confidence and adjust a and b parameters to achieve optimal performance.
[0085] 4. Compare the performance of the proposed method with various existing models.
[0086] The initial label data is set to 100, and 100 labeled data are added in each iteration. The experimental results are shown in Table 3. pt means using hard template prompt tuning, ft means using traditional fine-tuning; ec means integrated confidence, lc means minimum confidence, entropy means entropy sampling, margin means edge sampling, and random means random sampling.
[0087] Table 3 Experimental results of SciCite dataset
[0088]
[0089] It can be observed from Table 3 that using different sampling methods for active learning, the discriminator uses hints to learn hard templates, which outperforms traditional fine-tuning in most cases. The hard template hint-integrated confidence method of the present invention has a significant advantage when training instances 400 and 500, surpassing all methods using a single sampling strategy.
[0090] Parameters a and b control the weights of the three sampling strategies in the integrated confidence. By fine-tuning a and b, the three sampling strategies can be balanced to maximize the model performance. 500 training instances were used to test the impact of different a and b, and finally a = 0.2, b = 0.5 was found to be the optimal solution. The impact of different a and b values on the macro F1 of 500 training instances is shown in Table 4.
[0091] Table 4 Parameter a and b fine-tuning results
[0092]
[0093] In order to further verify the effectiveness of the invention, the SciCite dataset is used to compare the existing models (Prompttuning-L2regulation, CitePrompt, SciBERT, RoBERTa, BERT, Tf-idf+SVM, Glove+LSTM, BiLSTM-Attention) and the model proposed in the present invention, as shown in Table 5. The model proposed in the present invention only uses 500 training instances, and still obtains better results than using the entire dataset for SciBERT fine-tuning.
[0094] Table 5 Macro F1 comparison between existing models and the proposed method on the SciCite dataset
[0095]
[0096] Embodiment 2
[0097] Based on the same inventive concept as the first embodiment, the present invention also provides an integrated active learning scientific text classification system based on prompt tuning. Figure 4 As shown, the system includes the following modules:
[0098] The model pre-training module 100 is used to obtain a first labeled text data set, an unlabeled second text data set, and a third labeled text data set, and use the first text data set to pre-train a text classification model constructed in advance to obtain a pre-trained text classification model;
[0099] A confidence calculation module 200, configured to predict the second text data set based on the pre-trained text classification model using a hard template including prediction points to obtain the confidence of the prediction label of each sample data;
[0100] A data screening module 300 is used to screen out sample data with a confidence level lower than a preset threshold from the second text data set according to the confidence level as a fourth text data set, mark the fourth text data set, and obtain a marked fourth text data set;
[0101] A label data updating module 400, configured to update the first text data set using the labeled fourth text data set to obtain an updated first text data set;
[0102] The text classification module 500 is used to train the next round of text classification model using the updated first text data set. In each iterative training, if the set number of iterations is reached or the evaluation index value no longer changes, a trained text classification model is obtained. For the text data to be classified, the classification result is obtained using the trained text classification model.
[0103] The present embodiment proposes an integrated active learning scientific text classification system based on prompt tuning, which is used to implement the aforementioned integrated active learning scientific text classification method based on prompt tuning. Therefore, the specific implementation method of the integrated active learning scientific text classification system based on prompt tuning can be seen in the embodiment part of the aforementioned integrated active learning scientific text classification method based on prompt tuning. For example, the model pre-training module 100, the confidence calculation module 200, the data screening module 300, the label data update module 400 and the text classification module 500 are respectively used to implement the steps S1, S2, S3, S4 and S5 in the integrated active learning scientific text classification method based on prompt tuning in the first embodiment. Therefore, its specific implementation method can refer to the description of the corresponding embodiments of each part. In order to avoid redundancy, it will not be repeated here.
[0104] Embodiment 3
[0105] The present invention also provides a text classification device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the integrated active learning scientific text classification method based on prompt tuning provided in the first embodiment is implemented.
[0106] Embodiment 4
[0107] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements instructions for the integrated active learning scientific text classification method based on prompt tuning provided in the first embodiment above.
[0108] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.
[0109] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0110] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0111] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0112] Obviously, the above embodiments are merely examples for clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived from these are still within the protection scope of the invention.
Claims
1. An integrated active learning scientific text classification method based on prompt tuning, characterized in that: The following steps are involved: S1: Obtain a first labeled text data set, an unlabeled second text data set, and a third labeled text data set, and use the first text data sets to pre-train a text classification model constructed in advance to obtain a pre-trained text classification model; S2: Based on the pre-trained text classification model, use a hard template including prediction points to predict the second text data set to obtain the confidence of the prediction label of each sample data; S3: According to the confidence level, sample data with a confidence level lower than a preset threshold is selected from the second text data set as a fourth text data set, and the fourth text data set is marked to obtain a marked fourth text data set; S4: using the marked fourth text data set to update the first text data set to obtain an updated first text data set; S5: Using the updated first text data set to train the next round of text classification model, in each iterative training, determine whether the set number of iterations is reached or the evaluation index value does not change: If not, return to step S2; If yes, a trained text classification model is obtained, and a classification result is obtained for the text data to be classified using the trained text classification model.
2. The integrated active learning scientific text classification method based on prompt tuning according to claim 1 is characterized in that: In S2, the method for predicting the second text data set using the hard template including the prediction point is as follows: The hard template is spliced to each original input text in the second text data set, and the text classification task is converted into a cloze task to form a new input text.
3. The integrated active learning scientific text classification method based on prompt tuning according to claim 1 is characterized in that: In S2, the method for obtaining the confidence of the predicted label of each sample data is: Ec(x)=L(x)+a*H(x)+b*[1-M(x)] Among them, Ec(x) is the integrated confidence, L(x) is the minimum confidence, H(x) is the entropy sampling value, M(x) is the edge sampling value, and a and b are both trainable parameters.
4. The integrated active learning scientific text classification method based on prompt tuning according to claim 3 is characterized in that: The minimum confidence L(x): in, is the most likely category label, P is the probability of the sample output by the discriminant model, and x is the sample input to the model.
5. The integrated active learning scientific text classification method based on prompt tuning according to claim 3 is characterized in that: The entropy sampling value H(x): H(x)=-∑P(y i |x)·lnP(y i |x) Among them, y i is the label of the i-th category.
6. The integrated active learning scientific text classification method based on prompt tuning according to claim 3 is characterized in that: The edge sampling value M(x): M(x)=P(y^1|x)-P(y^2|x) Here, y^1 and y^2 are the first and second most likely class labels.
7. The integrated active learning scientific text classification method based on prompt tuning according to claim 1 is characterized in that: The labels of the first text data set include background text, method text and conclusion text.
8. An integrated active learning scientific text classification system based on prompt tuning, characterized in that: include: A model pre-training module, used to obtain a first labeled text data set, an unlabeled second text data set, and a third labeled text data set, and use the first text data set to pre-train a text classification model constructed in advance to obtain a pre-trained text classification model; A confidence calculation module, used to predict the second text data set based on the pre-trained text classification model using a hard template including prediction points to obtain the confidence of the prediction label of each sample data; A data screening module, configured to screen out sample data with a confidence level lower than a preset threshold from the second text data set according to the confidence level as a fourth text data set, and mark the fourth text data set to obtain a marked fourth text data set; a label data updating module, configured to update the first text data set using the labeled fourth text data set to obtain an updated first text data set; The text classification module is used to train the next round of text classification model using the updated first text data set. In each iterative training, if the set number of iterations is reached or the evaluation index value no longer changes, a trained text classification model is obtained. For the text data to be classified, the trained text classification model is used to obtain the classification result.
9. A text classification device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the integrated active learning scientific text classification method based on prompt tuning is implemented as described in any one of claims 1 to 7.
10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the computer program implements instructions for the integrated active learning scientific text classification method based on prompt tuning according to any one of claims 1 to 7.