A simple and easy-to-use incremental learning text classification method and system

Abstract

The present invention discloses a simple and easy-to-use incremental learning text classification method and system. The method converts the text classification task into a masked language task based on a pre-trained model, combines a prompt template and a label word mapping function, and designs an incremental learning text classification model. The model can keep the original network structure of the model unchanged during incremental learning, reduce manual intervention, learn new and old category data at the same time, and reduce the difficulty of incremental learning of the model.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing, and more specifically, to a simple and easy-to-use incremental learning text classification method and system. Background Art

[0002] Deep neural networks have good application prospects in a large number of text classification tasks such as intent detection and relationship recognition. However, in the supervised learning of neural network models, the model often encounters new categories of data that have never been seen before. The neural network model will incorrectly classify this data into known categories, resulting in a decrease in classification accuracy. In order to solve the above problem, incremental learning methods came into being.

[0003] The goal of incremental learning is to allow computers to simulate human learning patterns, to continuously learn new knowledge from new samples, and to preserve previously learned old knowledge. The biggest problem facing incremental learning is "catastrophic forgetting", that is, the phenomenon of forgetting old tasks when learning new tasks. At present, three types of technologies are mainly used to solve catastrophic forgetting: playback-based, regularization-based, and parameter isolation-based. However, the above methods still require more manual intervention and are difficult to incrementally learn models. Summary of the invention

[0004] The present invention overcomes the shortcomings of the prior art and provides a simple and easy-to-use incremental learning text classification method and system, in the hope of solving the problems in the prior art that incremental learning requires a lot of manual intervention and the model incremental learning is difficult.

[0005] On the one hand, the present invention provides a simple and easy-to-use incremental learning text classification method, comprising the following steps:

[0006] S1. Get the labeled original classification dataset D ori , the category set is Y n , and the original classification data set D ori Converted into MLM training sample set with prompt template;

[0007] S11. Create a prompt template prompt;

[0008] S12, construct a label word mapping function f, and use the label word mapping function f to transform the category set Y n The categories in are mapped to category labels to form a category label set L n ;

[0009] S13, based on the prompt template prompt and the label word mapping function f, converting the original classified text into prompt template data including a mask filling position [MASK], wherein the mask filling position [MASK] is used to fill in the category label matching the text to be classified;

[0010] S2. Convert the text classification task into a masked language (i.e., cloze) task, build an MLM (Mask Language Model) model, which is denoted by M later as a classification model, and use it as the original MLM training sample set T n Perform initial model training and mark the trained classification model as M n ;

[0011] S3. From the original classification dataset D ori Filter the historical data set D with rich information content old ;

[0012] S31, the original classification data D ori Perform feature extraction and transform the original MLM training sample T n Input into the trained model M, and use the [CLS] vector representation output by the model as the feature of the input sample;

[0013] S32, clustering the samples based on the extracted sample feature representation;

[0014] S33. Select typical samples D old , process each cluster after clustering, calculate the Euclidean distance between the sample and the cluster center, and select the Top-K samples close to the cluster center as representative samples for storage.

[0015] S4. Collect new category dataset D new , and the historical data set D old Summarize to form a new training data set D t , and D t Converted to MLM training sample set T with prompt template n+1 ;

[0016] S5, based on the new MLM training sample T n+1 For the classification model M n Perform incremental training to obtain a new model M that can predict new and old category data n+1 .

[0017] The dimension size of the predicted probability distribution is equal to the size of the pre-trained dictionary and has nothing to do with the size of the set after the new and old categories are merged. This makes it unnecessary to change the structure of the neural network model in incremental training, reducing the burden of model construction.

[0018] The present invention transforms the text classification task into a masked language task based on a pre-trained model, and designs an incremental learning text classification model by combining the prompt template and the label word mapping function. The model can keep the original network structure of the model unchanged during incremental learning, reduce manual intervention, learn new and old category data at the same time, and reduce the difficulty of incremental learning of the model.

[0019] A further technical solution is that in S2, the text classification task is converted into a masked language task, an MLM model is built as a classification model, and based on the original MLM training sample set T n Perform initial model training and mark the trained classification model as M n , specifically including the following steps:

[0020] S21, the text classification model uses the MLM pre-training model based on the transformer structure, and the dictionary size of the pre-training model is υ;

[0021] S22. Based on the MLM model, the text classification task is transformed into a cloze task. The probability that the input text x belongs to a certain class y is proportional to x. prompt The probability of [MASK] being predicted as the mapped label word f(y) is as shown in Formula 1:

[0022] p(y / x)∝M(([MASK]=f(y)) / x prompt ) (1)

[0023] Among them, x is the input text;

[0024] p(y / x) is the probability that x belongs to category y;

[0025] M is the MLM (Mask Language Model) model;

[0026] [MASK] is the mask filling position;

[0027] f(y) is the category label generated by the label word mapping function f for category y;

[0028] x prompt It is the result of x based on the prompt template prompt and the label word mapping function f.

[0029] The pre-trained model is used to predict the probability of the [MASK] position when different category labels are filled in, and the category corresponding to the category label with the maximum probability is used as the predicted category of the text to be classified.

[0030] S23. The model predicts the probability distribution of the [MASK] position as:

[0031]

[0032] Where, q([MASK] / x prompt ) is the cross entropy between the loss function of model training and the true value at the [MASK] position;

[0033] υ is the probability distribution dimension of the MLM model output;

[0034] Since υ and the category set Y n This makes it possible to keep the structure of the neural network model unchanged during incremental training, thus reducing the burden of model construction.

[0035] A further technical solution is that the S4 collects a new category data set D new , and the historical data set D old Summarize to form a new training data set D t , and D t Converted to MLM training sample set T with prompt template n+1 , specifically including the following steps:

[0036] S41. Obtain new category data and annotate it to obtain annotated new category data D new , where the new category set is Y new , Y n ∩Y new =Φ, where Φ represents the empty set;

[0037] S42, through the label word mapping function f, the new category set Y new The categories in are mapped to category labels to form a category label set L new , L n ∩L new =Φ;

[0038] S43, merge the new and old category label sets to form a new label set L n ∪L new →L n+1 , merge the new and old category sets to form a new category set Y n ∪Y new →Y n+1 ;

[0039] S44, based on the prompt template prompt and the label word mapping function f, convert the new category data into a new MLM training sample;

[0040] S45, merging the new MLM training sample and the old typical training sample to form T n+1 .

[0041] A further technical solution is that the classification model is specifically an MLM pre-training model based on a transformer structure.

[0042] A further technical solution is that the feature extraction adopts a semantic feature extraction model.

[0043] A further technical solution is that the clustering adopts kmeans or a variant of kmeans.

[0044] A further technical solution is that the typical samples are selected by selecting Top-K samples or setting a threshold.

[0045] A further technical solution is that, in the incremental training, the initial network parameters of the model can be the model that has been trained in the previous round, or the pre-trained model parameters that have not been fine-tuned.

[0046] The present invention also provides a simple and easy-to-use incremental learning text classification system, comprising:

[0047] The original classification data set acquisition module is used to collect the original classification data set;

[0048] The text classification model building and training module is used to build and train the classification model. The trained model is represented as M n ;

[0049] Typical old sample acquisition module, used to obtain typical samples from the original classification data set;

[0050] A new category sample acquisition module is added to collect and annotate new category data, mix the annotated data with typical old sample data, and convert them into new MLM training samples;

[0051] Incremental learning module, based on the new MLM training samples, the model M completed in the previous round of training n Continue incremental learning to obtain a new model M that can simultaneously predict new and old category data n+1 .

[0052] Compared with the prior art, the present invention has at least the following beneficial effects: The present invention provides a simple, novel and effective incremental learning method for alleviating catastrophic forgetting and improving the accuracy of model text classification. By converting the text classification task into a masked language task based on a pre-trained model, combined with the prompt template and label word mapping function, an incremental learning text classification model is designed. The model can keep the original network structure of the model unchanged in incremental learning, reduce manual intervention, and learn new and old category data at the same time, reducing the difficulty of incremental learning of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is a schematic diagram of a process of Embodiment 1 of the present invention;

[0054] Figure 2 This is a structural block diagram of the second embodiment of the present invention. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0056] Embodiment 1

[0057] See also Figure 1 , a simple and easy-to-use incremental learning text classification method, including the following steps:

[0058] S1. Get the labeled original classification dataset D ori , the category set is Y n , and the original classification data set D ori Converted into MLM training sample set with prompt template;

[0059] S11. Create a prompt template prompt;

[0060] S12, construct a label word mapping function f, and use the label word mapping function f to transform the category set Y n The categories in are mapped to category labels to form a category label set L n ;

[0061] S13, based on the prompt template prompt and the label word mapping function f, converting the original classified text into prompt template data including a mask filling position [MASK], wherein the mask filling position [MASK] is used to fill in the category label matching the text to be classified;

[0062] After this step, the original classification data set D ori Convert to MLM training sample set T n .

[0063] S2. Convert the text classification task into a masked language (i.e., cloze) task, build an MLM model (denoted by M) as the classification model, and use the original MLM training sample set T as the training sample. n Perform initial model training and mark the trained classification model as M n ;

[0064] S21, the text classification model uses the MLM (Mask Language Model) pre-training model based on the transformer structure, and the dictionary size of the pre-training model is υ;

[0065] S22. Based on the MLM model, the text classification task is transformed into a cloze task. The probability that the input text x belongs to a certain class y is proportional to x. prompt [MASK] is predicted as the probability of the mapped label word f(y), see formula 1;

[0066] p(y / x)∝M(([MASK]=f(y)) / x prompt ) (1)

[0067] Among them, x is the input text;

[0068] p(y / x) is the probability that x belongs to category y;

[0069] M is the MLM (Mask Language Model) model;

[0070] [MASK] is the mask filling position;

[0071] f(y) is the category label generated by the label word mapping function f for category y;

[0072] x prompt It is the result of x based on the prompt template prompt and the label word mapping function f.

[0073] The pre-trained model is used to predict the probability of the [MASK] position when different category labels are filled in, and the category corresponding to the category label with the maximum probability is used as the predicted category of the text to be classified.

[0074] S23. The model predicts the probability distribution of the [MASK] position as:

[0075]

[0076] Where, q([MASK] / x prompt ) is the cross entropy between the loss function of model training and the true value at the [MASK] position;

[0077] υ is the probability distribution dimension of the MLM model output;

[0078] S3. From the original classification dataset D ori Filter the historical data set D with rich information content old ;

[0079] S31, the original classification data D ori Perform feature extraction and convert the original MLM training sample T n Input into the trained model M, and use the [CLS] vector representation output by the model as the feature of the input sample;

[0080] It is worth noting that the feature extraction adopts the semantic feature extraction model, which can be a pre-trained model that has been fine-tuned or not, a convolutional neural network, a long short-term memory network and other models. In this embodiment, a pre-trained model that has not been fine-tuned is adopted.

[0081] S32, based on the extracted sample feature representation, clustering the samples using kmeans or its variant kmeans;

[0082] The clustering adopts kmeans or a variant of kmeans. In this embodiment, kmeans is adopted.

[0083] S33. Select typical samples D old , process each cluster after clustering, calculate the Euclidean distance between the sample and the cluster center, and select the Top-K samples close to the cluster center as representative samples for storage.

[0084] The typical samples are selected by selecting Top-K samples or setting a threshold. In this embodiment, the threshold is set.

[0085] S4. Collect new category dataset D new , and the historical data set D old Summarize to form a new training data set D t , and D t Converted to MLM training sample set T with prompt template n+1 ;

[0086] S41. Obtain new category data and annotate it to obtain annotated new category data D new , where the new category set is Y new , Y n ∩Y new =Φ;

[0087] S42, through the label word mapping function f, the new category set Y new The categories in are mapped to category labels to form a category label set L new , L n ∩L new =Φ;

[0088] S43, merge the new and old category label sets to form a new label set L n ∪L new →L n+1 , merge the new and old category sets to form a new category set Y n ∪Y new →Y n+1 ;

[0089] S44, based on the prompt template prompt and the label word mapping function f, convert the new category data into a new MLM training sample;

[0090] S45, merging the new MLM training sample and the old typical training sample to form T n+1 .

[0091] S5, based on the new MLM training sample T n+1 For the classification model M n Perform incremental training to obtain a new model M that can predict new and old category data n+1 .

[0092] Embodiment 2

[0093] See also Figure 2 , a simple and easy-to-use incremental learning text classification system, including:

[0094] The original classification data set acquisition module is used to collect the original classification data set;

[0095] The text classification model building and training module is used to build and train the classification model. The trained model is represented as M n ;

[0096] Typical old sample acquisition module, used to obtain typical samples from the original classification data set;

[0097] A new category sample acquisition module is added to collect and annotate new category data, mix the annotated data with typical old sample data, and convert them into new MLM training samples;

[0098] Incremental learning module, based on the new MLM training samples, the model M completed in the previous round of training n Continue incremental learning to obtain a new model M that can simultaneously predict new and old category data n+1 .

[0099] Although the present invention is described herein with reference to the illustrative embodiments of the present invention, it should be understood that those skilled in the art can design many other modifications and implementations, which will fall within the scope and spirit of the principles disclosed in the present application. More specifically, within the scope disclosed in the present application, multiple variations and improvements can be made to the components and / or layout of the subject combination layout. In addition to the variations and improvements made to the components and / or layout, other uses will also be apparent to those skilled in the art.

Claims

1. A simple and easy-to-use incremental learning text classification method, characterized in that: The following steps are involved: S1. Get the labeled original classification dataset D ori , the category set is Y n , and the original classification data set D ori Converted into MLM training sample set with prompt template; S11. Create a prompt template prompt; S12, construct a label word mapping function f, and use the label word mapping function f to transform the category set Y n The categories in are mapped to category labels to form a category label set L n ; S13, based on the prompt template prompt and the label word mapping function f, converting the original classified text into prompt template data including a mask filling position [MASK], wherein the mask filling position [MASK] is used to fill in the category label matching the text to be classified; S2. Convert the text classification task into a masked language task and build an MLM model. The model is denoted by M later as a classification model and is based on the original MLM training sample set T n Perform initial model training and mark the trained classification model as M n ; S3. From the original classification dataset D ori Filter the historical data set D with rich information content old ; S31, the original classification data D ori Perform feature extraction and convert the original MLM training sample T n Input into the trained model M, and use the [CLS] vector representation output by the model as the feature of the input sample; S32, clustering the samples based on the extracted sample feature representation; S33. Select typical samples D old , process each cluster after clustering, calculate the Euclidean distance between the sample and the cluster center, and select the Top-K samples close to the cluster center as representative samples for storage; S4. Collect new category dataset D new , and the historical data set D old Summarize to form a new training data set D t , and D t Converted to MLM training sample set T with prompt template n+1 ; S5, based on the new MLM training sample T n+1 For the classification model M n Perform incremental training to obtain a new model M that can predict new and old category data n+1 .

2. A simple and easy-to-use incremental learning text classification method as claimed in claim 1, characterized in that: In S2, the text classification task is converted into a masked language task, and an MLM model is built as a classification model. n Perform initial model training and mark the trained classification model as M n , specifically including the following steps: S21, the text classification model uses the MLM pre-training model based on the transformer structure, and the dictionary size of the pre-training model is υ; S22. Based on the MLM model, the text classification task is transformed into a cloze task. The probability that the input text x belongs to a certain class y is proportional to x. prompt [MASK] is predicted as the probability of the mapped label word f(y), see formula 1; p(y / x)∝M(([MASK]=f(y)) / x prompt ) (1) Among them, x is the input text; p(y / x) is the probability that x belongs to category y; M is the MLM (Mask Language Model) model; [MASK] is the mask filling position; f(y) is the category label generated by the label word mapping function f for category y; x prompt x is the result generated by the prompt template prompt and the label word mapping function f; The pre-trained model is used to predict the probability of the [MASK] position when different category labels are filled in, and the category corresponding to the category label with the maximum probability is used as the predicted category of the text to be classified; S23. The model predicts the probability distribution of the [MASK] position as: Where, q([MASK] / x prompt ) is the cross entropy between the loss function of model training and the true value at the [MASK] position; υ is the probability distribution dimension of the MLM model output.

3. A simple and easy-to-use incremental learning text classification method as claimed in claim 1, characterized in that: In S4, a new category dataset D is collected. new , and the historical data set D old Summarize to form a new training data set D t , and D t Converted to MLM training sample set T with prompt template n+1 , specifically including the following steps: S41. Obtain new category data and annotate it to obtain annotated new category data D new , where the new category set is Y new , Y n ∩Y new =Φ, where Φ represents the empty set; S42, through the label word mapping function f, the new category set Y new The categories in are mapped to category labels to form a category label set L new , L n ∩L new =Φ; S43, merge the new and old category label sets to form a new label set L n ∪L new →L n+1 , merge the new and old category sets to form a new category set Y n ∪Y new →Y n+1 ; S44, based on the prompt template prompt and the label word mapping function f, convert the new category data into a new MLM training sample; S45, merging the new MLM training sample and the old typical training sample to form T n+1 .

4. A simple and easy-to-use incremental learning text classification method as claimed in claim 1, characterized in that: The classification model is specifically an MLM pre-training model based on a transformer structure.

5. A simple and easy-to-use incremental learning text classification method as claimed in claim 1, characterized in that: The feature extraction adopts a semantic feature extraction model.

6. A simple and easy-to-use incremental learning text classification method as claimed in claim 1, characterized in that: The clustering adopts kmeans or one of its variants.

7. A simple and easy-to-use incremental learning text classification method as claimed in claim 1, characterized in that: The typical samples are selected by selecting Top-K samples or setting a threshold.

8. A simple and easy-to-use incremental learning text classification method as claimed in claim 1, characterized in that: In the incremental training, the initial network parameters of the model can be the model that has been trained in the previous round, or the parameters of the pre-trained model that has not been fine-tuned.

9. A simple and easy-to-use incremental learning text classification system, characterized by: Using the incremental learning text classification method as described in any one of claims 1 to 8, comprising: The original classification data set acquisition module is used to collect the original classification data set; The text classification model building and training module is used to build and train the classification model. The trained model is represented as M n ; Typical old sample acquisition module, used to obtain typical samples from the original classification data set; A new category sample acquisition module is added to collect and annotate new category data, mix the annotated data with typical old sample data, and convert them into new MLM training samples; Incremental learning module, based on the new MLM training samples, the model M completed in the previous round of training n Continue incremental learning to obtain a new model M that can simultaneously predict new and old category data n+1 .

Images