A text continuous classification method based on an interpretable model

Abstract

The present invention relates to a text continuous classification method based on an interpretable model, including various techniques such as text classification, sustainable learning, interpretable learning, and experience replay, belonging to the field of natural processing technology. The present invention includes two major parts. The first part is a self-explaining text classification model, which inputs a sample and outputs the predicted category and interpretable words corresponding to the sample. The second part is a feature pool, which is used to store the interpretable features output by the self-explaining model as the old task features for replay. Here, the self-explaining text classification model is specifically a self-explaining model that integrates phrase structure knowledge, including three layers. The first layer is the input layer, which obtains the text representation through an encoder and further obtains the phrase structure knowledge corresponding to the input sample. The second layer is the interpretation layer where the phrase structure knowledge is further integrated. The third layer is the linear layer, which maps the output of the interpretation layer to the category vector space to obtain the predicted category label and the interpretation of the sample.

Description

Technical Field

[0001] The present invention relates to a method for continuous text classification, specifically to a method for continuous text classification based on an interpretable model, including multi-faceted technologies such as text classification, sustainable learning, interpretable learning, and experience replay, belonging to the field of natural processing technology. Background Art

[0002] At present, the technology of continuous learning has not been fully developed, and the continuous learning performance of many methods has not achieved the expected results. There is great room for research and development in this field, and it is also one of the important technologies to promote the wider application of various deep learning models in actual scenarios.

[0003] The task of text classification (TC) aims to label a given text with predefined tags to determine the classification category of the given text. This technology is one of the most important and basic tasks in natural language processing. At present, the technology of text classification has been very mature, and it performs very well under the supervision of a given fully labeled data set. However, similar to most deep learning-based tasks, most traditional models do not have the ability of continuous learning and are not interpretable.

[0004] The main goal of continuous text classification is to reduce the degree of forgetting of the old task knowledge by the classifier when learning new task data, so that the classifier performs well on both new and old tasks after sequentially learning new tasks. A typical continuous learning method is experience replay, that is, training the data of old tasks is interspersed during the training of new data. At present, the continuous learning methods based on experience replay replay the original samples of old tasks or the generative pseudo-data samples used to replace old samples. However, since the replayed data is small, how to make the model recall old knowledge through a small number of samples becomes the key to whether the model can continuously classify. Research shows that some currently proposed interpretable text classification models can output explanations while maintaining the classification accuracy of the model. This method finds the sample features that the model focuses on, and then improves the performance of the classifier on the data set through these features.

[0005] Therefore, the present application proposes a method for continuous text classification that integrates sample features. This method is used to optimize the performance of the experience replay method in continuous text classification. By integrating interpretable attributes into the model, the output interpretable features are used as important sample features and put into the storage memory module for subsequent model replay. It should be noted that what is replayed here is the sample features, rather than the complete original samples, so as to improve the classifier's mastery of the old task knowledge and enhance the continuous text classification method of the model. Summary of the Invention

[0006] The object of the present invention is to address the two major challenges of continuous learning and interpretability research in text classification, break the barriers between these two challenges, obtain an interpretable text continuous classification model, and at the same time use the interpretations output by the interpretable text classification as important features of the text to improve the effect of text continuous classification, and propose a text continuous classification method based on an interpretable model.

[0007] The innovation of the present invention lies in: in the text classification task, creatively combining interpretability and continuous learning to obtain a model that is both interpretable and can continuously classify, and at the same time the output interpretations are used as replay data to improve the effect of continuous classification.

[0008] The object of the present invention is achieved through the following technical solutions.

[0009] Define the task data sequential input sequence (D0, D1, …… D N ), set the experience replay frequency α, and initialize the feature pool to be empty. Let the current learning task be D i , divide the task data into m batches for training, and the frequency α means that in the current task, for every α batches of D i data, one batch of feature data is replayed.

[0010] A text continuous classification method based on an interpretable model includes the following steps:

[0011] Step 1: Input the text representation that fuses phrase structure knowledge, input by batch,

[0012] Let the current learning task be D i . If the current task is not the first task (i.e., i is not equal to 0), then on the premise that the feature pool is not empty and the current batch count is a multiple of α, randomly extract 1 batch of features from the feature pool for replay learning, that is, the data learned at this time is the features of the old task in the feature pool; otherwise, the input is the data of the current task.

[0013] Step 1-1: Obtain the text encoding.

[0014] Use an encoder to encode the input text into a text representation vector Word Embeddings.

[0015] Step 1-2: Obtain the phrase structure information of the input of the new task.

[0016] The phrase structure information is parsed into a phrase structure tree by using the encoded vector obtained in Step 2-1 through syntactic analysis (for example, the NLTK parser can be used to obtain the phrase structure tree corresponding to the sentence); if it is a replay sample, there is no need to obtain the phrase structure information, and it will be uniformly processed by the subtree matrix in Step 3 later.

[0017] Step 2: Extract interpretable features

[0018] It should be noted that only the current task data needs to extract interpretable features for subsequent storage in the data pool as the features for replaying this task when future tasks arrive.

[0019] Step 2-1: Fusion of phrase structure knowledge and text vectors

[0020] Further fuse the Word Embeddings of the input text obtained in Step 1 and the corresponding phrase structure tree knowledge.

[0021] It should be noted that if the input sample is a sample in the feature pool, the short sentence structure tree matrix is directly set as a matrix of all 1s, that is, the replayed is already a feature and there is no need to extract it again.

[0022] Step 2-2: Extract interpretable phrases

[0023] Calculate the interpretable coefficient and interpretable vector according to the fusion extraction result of Step 2-1.

[0024] Step 3: Output the explanation and predicted class label

[0025] Obtain the class label and explanation according to the interpretable vector; this explanation is used as the important feature data of this task and stored in the feature pool for the replay samples of subsequent continuous classification.

[0026] Step 4: Obtain new task data and continue with text continuous classification

[0027] When new task data is obtained, repeat Steps 1-3 to perform the process of continuous classification of the model.

[0028] Beneficial effects

[0029] Compared with existing methods, the present invention utilizes the advantage that an interpretable model can improve the classification effect while outputting an explanation, integrates it into a continuous text classification model based on experience replay, and obtains an interpretable and continuously learnable text classification model. It not only bridges the research of interpretable learning and continuous learning, but also makes them complement each other. It solves the needs of people for the interpretability and continuous learning of the model, and further improves the application of the text classification model in practice. Description of the drawings

[0030] Figure 1It is the overall architecture diagram of the method of the present invention;

[0031] Figure 2 It is the construction diagram of the self-explanatory text classification model;

[0032] Figure 3 It is the input layer of step 2 and the embodiment of the method of the present invention. Detailed implementation manners

[0033] The method of the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0034] Embodiment

[0035] As Figure 1 shown, the present invention includes two major parts. The first part is a self-explanatory text classification model, which inputs samples and outputs the predicted categories and interpretable words corresponding to the samples; the second part is a feature pool, which is used to store the interpretable features output by the self-explanatory model as the old task features for replay. Here, the self-explanatory text classification model is specifically a self-explanatory model that fuses phrase structure knowledge. As Figure 2 shown, it includes three layers. The first layer is the input layer, which obtains the text representation through the encoder and further obtains the phrase structure knowledge corresponding to the input samples. The second layer is the interpretation layer where the phrase structure knowledge is further fused, and the third layer is the linear layer, which maps the output of the interpretation layer to the category vector space to obtain the predicted category label and the interpretation of the sample.

[0036] Define the task data sequential input sequence (D0, D1, …… D N ), set the experience replay frequency α, and initialize the feature pool to be empty. Let the current learning task be D i , divide the task data into m batches for training. The frequency α means that in the current task, for every α batches of D i data, one batch of feature data is replayed.

[0037] Specifically, it includes the following steps:

[0038] Step 1: Input the text representation that fuses phrase structure knowledge, and input it batch by batch

[0039] Let the current learning task be D i . If the current task is not the first task (i.e., i is not equal to 0), then on the premise that the feature pool is not empty and the current batch count is a multiple of α, randomly extract 1 batch of features from the feature pool for replay learning, that is, the data learned at this time is the features of the old task in the feature pool; otherwise, the input is the data of the current task.

[0040] As Figure 3 shown, the model input includes a text vector representation and the corresponding phrase structure tree.

[0041] Step 1-1: Obtain text encoding.

[0042] Use an encoder to encode the input text into vector Word Embeddings. Use the pre-trained model Bert for text encoding.

[0043] Word Embeddings = Bert(w1, w2,..., w n ) (1)

[0044] Step 1-2: Obtain phrase structure information for the input of the new task.

[0045] The phrase structure information is parsed into a phrase structure tree by using a syntax analysis on the encoded vector obtained in Step 2-1 (for example, the NLTK parser can be used to obtain the phrase structure tree corresponding to the sentence); if it is a playback sample, there is no need to obtain phrase structure information, and it will be uniformly processed by the subtree matrix in Step 3 later.

[0046] Step 2: Extract interpretable features

[0047] It should be noted that only the current task data needs to extract interpretable features for subsequent addition to the data pool as the features for playing back this task when future tasks arrive.

[0048] Step 2-1: Fusion of phrase structure knowledge and text vector In Step 1, the WordEmbeddings of the input text and the corresponding phrase structure tree knowledge are obtained. Here, the phrase structure is further fused with the text vector:

[0049] h_sub = sub_matrix × Word Embeddings (2)

[0050] where sub_matrix is the subtree matrix of the corresponding phrase structure tree, which represents the phrase structure included in the input sample. For example, Figure 3 in the example sample "The girls cooks the food", the corresponding phrase structure matrix representation is:

[0051]

[0052] It should be noted that if the input sample is a sample in the feature pool, the phrase subtree matrix sub_matrix is directly set to a matrix of all 1s, that is, the playback is already a feature and there is no need to extract it again.

[0053] Step 2-2: Extract interpretation phrases

[0054] The interpretable coefficient μ and vector h_sub’ are obtained according to the fusion extraction result in step 2-1.

[0055] μ = Softmax(Linear(h_sub) - Linear(mask_matrix)) (4)

[0056] h_sub′ = μ × h_sub (5)

[0057] Here, mask_matrix is the phrase structure mask matrix corresponding to sub_matrix, that is, the elements with a value of 1 in sub_matrix are set to 0, and the elements with a value of 1 are set to 1e6. The mask_matrix corresponding to formula (2) can be expressed as:

[0058]

[0059] Step 3: Output the explanation and the predicted class label

[0060] The class label and the explanation are obtained based on the interpretable vector h_sub’; this explanation is used as important feature data for this task and is stored in the feature pool for the replay samples of subsequent continuous classification.

[0061] Step 4: Obtain new task data and continue with text continuous classification

[0062] When new task data is obtained, repeat steps 1-3 to perform the process of continuous model classification.

[0063] To illustrate the content and implementation method of the present invention, a specific embodiment is given in this specification. The purpose of introducing details in the embodiment is not to limit the scope of the claims, but to help understand the method described in the present invention. Those skilled in the art should understand that various modifications, changes or substitutions of the steps of the best embodiment are possible without departing from the spirit and scope of the present invention and its appended claims. Therefore, the present invention is not limited to the content disclosed in the best embodiment and the accompanying drawings.

Claims

1. A text continuous classification method based on an interpretable model, characterized in that It includes the following steps: Step 1: Input the text representation that integrates phrase structure knowledge in batches; Let the current learning task be D i If the current task is not the first task, that is, i is not equal to 0, then on the premise that the feature pool is not empty and the current batch count is a multiple of α, randomly extract 1 batch of features from the feature pool for replay learning, that is, the data for learning at this time is the features of the old task in the feature pool; otherwise, the input is the data of the current task; Define the task data sequential input sequence (D0, D1, …… D N ), set the experience replay frequency α, initialize the feature pool as empty, and set the current learning task as D i . Divide the task data into m batches for training. The frequency α indicates that in the current task, for every α batches of D i data, a batch of feature data is replayed; Step 1-1: Obtain the text encoding Use the encoder to encode the input text into a text representation vector, i.e., Word Embeddings; Step 1-2: Obtain the phrase structure information for the input of the new task The phrase structure information is obtained by parsing the encoding vector obtained in Step 1-1 into a phrase structure tree using a syntactic analysis, and the NLTK parser is used to obtain the phrase structure tree corresponding to the sentence; if it is a replay sample, there is no need to obtain the phrase structure information, and it will be uniformly processed by the subtree matrix in Step 3 later; Step 2: Extract interpretable features It should be noted that only the data of the current task needs to extract interpretable features for subsequent storage in the data pool as the features for replaying this task when future tasks arrive; Step 2-1: Integrate the phrase structure knowledge and the text vector Further integrate the Word Embeddings of the input text obtained in Step 1 and the corresponding phrase structure tree knowledge; Step 2-2: Extract interpretable phrases Calculate the interpretable coefficient and interpretable vector according to the integration extraction result in Step 2-1; Step 3: Output the interpretation and the predicted class label Obtain the class label and interpretation based on the interpretable vector; this interpretation is used as the important feature data of this task and is stored in the feature pool for subsequent replay samples for continuous classification; Step 4: Obtain new task data and continue with text continuous classification When new task data is obtained, repeat Steps 1 - 3 to perform the process of continuous classification of the model.

Images