Text classification model training method and device based on incremental learning

By adopting incremental learning method and autoencoder reconstruction technology in text classification model, the problem of forgetting text classification model in incremental learning is solved, and learning ability and anti-forgeting ability are improved. Text classification model update suitable for data security scenarios is improved. .

CN118568265BActive Publication Date: 2024-11-08HEFEI HIGH DIMENSIONAL DATA TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411052266.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-02
Publication Date
2024-11-08
Estimated Expiration
2044-08-02

AI Technical Summary

Technical Problem

There is a catastrophic forgetting problem in the incremental learning process of existing text classification models, and it is difficult to retain previous knowledge and learned classification rules. Especially in application scenarios that emphasize privacy and data security, real training data is not allowed to be stored. How to make It is a challenge to not forget previous knowledge when adapting to new data and new categories.

Method used

Using the text classification model training method based on incremental learning, the text data training base model is constructed, and the initial training text data is fine-tuned and trained using the preset language model, and the semantic features are reconstructed and updated with the autoencoder to ensure that the model is During incremental learning, the previous knowledge is not lost, and the text encoding library is updated through similarity calculation and training of the autoencoder to achieve adaptation to new data.

Benefits of technology

This method improves the learning ability and anti-forgeting ability of the model in incremental learning scenarios, reduces the training cost of model updates, and maintains the performance of the text classification model without storing data. It is suitable for paying attention to privacy and data security Application scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118568265B_ABST
    Figure CN118568265B_ABST
Patent Text Reader

Abstract

The present invention relates to a text classification model training method and device based on incremental learning, which is applied to the field of natural language processing technology. The training method of the text classification model includes: constructing a text data training base model; each time the incremental learning training is performed, the language model A2 is first trained, new labels are added, and the features corresponding to the new labels in the semantic space are obtained; then the language model A1 is trained; after the training of the language models A1 and A2 is completed, the text autoencoder AE is finally trained. The present invention does not need to save the initial text in the incremental learning scenario of the text classification model, and has better learning ability and anti-forgetting ability.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present application relate to the field of natural language processing technology, and in particular to a text classification model training method and device based on incremental learning. Background Art

[0002] After training, conventional text classification models can only achieve high accuracy in scenarios that match the distribution of training data. However, in actual application scenarios of text classification, the distribution of data to be classified and the categories to be classified often change constantly, and the original text classification model is no longer applicable. Therefore, it is necessary to adopt incremental learning methods to enable the model to adapt to classification tasks of new data and new categories. Incremental learning is a technology that allows the original machine learning model to gradually accept new samples and data for continuous iteration and training without retraining the entire model. At the same time, text classification models have application scenarios that attach importance to privacy and data security. For example, in the medical and financial fields, real training data is not allowed to be stored. Therefore, how to reduce the forgetting of previous knowledge during the incremental learning process of the model is a big challenge.

[0003] In traditional text classification methods, the text to be classified is input into a pre-trained language model to obtain text representation, and the probability distribution of the text belonging to different categories is obtained through the classification network. In the incremental learning process, since the probability distribution output by the classification network has a low amount of information, the new category data has knowledge that the classification network did not originally have, so the classification network performs poorly in adapting to the new category data.

[0004] At present, the academic community has conducted a lot of research on incremental learning methods, and has designed a variety of methods such as regularization, knowledge distillation, data playback, network parameter isolation, etc. Existing incremental learning methods do not fully utilize the characteristics of text data and text semantic feature extraction models, resulting in catastrophic forgetting problems in existing text classification models, that is, it is difficult to retain previously learned classification rules when learning new classification rules and it is difficult to learn new tasks.

[0005] In view of this, the present invention is proposed. Summary of the invention

[0006] The purpose of the present invention is to propose a text classification model training method and device based on incremental learning. The method is a text classification model fine-tuning method based on incremental learning. In the incremental learning scenario of the text classification model, there is no need to save the initial text, and it has better learning ability and anti-forgetting ability.

[0007] In a first aspect, an embodiment of the present application provides a text classification model training method based on incremental learning, comprising: S11. Constructing a text data training base model, comprising:

[0008] S11a obtains initial training text data D0, corresponding to the initial training label L0, preset language models A1 and A2, the language models A1 and A2 use the same initial pre-trained language model;

[0009] S11b. The initial training text data D0 is input into the language model A1 to obtain the semantic feature FD of the initial training text data D0, and the initial training label L0 is input into the language model A2 to obtain the semantic feature FL of the initial training label L0;

[0010] S11c. Fine-tune the language models A1 and A2, and calculate the similarity between the semantic feature FD of the initial training text data D0 and the semantic feature FL of the initial training label L0. The training goal is:

[0011]

[0012] Among them, d l is the label corresponding to the training text d, sim is the similarity calculation function, A1(d) represents the output of the language model A1 inputted by the initial training text data D0, and A2(d l ) represents the output of the label corresponding to the training text d obtained by the language model A2, and A2(l) represents the output of the initial training label L0 obtained by the language model A2; S11d. Traverse all the data in the initial training text data D0 and the initial training label L0, complete the training of the language models A1 and A2 according to the steps S11a-S11c, and then train the text autoencoder AE to obtain a trained encoder, including:

[0013] S11d1. Input all the data in the initial training text data D0 into the frozen parameter layer of the language model A1 to obtain the intermediate semantic features F of all the data in the initial training text data D0, input the intermediate semantic features F into the encoder of the autoencoder AE to continuously reduce the dimension to obtain the encoding result, and then input the encoding result into the decoder of the encoder AE to continuously increase the dimension to obtain the reconstructed intermediate semantic features F' of all the data in the initial training text data D0. The training goal of the autoencoder AE is:

[0014]

[0015] Wherein, KL represents KL divergence, which is used to measure the effect of reconstructing the intermediate semantic features F of all data in the initial training text data D0;

[0016] S11d2. For each category of all data in the initial training text data D0, select k samples with the smallest KL divergence of the reconstructed intermediate semantic features F' and the intermediate semantic features F, and save the results of encoding the k samples by the autoencoder AE as a text encoding library E;

[0017] S12. Each time incremental learning training is performed, the language model A2 is first trained, new labels are added, and features in the semantic space corresponding to the new labels are obtained, including:

[0018] S12a. The preset data set D that has been learned, the classification label L that has been learned, the data set ΔD that is newly added in this incremental learning, and the label ΔL that is newly added in this incremental learning that has not been learned;

[0019] S12b. Input the classification label L and the unlearned label ΔL into the language model A2 for training. The training goal is:

[0020]

[0021] Wherein, α is the training weight coefficient of the similarity of the classification label L, which is used to ensure that the classification label L does not change in the semantic space, A2' is the initial parameter of the language model A2 at the beginning of this incremental learning training, sim is the similarity calculation function, A2(l) represents the output of the classification label L obtained by the language model A2, A2(l 1 ) represents the output of the unlearned label ΔL obtained by the language model A2;

[0022] S13. Training the language model A1 again includes:

[0023] The autoencoder AE is used to decode the text encoding library E updated in the last incremental learning process of the text to obtain the reconstructed intermediate semantic features F of the learned data set D. D ', input the newly added data set ΔD into the frozen parameter layer of the language model A1, and obtain the intermediate semantic features F of the newly added data set ΔD ΔD ', train the language model A1, the training goal is:

[0024]

[0025] Among them, f l is the label of the text corresponding to the intermediate semantic feature f, sim is the similarity calculation function, A1(f) represents the output of the intermediate semantic feature f through the non-frozen parameter layer of the language model A1, A2(f l) represents the output of the label of the text corresponding to the intermediate semantic feature f obtained by the language model A2, and A2(l) represents the output of the classification label L and the unlearned label ΔL obtained by the language model A2;

[0026] S14. After completing the training of the language models A1 and A2, finally training the text autoencoder AE includes:

[0027] S14a. Reconstruct the intermediate semantic features F of the learned dataset D D ' and the intermediate semantic features F of the newly added dataset ΔD ΔD 'The encoder of the autoencoder AE is input to continuously reduce the dimension to obtain the encoding result, and then the encoding result is input to the decoder of the encoder AE to continuously increase the dimension to obtain the intermediate semantic feature F reconstructed from the learned data set D D ” and the reconstructed intermediate semantic features F of the newly added dataset ΔD ΔD ”, where the training objective of the autoencoder AE is:

[0028]

[0029] Among them, KL stands for KL divergence, which is used to measure the intermediate semantic features F of the learned dataset D D ' and the intermediate semantic features F of the newly added dataset ΔD ΔD 'The effect of reconstruction;

[0030] S14b. Select the intermediate semantic feature F for the learned dataset D and the newly added dataset ΔD D ', the intermediate semantic feature F ΔD ' and the intermediate semantic feature F D ", the intermediate semantic feature F ΔD "The k samples with the smallest KL divergence are encoded by the autoencoder AE, and the text encoding library E updated in the last incremental learning process of the text is updated again, and the semantic features of the learned data set D and the newly added data set ΔD are reconstructed for updating the text database in the subsequent incremental learning process.

[0031] Furthermore, α is the number of new labels in the newly added unlearned labels ΔL in this incremental learning.

[0032] Furthermore, the sim is any one of a cosine similarity function, a Euclidean distance or mutual information.

[0033] Furthermore, the encoder and decoder of the autoencoder AE are both multi-layer transformer decoder and multi-layer transformer encoder structures, and the dimension of the intermediate layer feature space is smaller than that of the input and output.

[0034] Furthermore, the newly added data set ΔD in this incremental learning includes new category data and new data of old categories.

[0035] Furthermore, when the text classification model performs the first incremental learning training, the already learned data set D is the initial training text data D0, and the already learned classification label L is the initial training label L0.

[0036] Furthermore, when the text classification model is subjected to the first incremental learning training, the text encoding library E updated in the previous incremental learning process is the text encoding library E in S11.

[0037] In a second aspect, an embodiment of the present application provides a text classification method, which adopts the text classification model training method based on incremental learning as described in the first aspect, including:

[0038] S21. Preset text data d to be classified, all classification label data sets L, language models A1 and A2, wherein the all classification label data sets L, the language models A1 and A2 have been trained by the text classification model training method based on incremental learning as described in the first aspect;

[0039] S22. Input the classification label data set L into the language model A2, obtain and save the semantic feature vector EL corresponding to the classification label data set L;

[0040] S23. Input the text data d to be classified into the preset language model A1 to obtain the semantic feature vector e corresponding to the text data d to be classified;

[0041] S24. Calculate the similarity sim(e, el) between the semantic feature vector e and all vectors el in the semantic feature vector EL, where el∈EL, and the sim is calculated in the same manner as in S13;

[0042] S25. Sort the similarities sim(e, el) between the semantic feature vector e and all vectors el in the semantic feature vector EL by similarity, and select the label corresponding to the vector with the maximum similarity as the text classification result of the text data d to be classified.

[0043] In a third aspect, an embodiment of the present application provides a text classification model training device, which adopts the text classification model training method based on incremental learning as described in the first aspect, including:

[0044] D31. Text data training base model module, which serves as the basis for the subsequent incremental learning process;

[0045] D32 adds a new label module for training the language model A2 to obtain semantic features corresponding to the newly added unlearned label ΔL in this incremental learning;

[0046] D33. A language model fine-tuning module, which is used to train the language model A1 to obtain semantic features corresponding to the newly added data set ΔD in this incremental learning;

[0047] D34. The autoencoder update module, after completing the training of the language models A1 and A2, trains the text autoencoder AE, updates the text encoding library E updated during the last incremental learning process of the text, and saves the semantic features corresponding to the learned data set D and the newly added data set ΔD.

[0048] In a fourth aspect, an embodiment of the present application provides a text classification device, which adopts the text classification method as described in the second aspect, including:

[0049] D41. Data acquisition module, obtains the preset full classification label data set L, the text data to be classified d, and the language models A1 and A2;

[0050] D42. Classification label semantic feature vector acquisition module, obtains the semantic feature vector EL corresponding to the classification label data set L; D43. To-be-classified text semantic feature vector acquisition module, obtains the semantic feature vector e corresponding to the to-be-classified text data d; D44. Semantic feature vector similarity calculation module, calculates the similarity sim(e, el) between the semantic feature vector e and all vectors el in the semantic feature vector EL;

[0051] D45. A text classification result acquisition module selects the label corresponding to the maximum similarity vector in the similarities sim(e, el) between the semantic feature vector e and all vectors el in the semantic feature vector EL as the text classification result of the text data d to be classified.

[0052] In a fifth aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement the training method of the text classification model described in any of the above embodiments, or to implement the text classification method described in any of the above embodiments.

[0053] In a sixth aspect, an embodiment of the present application provides a computer program product or a computer program, the computer program product or the computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the training method of the text classification model described in any of the above embodiments, or implements the text classification method described in any of the above embodiments.

[0054] The text classification model training method and device based on incremental learning provided by the present invention can at least bring the following

[0055] Beneficial effects:

[0056] 1. The present invention uses a text semantic feature matching scheme and an autoencoder to reconstruct the training data. There is no need to repeatedly train the trained data. Only new data containing new categories and new data of old categories are incrementally trained, thereby reducing the training cost of model updating.

[0057] 2. The present invention solves the problem of inconvenience in storing data in application scenarios that emphasize privacy and data security, and has better learning ability and anti-forgetting ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The accompanying drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present invention, and together with the specification are used to explain the principles of the present invention. Obviously, the accompanying drawings described below are only some embodiments of the present invention, and for those of ordinary skill in the art, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0059] Figure 1 A schematic diagram showing the principles of the text data training base model training method provided in an embodiment of the present application is shown.

[0060] Figure 2 A schematic diagram showing the principles of the incremental learning and training method for text data provided in an embodiment of the present application is shown.

[0061] Figure 3 A flowchart of a text classification method provided in an embodiment of the present application is shown.

[0062] Figure 4 A schematic diagram of a text classification model training device based on incremental learning provided in an embodiment of the present application is shown.

[0063] Figure 5 The structural block diagram of the text classification device provided in the embodiment of the present application is shown.

[0064] Explanation of the accompanying drawings: text data training base model module 31; adding new label module 32; language model fine-tuning module 33; autoencoder update module 34; data acquisition module 41; classification label semantic feature vector acquisition module 42; to-be-classified text semantic feature vector acquisition module 43; semantic feature vector similarity calculation module 44; text classification result acquisition module 45. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0066] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings, and "multiple" generally includes at least two.

[0067] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0068] As used herein, the words "if" and "if" may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0069] It should also be noted that the term "includes", "comprising" or any other variation thereof is intended to cover non-exclusive inclusion, so that a commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprising a ..." do not exclude the existence of other identical elements in the commodity or device including the elements.

[0070] The optional embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0071] First, as Figure 1 , Figure 2 As shown, the embodiment of the present application provides a text classification model training method based on incremental learning, including:

[0072] S11. Build a text data training base model, including:

[0073] S11a obtains initial training text data D0, corresponding to the initial training label L0, preset language models A1 and A2, the language models A1 and A2 use the same initial pre-trained language model;

[0074] S11b. The initial training text data D0 is input into the language model A1 to obtain the semantic feature FD of the initial training text data D0, and the initial training label L0 is input into the language model A2 to obtain the semantic feature FL of the initial training label L0;

[0075] S11c. Fine-tune the language models A1 and A2, and calculate the similarity between the semantic feature FD of the initial training text data D0 and the semantic feature FL of the initial training label L0. The training goal is:

[0076]

[0077] Among them, d l is the label corresponding to the training text d, sim is the similarity calculation function, A1(d) represents the output of the language model A1 inputted by the initial training text data D0, and A2(d l ) represents the output of the label corresponding to the training text d obtained by the language model A2, and A2(l) represents the output of the initial training label L0 obtained by the language model A2; S11d. Traverse all the data in the initial training text data D0 and the initial training label L0, complete the training of the language models A1 and A2 according to the steps S11a-S11c, and then train the text autoencoder AE to obtain a trained encoder, including:

[0078] S11d1. Input all the data in the initial training text data D0 into the frozen parameter layer of the language model A1 to obtain the intermediate semantic features F of all the data in the initial training text data D0, input the intermediate semantic features F into the encoder of the autoencoder AE to continuously reduce the dimension to obtain the encoding result, and then input the encoding result into the decoder of the encoder AE to continuously increase the dimension to obtain the reconstructed intermediate semantic features F' of all the data in the initial training text data D0. The training goal of the autoencoder AE is:

[0079]

[0080] Wherein, KL represents KL divergence, which is used to measure the effect of reconstructing the intermediate semantic features F of all data in the initial training text data D0;

[0081] S11d2. For each category of all data in the initial training text data D0, select k samples with the smallest KL divergence of the reconstructed intermediate semantic features F' and the intermediate semantic features F, and save the results of encoding the k samples by the autoencoder AE as a text encoding library E;

[0082] S12. Each time incremental learning training is performed, the language model A2 is first trained, new labels are added, and features in the semantic space corresponding to the new labels are obtained, including:

[0083] S12a. The preset data set D that has been learned, the classification label L that has been learned, the data set ΔD that is newly added in this incremental learning, and the label ΔL that is newly added in this incremental learning that has not been learned;

[0084] S12b. Input the classification label L and the unlearned label ΔL into the language model A2 for training. The training goal is:

[0085]

[0086] Wherein, α is the training weight coefficient of the similarity of the classification label L, which is used to ensure that the classification label L does not change in the semantic space, A2' is the initial parameter of the language model A2 at the beginning of this incremental learning training, sim is the similarity calculation function, A2(l) represents the output of the classification label L obtained by the language model A2, A2(l 1 ) represents the output of the unlearned label ΔL obtained by the language model A2;

[0087] S13. Training the language model A1 again includes:

[0088] The autoencoder AE is used to decode the text encoding library E updated in the last incremental learning process of the text to obtain the reconstructed intermediate semantic features F of the learned data set D. D ', input the newly added data set ΔD into the frozen parameter layer of the language model A1, and obtain the intermediate semantic features F of the newly added data set ΔD ΔD ', train the language model A1, the training goal is:

[0089]

[0090] Among them, f lis the label of the text corresponding to the intermediate semantic feature f, sim is the similarity calculation function, A1(f) represents the output of the intermediate semantic feature f through the non-frozen parameter layer of the language model A1, A2(f l ) represents the output of the label of the text corresponding to the intermediate semantic feature f obtained by the language model A2, and A2(l) represents the output of the classification label L and the unlearned label ΔL obtained by the language model A2;

[0091] S14. After completing the training of the language models A1 and A2, finally training the text autoencoder AE includes:

[0092] S14a. Reconstruct the intermediate semantic features F of the learned dataset D D ' and the intermediate semantic features F of the newly added dataset ΔD ΔD 'The encoder of the autoencoder AE is input to continuously reduce the dimension to obtain the encoding result, and then the encoding result is input to the decoder of the encoder AE to continuously increase the dimension to obtain the intermediate semantic feature F reconstructed from the learned data set D D ” and the reconstructed intermediate semantic features F of the newly added dataset ΔD ΔD ”, where the training objective of the autoencoder AE is:

[0093]

[0094] Among them, KL stands for KL divergence, which is used to measure the intermediate semantic features F of the learned dataset D D ' and the intermediate semantic features F of the newly added dataset ΔD ΔD 'The effect of reconstruction;

[0095] S14b. Select the intermediate semantic feature F for the learned dataset D and the newly added dataset ΔD D ', the intermediate semantic feature F ΔD ' and the intermediate semantic feature F D ", the intermediate semantic feature F ΔD "The k samples with the smallest KL divergence are encoded by the autoencoder AE, and the text encoding library E updated in the last incremental learning process of the text is updated again, and the semantic features of the learned data set D and the newly added data set ΔD are reconstructed for updating the text database in the subsequent incremental learning process.

[0096] Among them, the preset language models A1 and A2 in S11a are language models that have been pre-trained on extremely large-scale corpus data, which is a commonly used technology in natural processing technology, and the language models A1 and A2 use the same initial pre-trained language model to ensure the semantic space alignment of text and labels.

[0097] When fine-tuning the language models A1 and A2 in S11c, the training process maximizes the semantic similarity between the training text data and the correct labels, and minimizes the semantic similarity between the training text data and the incorrect labels.

[0098] During each incremental learning training in S12-S14, the language model A1 will use the text encoding library E updated and saved during the last incremental learning training to reconstruct the semantic features of the data set that has been learned, and then train a new autoencoder based on the newly added data set ΔD in this incremental learning process, and update the text encoding library E updated in the last incremental learning process of the text again to generate a new semantic feature text encoding library E. During the next incremental learning training, the updated new semantic feature text encoding library E will be used to reconstruct the semantic features of the data set that has been learned again.

[0099] Furthermore, α is the number of new labels in the newly added unlearned labels ΔL in this incremental learning.

[0100] Furthermore, the sim is any one of a cosine similarity function, a Euclidean distance or mutual information.

[0101] Furthermore, the encoder and decoder of the autoencoder AE are both multi-layer transformer decoder and multi-layer transformer encoder structures, and the dimension of the intermediate layer feature space is smaller than that of the input and output.

[0102] Furthermore, the newly added data set ΔD in this incremental learning includes new category data and new data of old categories.

[0103] Furthermore, when the text classification model performs the first incremental learning training, the already learned data set D is the initial training text data D0, and the already learned classification label L is the initial training label L0.

[0104] Furthermore, when the text classification model performs the first incremental learning training, the text encoding library E updated in the previous incremental learning process is the text encoding library E in S11.

[0105] Wherein, when the text classification model performs incremental learning training, the learned data set D=D0+ΔD1+ΔD2+......+ΔD(k-1) during the kth incremental learning training, and the learned classification label L=L0+ΔL1+ΔL2+......+ΔL(k-1).

[0106] In the embodiment of the present application, the training process of the language models A1 and A2 always freezes the shallow parameters of the model and does not update them, so that the model training process is more focused on extracting deep semantic features. The function of the frozen parameter layer of the language model is to obtain the intermediate semantic features of the input text data, and the function of the non-frozen parameter layer is to convert the intermediate semantic features into semantic features that can be used for classification matching.

[0107] In addition, this application uses a text semantic feature matching scheme. In the process of incremental learning of new categories, the text semantic features have a high dimension and contain a large amount of information, which can better adapt to the learning of new categories.

[0108] In addition, the present application also updates and reconstructs the text encoding library through the autoencoder training data. First, the text data is vectorized, that is, the text data is passed through the embedding layer of the pre-trained language model to obtain text vector data; then the intermediate semantic features of the text data are continuously reduced and increased in dimension through the autoencoder to obtain the intermediate semantic features of the reconstructed text data. Finally, by retaining the encoding results corresponding to the samples with the smallest reconstruction error after encoding by the autoencoder and the autoencoder model, a large number of features of the original text data can be saved for application in subsequent text classification incremental learning.

[0109] Second, as Figure 3 As shown, the embodiment of the present application provides a text classification method, which adopts the text classification model training method based on incremental learning as described in the first aspect, including:

[0110] S21. Preset text data d to be classified, all classification label data sets L, language models A1 and A2, wherein the all classification label data sets L, the language models A1 and A2 have been trained by the text classification model training method based on incremental learning as described in the first aspect;

[0111] S22. Input the classification label data set L into the language model A2, obtain and save the semantic feature vector EL corresponding to the classification label data set L;

[0112] S23. Input the text data d to be classified into the preset language model A1 to obtain the semantic feature vector e corresponding to the text data d to be classified;

[0113] S24. Calculate the similarity sim(e, el) between the semantic feature vector e and all vectors el in the semantic feature vector EL, where el∈EL, and the sim is calculated in the same manner as in S13;

[0114] S25. Sort the similarities sim(e, el) between the semantic feature vector e and all vectors el in the semantic feature vector EL by similarity, and select the label corresponding to the vector with the maximum similarity as the text classification result of the text data d to be classified.

[0115] Thirdly, Figure 4 As shown, the embodiment of the present application provides a text classification model training device, which adopts the text classification model training method based on incremental learning as described in the first aspect, including:

[0116] D31. Text data training base model module, which serves as the basis for the subsequent incremental learning process;

[0117] D32. Add a new label module to train the language model A2 to obtain the semantic features corresponding to the newly added unlearned label ΔL in this incremental learning;

[0118] D33. A language model fine-tuning module, which is used to train the language model A1 to obtain semantic features corresponding to the newly added data set ΔD in this incremental learning;

[0119] D34. The autoencoder update module, after completing the training of the language models A1 and A2, trains the text autoencoder AE, updates the text encoding library E updated during the last incremental learning process of the text, and saves the semantic features corresponding to the learned data set D and the newly added data set ΔD.

[0120] Fourthly, Figure 5 As shown, the embodiment of the present application provides a text classification device, which adopts the text classification method as described in the second aspect, including:

[0121] D41. Data acquisition module, obtains the preset full classification label data set L, the text data to be classified d, and the language models A1 and A2;

[0122] D42. Classification label semantic feature vector acquisition module, obtains the semantic feature vector EL corresponding to the classification label data set L; D43. To-be-classified text semantic feature vector acquisition module, obtains the semantic feature vector e corresponding to the to-be-classified text data d; D44. Semantic feature vector similarity calculation module, calculates the similarity sim(e, el) between the semantic feature vector e and all vectors el in the semantic feature vector EL;

[0123] D45. A text classification result acquisition module selects the label corresponding to the maximum similarity vector in the similarities sim(e, el) between the semantic feature vector e and all vectors el in the semantic feature vector EL as the text classification result of the text data d to be classified.

[0124] In a fifth aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement the training method of the text classification model described in any of the above embodiments, or to implement the text classification method described in any of the above embodiments.

[0125] In a sixth aspect, an embodiment of the present application provides a computer program product or a computer program, the computer program product or the computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the training method of the text classification model described in any of the above embodiments, or implements the text classification method described in any of the above embodiments.

[0126] Specifically, a system or device equipped with a storage medium can be provided, on which software program code that implements the functions of any of the above-mentioned embodiments is stored, and a computer (or CPU or MPU) of the system or device can be enabled to read and execute the program code stored in the storage medium.

[0127] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute a part of the present invention.

[0128] The storage medium embodiments for providing the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RYM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program code can be downloaded from a server computer via a communication network.

[0129] In addition, it should be clear that the functions of any of the above embodiments can be implemented not only by executing the program code read by the computer, but also by enabling an operating system operating on the computer to complete part or all of the actual operations based on instructions from the program code.

[0130] In addition, it can be understood that the program code read from the storage medium is written to a memory provided in an expansion board inserted into the computer or written to a memory provided in an expansion unit connected to the computer, and then based on the instructions of the program code, a CPU installed on the expansion board or the expansion unit is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above-mentioned embodiments.

[0131] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A text classification model training method based on incremental learning, It is characterized in that include: S11. Build a text data training base model, including: S11a obtains initial training text data D0, corresponding to the initial training label L0, preset language models A1 and A2, the language models A1 and A2 use the same initial pre-trained language model; S11b. The initial training text data D0 is input into the language model A1 to obtain the semantic feature FD of the initial training text data D0, and the initial training label L0 is input into the language model A2 to obtain the semantic feature FL of the initial training label L0; S11c. Fine-tune the language models A1 and A2, and calculate the similarity between the semantic feature FD of the initial training text data D0 and the semantic feature FL of the initial training label L0. The training goal is: Among them, d l is the label corresponding to the training text d, sim is the similarity calculation function, A1(d) represents the output of the language model A1 inputted by the initial training text data D0, and A2(d l ) represents the output of the label corresponding to the training text d obtained by the language model A2, and A2(l) represents the output of the initial training label L0 obtained by the language model A2; S11d. After traversing all the data in the initial training text data D0 and the initial training label L0, the language models A1 and A2 are trained according to the steps S11a-S11c, and then the text autoencoder AE is trained to obtain a trained encoder, including: S11d1. Input all the data in the initial training text data D0 into the frozen parameter layer of the language model A1 to obtain the intermediate semantic features F of all the data in the initial training text data D0, input the intermediate semantic features F into the encoder of the autoencoder AE to continuously reduce the dimension to obtain the encoding result, and then input the encoding result into the decoder of the encoder AE to continuously increase the dimension to obtain the reconstructed intermediate semantic features F' of all the data in the initial training text data D0. The training goal of the autoencoder AE is: Wherein, KL represents KL divergence, which is used to measure the effect of reconstructing the intermediate semantic features F of all data in the initial training text data D0; S11d2. For each category of all data in the initial training text data D0, select k samples with the smallest KL divergence of the reconstructed intermediate semantic features F' and the intermediate semantic features F, and save the results of encoding the k samples by the autoencoder AE as a text encoding library E; S12. Each time incremental learning training is performed, the language model A2 is first trained, new labels are added, and features in the semantic space corresponding to the new labels are obtained, including: S12a. The preset data set D that has been learned, the classification label L that has been learned, the data set ΔD that is newly added in this incremental learning, and the label ΔL that is newly added in this incremental learning that has not been learned; S12b. Input the classification label L and the unlearned label ΔL into the language model A2 for training. The training goal is: Wherein, α is the training weight coefficient of the similarity of the classification label L, which is used to ensure that the classification label L does not change in the semantic space, A2' is the initial parameter of the language model A2 at the beginning of this incremental learning training, sim is the similarity calculation function, A2(l) represents the output of the classification label L obtained by the language model A2, A2(l 1 ) represents the output of the unlearned label ΔL obtained by the language model A2; S13. Training the language model A1 again includes: The autoencoder AE is used to decode the text encoding library E updated in the last incremental learning process of the text to obtain the reconstructed intermediate semantic features F of the learned data set D. D ', input the newly added data set ΔD into the frozen parameter layer of the language model A1, and obtain the intermediate semantic features F of the newly added data set ΔD ΔD ', train the language model A1, the training goal is: Among them, f l is the label of the text corresponding to the intermediate semantic feature f, sim is the similarity calculation function, A1(f) represents the output of the intermediate semantic feature f through the non-frozen parameter layer of the language model A1, A2(f l ) represents the output of the label of the text corresponding to the intermediate semantic feature f obtained by the language model A2, and A2(l) represents the output of the classification label L and the unlearned label ΔL obtained by the language model A2; S14. After completing the training of the language models A1 and A2, finally training the text autoencoder AE includes: S14a. Reconstruct the intermediate semantic features F of the learned dataset D D ' and the intermediate semantic features F of the newly added dataset ΔD ΔD 'The encoder of the autoencoder AE is input to continuously reduce the dimension to obtain the encoding result, and then the encoding result is input to the decoder of the encoder AE to continuously increase the dimension to obtain the intermediate semantic feature F reconstructed from the learned data set D D ” and the reconstructed intermediate semantic features F of the newly added dataset ΔD ΔD ”, where the training objective of the autoencoder AE is: Among them, KL stands for KL divergence, which is used to measure the intermediate semantic features F of the learned dataset D D ' and the intermediate semantic features F of the newly added dataset ΔD ΔD 'The effect of reconstruction; S14b. Select the intermediate semantic feature F for the learned dataset D and the newly added dataset ΔD D ', the intermediate semantic feature F ΔD ' and the intermediate semantic feature F D ", the intermediate semantic feature F ΔD "The k samples with the smallest KL divergence are encoded by the autoencoder AE, and the text encoding library E updated in the last incremental learning process of the text is updated again, and the semantic features of the learned data set D and the newly added data set ΔD are reconstructed for updating the text database in the subsequent incremental learning process.

2. A text classification model training method based on incremental learning as claimed in claim 1, It is characterized in that The α in S12 is the number of new labels in the labels ΔL that have not been learned in this incremental learning.

3. A text classification model training method based on incremental learning as claimed in claim 1, It is characterized in that The sim is any one of cosine similarity function, Euclidean distance or mutual information.

4. A text classification model training method based on incremental learning as claimed in claim 1, It is characterized in that The encoder and decoder of the autoencoder AE are both multi-layer transformer decoder and multi-layer transformer encoder structures, and the dimension of the intermediate layer feature space is smaller than the input and output.

5. A text classification model training method based on incremental learning as claimed in claim 1, It is characterized in that The newly added data set ΔD in this incremental learning includes new category data and new data of old categories.

6. A text classification model training method based on incremental learning as claimed in claim 1, It is characterized in that When the text classification model performs the first incremental learning training, the learned data set D is the initial training text data D0, and the learned classification label L is the initial training label L0.

7. A text classification model training method based on incremental learning as claimed in claim 1, It is characterized in that When the text classification model performs the first incremental learning training, the text encoding library E updated in the previous incremental learning process is the text encoding library E in S11.

8. A text classification method, using the text classification model training method based on incremental learning as claimed in claim 1, include: S21. Preset text data d to be classified, all classification label data sets L, language models A1 and A2, wherein the all classification label data sets L, the language models A1 and A2 have been trained by the text classification model training method based on incremental learning as described in claim 1; S22. Input the classification label data set L into the language model A2, obtain and save the semantic feature vector EL corresponding to the classification label data set L; S23. Input the text data d to be classified into the preset language model A1 to obtain the semantic feature vector e corresponding to the text data d to be classified; S24. Calculate the similarity sim(e, el) between the semantic feature vector e and all vectors el in the semantic feature vector EL, where el∈EL, and the sim is calculated in the same manner as in S13; S25. Sort the similarities sim(e, el) between the semantic feature vector e and all vectors el in the semantic feature vector EL by similarity, and select the label corresponding to the vector with the maximum similarity as the text classification result of the text data d to be classified.

9. A text classification model training device, using the text classification model training method based on incremental learning as described in any one of claims 1 to 7, include: D31. Text data training base model module, which serves as the basis for the subsequent incremental learning process; D32. Add a new label module to train the language model A2 to obtain the semantic features corresponding to the newly added unlearned label ΔL in this incremental learning; D33. A language model fine-tuning module, which is used to train the language model A1 to obtain semantic features corresponding to the newly added data set ΔD in this incremental learning; D34. The autoencoder update module, after completing the training of the language models A1 and A2, trains the text autoencoder AE, updates the text encoding library E updated in the last incremental learning process of the text, and saves the semantic features corresponding to the learned data set D and the newly added data set ΔD.

10. A text classification device, using the text classification method according to claim 8, include: D41. Data acquisition module, obtains the preset full classification label data set L, the text data to be classified d, and the language models A1 and A2; D42. A classification label semantic feature vector acquisition module, which acquires the semantic feature vector EL corresponding to the classification label data set L; D43. A module for acquiring semantic feature vectors of text to be classified, which acquires the semantic feature vector e corresponding to the text data d to be classified; D44. A semantic feature vector similarity calculation module, which calculates the similarity sim(e, el) between the semantic feature vector e and all vectors el in the semantic feature vector EL; D45. A text classification result acquisition module selects the label corresponding to the maximum similarity vector in the similarities sim(e, el) between the semantic feature vector e and all vectors el in the semantic feature vector EL as the text classification result of the text data d to be classified.