A cascaded label classification method based on multi-task learning knowledge enhancement

Abstract

The application discloses a kind of cascade label classification methods based on multi-task learning knowledge enhancement, belong to natural language processing technical field, including the following steps: S1: semantic representation;S2: multi-task classification.The context semantic representation and semantic representation of the present application are fused to the knowledge representation obtained by pre-training model, and the method of multi-task learning is added simultaneously so that the model can be iteratively trained, learn the relationship between different labels, improve the accuracy of general pre-training language model in operator complaint work order classification task.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing, and particularly relates to a cascade label classification method based on multi-task learning knowledge enhancement. BACKGROUND

[0002] There are rich business text or voice data in the field of operators, and operator telephone complaints are a key service affecting customer satisfaction, which widely serves users in all parts of the country. In previous actual work, the classification labels of each complaint work order are generally manually annotated by customer service personnel, and multi-level annotation is required, each level representing different meanings. The "cascade label classification" is to use a model to classify the text in cascade, and the cascade labels have a correlation or causal relationship, and the model needs to understand both the text content and the correlation between the cascade labels.

[0003] The traditional cascade label classification usually adopts a multi-label classification method, that is, it is classified as a single label at the same time, without considering the semantic relationship between the labels. Therefore, a cascade label classification method based on multi-task learning knowledge enhancement is proposed. SUMMARY

[0004] The technical problem to be solved by the present application is how to solve the problem of low accuracy of the current operator complaint work order text classification model, which cannot well complete the cascade label classification task. A cascade label classification method based on multi-task learning knowledge enhancement is provided, which makes the model learn the relationship information between the labels through multi-task learning, deepens the understanding of the complaint work order text through knowledge enhancement, and improves the accuracy of the complaint work order cascade label classification.

[0005] As shown in Figure 1 The present application solves the above technical problems by the following technical solutions. The present application comprises the following steps:

[0006] S1: semantic representation

[0007] The complaint work order text is represented by a context encoder and a knowledge enhancement encoder from the perspectives of context and semantic knowledge, respectively, to obtain a semantic representation vector of the complaint work order text.

[0008] S2: multi-task classification

[0009] Through four classifiers corresponding to different tasks in the multi-task learning classifier, different levels of classification labels are obtained according to the shared semantic representation obtained by the iterative training of the encoders of different classifiers.

[0010] Further, in the step S1, the context encoder is a pre-trained model preBert, and the pre-trained model preBert is used to train the complaint work order data to obtain a context-based semantic representation. For a complaint work order text sequence t, a context semantic representation vector of the text sequence t is obtained by encoding the pre-trained model preBert. The definitions are as follows:

[0011]

[0012] Wherein, preBert(·) represents a vector encoder based on a BERT model.

[0013] Further, in the step S1, the knowledge enhanced encoder includes a semantic primitive-based knowledge learning module and a sequence-based knowledge encoding module. The semantic primitive-based knowledge learning module is used to learn a word representation of each word based on a semantic primitive level, and then the sequence-based knowledge encoding module is used to aggregate and obtain a knowledge enhanced sentence representation.

[0014] Further, the specific processing process of the semantic primitive-based knowledge learning module is as follows:

[0015] S101: Knowledge acquisition

[0016] Count all the words in the text sequence to construct a vocabulary V, which includes N words, and the i-th word is represented as w i By querying the Chinese knowledge base Hownet, the i-th word w i has p semantic primitives The j-th semantic primitive s j has q semantic primitives

[0017] S102: Semantic primitive representation acquisition

[0018] The semantic primitive is encoded based on the pre-trained model preBert to obtain the corresponding semantic primitive representation vector, which is respectively

[0019] S103: Semantic representation acquisition

[0020] Suppose represents the j-th semantic primitive of the word w i is obtained by combining all semantic primitive embedding vectors of the semantic primitive, and is defined as follows:

[0021]

[0022] ​

[0023] wherein, denotes the word meaning s j the set of all primitives, denotes the importance weight of the jth word meaning s i in the semantic representation of the word w ; denotes the semantic representation of the word w i based on the context vector representation obtained by pre-training the model preBert, is the semantic representation vector of the kth primitive of the jth word meaning s i in the word w ;

[0024] S104: Knowledge-enhanced word representation acquisition

[0025] The i-th word-based semantic representation of knowledge is obtained by integrating the representation of each word meaning of the word; The i-th word-based semantic representation of knowledge is defined as follows:

[0026]

[0027]

[0028] wherein, denotes the set of word meanings s i of the word w denotes the importance weight of the jth word meaning s i in the semantic representation of the word w , and finally the knowledge-enhanced semantic representation of each word is obtained by weighted combination based on two-layer attention of primitives-word meanings;

[0029] S105: Constructing a knowledge representation vector matrix

[0030] Perform steps S101-S104 on all words in the vocabulary V to obtain the knowledge-enhanced semantic representation of each word in the vocabulary V, and construct a word vector matrix M, wherein M∈R N×D , N is the size of the dictionary, and D is the dimension of each word vector.

[0031] Further, the specific processing process of the sequence-based knowledge encoding module is as follows:

[0032] S201: Obtain the semantic representation of the text sequence t by aggregating the knowledge-enhanced semantic representation of the word;

[0033] S202: Convert the sentence embedding vector into the same form as the representation vector output by the context encoder through a pooling layer, defined as:

[0034]

[0035] where Pooler(·) represents a pooling function.

[0036] Further, in the step S201, the text sequence t contains n words, and the word vectors corresponding to the n words in the sequence t are mapped based on the word vector matrix M to obtain n , and the aggregated word vectors obtain the semantic vector of the enhanced knowledge representation of the text sequence t is defined as follows:

[0037]

[0038] Further, in the step S2, the input of the multi-task learning classifier is the embedded vector fusion obtained after the operator complaint work order text data is input into the context encoder and the knowledge enhanced encoder, and is defined as follows:

[0039]

[0040] where t represents the operator complaint work order text input.

[0041] Further, in the step S2, each classifier is a fully connected layer, and the predicted label l max is obtained by taking the maximum probability value through the Argmax function.

[0042] logits=Classifier z (r(t))

[0043] l max =Argmax(logits)

[0044] where Classifier z (·) represents the zth classifier, and Argmax(·) represents a function of taking the maximum probability.

[0045] Further, in the step S2, the classification labels of different levels include first to fourth levels, wherein the first level represents the coarse classification of the business to which the complaint belongs, the second level label represents the fine classification of the business, the third level represents the specific problem category of the complaint, and the fourth level represents the specific problem fine classification of the complaint.

[0046] Compared with the prior art, the cascade label classification method based on multi-task learning knowledge enhancement has the following advantages: the context semantic representation and the semantic origin knowledge representation obtained by the pre-training model are fused, and the multi-task learning method is added, so that the model can be iteratively trained, the relationship between different labels is learned, and the accuracy of the general pre-training language model in the operator complaint work order classification task is improved. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 Figure 1 is a flowchart of the cascade label classification model based on multi-task learning knowledge enhancement in the embodiment of the present application. DETAILED DESCRIPTION

[0048] The embodiments of the present application will be described in detail below. The embodiments are implemented on the premise of the technical solutions of the present application, and detailed implementation methods and specific operation processes are given, but the protection scope of the present application is not limited to the following embodiments.

[0049] Embodiment one

[0050] The embodiment provides a technical solution: a cascade label classification method based on multi-task learning knowledge enhancement, which proposes a cascade label classification model based on multi-task learning knowledge enhancement, specifically including the following contents:

[0051] In this paper, according to the basic characteristics of the operator field complaint work order classification data, the manual customer service is divided into the complaint work order label, as shown in Table 1, the first level represents the general classification of the complaint business (coarse classification), the second level label represents the fine classification of the business, the third level represents the specific problem category of the complaint, and the fourth level represents the specific problem fine classification of the complaint. The present application encodes the complaint work order text through two encoders (context encoder, knowledge enhancement encoder) to obtain the semantic representation vector of the complaint work order text from the perspective of context and the perspective of semantic origin knowledge. Then connect the multi-task module, through four different classifiers, according to the shared semantic representation generated by the iteratively trained encoder of different classifiers, to obtain different levels of classification labels.

[0052] Table 1 an operator complaint work order classification data instance

[0053]

[0054]

[0055] (1) Context encoder (pre-training language model in the operator field)

[0056] Since the general pre-training language model performs poorly in the classification task on the operator field data, the present application collates more than 200 million pieces of text data in the operator field to continue pre-training on the RoBERTa general model to obtain a pre-training language model in the operator field. The present application uses 190,000 pieces of data for fine-tuning to train a complaint classification model preBert in the operator field. The present application uses this model preBert to train the complaint work order data to obtain a context-based semantic representation. Among them, for the complaint work order text sequence t, the context semantic representation vector of the text sequence t can be obtained through the pre-training model preBert The definitions are as follows:

[0057]

[0058] Among them, preBert(·) represents a vector encoder based on the BERT model.

[0059] (2) Knowledge-enhanced encoder

[0060] The knowledge-enhanced encoder is used to further enrich the text semantics from the knowledge level. The present application solves the problem of Chinese operator complaint scenarios, and the knowledge used is from a Chinese knowledge base HowNet, which defines semantic primitives as the smallest knowledge unit. Each word in it contains multiple word senses, and each word sense contains multiple semantic primitives, for example, the word "package" has five word senses "cloth-wrapper", "parcel", "burden", "load", and "bundle" in Hownet. Each word sense has different numbers of semantic primitives, as shown in Table 2.

[0061] Table 2: An example of a HowNet knowledge base

[0062]

[0063] The knowledge-enhanced encoder includes two parts: a semantic primitive-based knowledge learning module and a sequence-based knowledge encoding module; among them, the semantic primitive-based knowledge learning module is used to learn the word representation based on the semantic primitive level of each word, and then the sequence-based knowledge encoding module is used to aggregate and obtain the knowledge-enhanced sentence representation. The following describes the two modules in detail:

[0064] 1) Semantic primitive-based knowledge learning module

[0065] The current context encoder learns the representation of a sentence from the context, but if only relying on the context, the language model often cannot correctly understand the true meaning of the same word in different fields. In view of the above problem, the present application aggregates the semantic primitive information from the knowledge base through the semantic representation learned by the context encoder to generate knowledge-based word vectors.

[0066] Specifically, the process includes the following:

[0067] Step 1: Knowledge Acquisition. First, count the words in all text sequences to construct a vocabulary V, which includes N words, where the i-th word is represented as w. i By querying the Chinese knowledge base Hownet, the i-th word w can be obtained. i It has p meanings The jth word meaning s j It has q semantic primitives

[0068] Step 2: Semantic Primitive Representation Acquisition. Since semantic primitives are the smallest units of knowledge in the HowNet knowledge base, and are essentially words, these semantic primitives are represented using the PreBERT model. Encoding yields the corresponding semantic primitive representation vectors, which are respectively

[0069] Step 3: Semantic representation acquisition. Assumption Representative word w i The j-th meaning An embedding, which is obtained by combining the embedding vector representations of all semantic primitives under that word meaning, is defined as follows:

[0070]

[0071]

[0072] in, It refers to the meaning of the word s j The set of all semantic primitives in Chinese. The original meaning is represented by the combination of words. i The j-th meaning The importance weight of each semantic primitive in semantic representation The word w i Context vector representation obtained based on PreBERT model encoding It is the word w i The j-th meaning The semantic representation vector of the kth primitive.

[0073] Step 4: Knowledge-enhanced word representation acquisition. The i-th word is represented by a knowledge-based semantic representation. It is obtained by integrating the semantic representation of each word; the i-th word is based on the semantic representation of knowledge. The definition is as follows:

[0074]

[0075]

[0076] wherein, represents the set of word senses of the word w i , represents the importance weight of the jth word sense when combined into the semantic representation of the word w i ; finally, the semantic representation of each word knowledge enhanced is obtained by weighted combination based on the two-layer attention of the sense primitive-word sense.

[0077] Fifth step: constructing the knowledge representation vector matrix. The one to four steps of operation are performed on all words in the vocabulary V to obtain the semantic representation of each word knowledge enhanced in V, and a word vector matrix M is constructed, wherein M R N×D , N is the size of the dictionary, and D is the dimension of each word vector.

[0078] 2) Sequence-based knowledge encoding module

[0079] The above module obtains the semantic representation of each word knowledge enhanced. The application further obtains the semantic representation of the text sequence t by aggregating the semantic representation of the knowledge enhanced words. Assuming that the text sequence t contains n words, the word vectors corresponding to the n words in the sequence t are mapped based on the word vector matrix M {ew1, ew2,..., ew n}. Then, the semantic vector of the text sequence t enhanced knowledge representation is obtained by aggregating the word vectors defined as follows:

[0080]

[0081] The knowledge enhanced encoder of the application can enhance the language model to understand the breadth of language through the sense primitive knowledge base, and assist the language model to accurately analyze the different meanings of words in different fields. For example: "cancel" and "refund" in the business scenario of the application represent the same meaning - "cancel the subscription service and refund the fee". Without the assistance of the knowledge base, the language model cannot directly understand that "cancel" and "refund" are synonymous.

[0082] Finally, a pooling layer is connected to form a sentence embedding vector into the same form as the representation vector output by the context encoder, defined as:

[0083]

[0084] wherein Pooler(·) represents a pooling function.

[0085] (3) Multi-task learning classifier (such as shown in Figure 1 )

[0086] In the operator complaint work order data, each sample data has multiple labels, which are the understanding of the text example data from different perspectives. These labels are related and even contain rich logical relationships. Therefore, it is crucial to make full use of the label correlation to build a multi-label classification model with strong generalization performance. The input of the multi-task learning classifier is the embedded vector obtained by inputting the operator complaint work order text data into the context encoder and the knowledge enhancement encoder, which is defined as follows:

[0087]

[0088] Where t represents this operator complaint work order text input.

[0089] Then four classifiers corresponding to different tasks are set, each of which (CrossEntropyLoss as the loss function) is a fully connected layer, and then the predicted label l is obtained by taking the maximum probability value through the Argmax function max , Specifically, the specific process of each classifier is as follows:

[0090] logits=Classifier z (r(t)) (9)

[0091] l max =Argmax(logits) (10)

[0092] Where Classifier z (·) represents the zth classifier, and Argmax(·) represents the function of taking the maximum probability.

[0093] Example two

[0094] The data used in this embodiment is the complaint work order data from the operator, a total of 100,000, in order to train the model, it will be divided into training set, test set and validation set according to 8:1:1. Compared with traditional machine learning models and common pre-trained language models, the results are shown in Table 3:

[0095] Machine learning methods (traditional machine learning models): The basic idea is to extract features using TF-IDF and then train various machine learning classification models. Here, Naive Bayes, Decision Tree, and XGBoost are chosen as baseline methods. Naive Bayes has stable classification efficiency, can handle multi-class tasks, is suitable for incremental training, is not very sensitive to missing data, and has a relatively simple algorithm, often used for text classification. Decision Tree models have excellent interpretability and low computational complexity. XGBoost adds a regularization term to the cost function, effectively controlling the model's complexity and supporting parallel processing. Furthermore, it builds all possible subtrees from top to bottom and then performs reverse pruning from bottom to top, making it less prone to getting trapped in local optima compared to GBM.

[0096] Pre-trained language models: In 2017, Google proposed the pre-trained language model BERT, which calculates language representation vectors based on a self-attention mechanism. In the Chinese domain, the base model is BERT-base-chinese, upon which BERT-base-wwm-ext is built using full-word masking. Facebook's chinese-roberta-wwm-ext improves the BERT model by employing three techniques: remove next-sentence prediction (NSP) task, dynamic masking, and text encoding. Google's albert-base-chinese significantly reduces the model's parameters while maintaining performance. Baidu's ERNIE series of models improves the masking strategy by learning various pre-training tasks of different paradigms, enabling the model to effectively capture general lexical and syntactic information from training data. The SKEP model performs sentiment word masking, constructing three sentiment knowledge prediction targets and embedding word-level, polarity-level, and aspect-level sentiment information into the pre-trained sentiment representation.

[0097] Table 3 Experiments on different models for the operator complaint ticket classification task.

[0098]

[0099]

[0100] As can be seen from the experimental results in Table 3, the overall accuracy of the machine learning method is less than 57.2%, and the precision, recall and F1-score are also less than 57%. The accuracy of the general pre-training language model in rows 5 to 11 is highest for the AlBERT model, which is 64.6%, while the precision, recall and F1-score are all best for the RoBERTa model, but only about 64%. Row 12 is the model proposed in the application, and it can be seen that the accuracy, precision, recall and F1-score all achieve the best effect, basically about 72%, which is about 8% higher than the general pre-training language model and about 15% higher than the traditional machine learning model.

[0101] In summary, the cascade label classification model based on multi-task learning knowledge enhancement of the above-mentioned embodiments can learn knowledge from both text and labels and from labels and labels through iterative training of multi-task learning. In addition, the added knowledge enhancement module enables the model to learn semantic representation from context and from semantic primitives, which are more fine-grained, and thus the accuracy on the operator complaint work order classification dataset is 70%, which is about 8% higher than the general pre-training language model and about 15% higher than the traditional machine learning model.

[0102] Although the embodiments of the application have been shown and described above, it should be understood that the above-mentioned embodiments are exemplary and should not be construed as limiting the application. Those skilled in the art can make changes, modifications, replacements and variations to the above-mentioned embodiments within the scope of the application.

Claims

1. A method for cascade label classification based on multi-task learning knowledge enhancement, characterized in that, The method comprises the following steps: S1: semantic representation The complaint work order text is subjected to semantic representation from the perspectives of context and semantic knowledge by using a context encoder and a knowledge enhanced encoder respectively, so as to obtain a semantic representation vector of the complaint work order text; S2: multi-task classification Different levels of classification labels are obtained by four classifiers corresponding to different tasks in a multi-task learning classifier according to shared semantic representations obtained by iteratively training encoders; In the step S1, the context encoder is the operator field complaint classification model preBert, that is, the pre-training model preBert, and the pre-training model preBert is used to train the complaint work order data to obtain a context-based semantic representation. For a complaint work order text sequence t, the context semantic representation vector of the text sequence t is obtained by encoding the existing pre-training model preBert The definitions are as follows: Wherein, preBert(·) represents a vector encoder based on a BERT model; In the step S1, the knowledge enhanced encoder comprises a semantic-based knowledge learning module and a sequence-based knowledge encoding module, wherein the semantic-based knowledge learning module is used to learn word representation based on semantic primitives of each word, and then the sequence-based knowledge encoding module is used to aggregate and obtain knowledge enhanced sentence representation; The specific processing procedure of the semantic-based knowledge learning module is as follows: S101: knowledge acquisition Count all the words in the text sequence, build a vocabulary V, which includes N words, the i-th word is denoted as w i ; By querying the Chinese knowledge base Hownet, the i-th word w i has p senses The j-th sense s j has q primitives S102: semantic primitive representation acquisition The corresponding semantic primitive representation vectors are obtained by encoding the semantic primitives based on the pre-trained model preBert ​ S103: word semantic representation acquisition Assume representative word w i jth word sense of the word w embedding, which is obtained by combining all the sense primitive embedding vectors under this word sense, defined as follows: wherein, denotes the word meaning s j the set of all sememes, denotes the importance weight of the jth word meaning i in the semantic representation of the word w , denotes the context vector representation of the word w i based on the pre-trained model preBert encoding, is the semantic representation vector of the kth sememe of the jth word meaning i in the word w . S104: knowledge enhanced word representation acquisition The ith term knowledge-based semantic representation is obtained by integrating the representation of each of the word senses; the ith term knowledge-based semantic representation is defined as follows: wherein, represents the set of word senses of the word w i , represents the importance weight of the jth word sense when combining into the semantic representation of the word w i , and finally the semantic representation of each word knowledge is enhanced by the weighted combination based on the two-layer attention of sense-prime-word sense. S105: construction of knowledge representation vector matrix After the operation of steps S101-S104 is performed on all words in the vocabulary V, the semantic representation of each word in the vocabulary V is enhanced by knowledge, and a word vector matrix M is constructed, where M ∈ R N×D , N is the size of the dictionary, and D is the dimension of each word vector. The specific processing procedure of the sequence-based knowledge encoding module is as follows: S201: semantic representation of a text sequence t is obtained by aggregating knowledge enhanced semantic representation of words; S202: a sentence embedding vector is converted into the same form as the representation vector output by the context encoder by a pooling layer, defined as follows: Wherein, Pooler(·) represents a pooling function; In the step S2, the input of the multi-task learning classifier is an embedded vector obtained by inputting the operator complaint work order text data into the context encoder and the knowledge enhanced encoder, defined as follows: Wherein, t represents the operator complaint work order text input; In the step S2, each classifier is a fully connected layer, and the predicted label l is obtained by taking the maximum probability value through the Argmax function max The specific process of each classifier is as follows: logits = Classifier z (r(t)) l max = Argmax(logits) where Classifier z (·) denotes the z-th classifier, and Argmax(·) denotes a function that takes the maximum probability.

2. The method of claim 1, wherein the method is based on multi-task learning knowledge enhancement. In the step S201, the text sequence t contains n words, and the n words in the sequence t are mapped to the word vectors {ew1, ew2,..., ew n} based on the word vector matrix M, and the aggregated word vectors obtain the semantic vector of the text sequence t to enhance the knowledge representation The definitions are as follows:

3. The method of claim 2, wherein the method is based on multi-task learning knowledge enhancement. In the step S2, different levels of classification labels comprise first to fourth levels, wherein the first level represents coarse classification of the business to which the complaint belongs, the second level label represents fine classification of the business, the third level represents a specific problem category of the complaint, and the fourth level represents a specific problem fine classification of the complaint.

Images