Text classification model training method, text classification method and device

Abstract

The application provides a text classification model training method, a text classification method and a device. The text classification model training method comprises: obtaining pre-training samples, wherein the pre-training samples comprise k rounds of dialogue text content of different dialogue roles, and k is an integer greater than 1; performing feature extraction on the pre-training samples to obtain a feature vector, wherein the feature extraction comprises role feature extraction; obtaining a loss value according to the feature vector, and pre-training an initial model according to the loss value to obtain a pre-training model; and fine-tuning parameters of the pre-training model to obtain a text classification model. The technical solution provided by the application can improve the classification effect of the text classification model.

Description

Technical Field

[0001] This invention relates to the field of neural network technology, specifically to a training method and apparatus for a text classification model, and a text classification method and apparatus. Background Technology

[0002] In recent years, pre-trained models have been widely used in the field of natural language processing. These models are first pre-trained on a large number of unlabeled samples using a self-supervised approach to learn a general text representation. Then, in application, the model only needs to be fine-tuned for specific tasks (such as text classification). Compared to training from scratch, fine-tuning based on a pre-trained model requires fewer training samples and the model has stronger generalization ability.

[0003] However, current pre-trained models, such as BERT (Bidirectional Encoder Representation from Transformers), typically use training corpora that directly concatenate dialogue text content without targeting specific dialogue scenarios. As a result, the training corpora differ significantly from the dialogue scenario corpora used in applications, affecting the classification accuracy of text classification models. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a training method and apparatus for a text classification model, and a text classification method and apparatus, which can improve the classification performance of the text classification model.

[0005] According to a first aspect of the present invention, a method for training a text classification model is provided, comprising: acquiring pre-training samples, wherein the pre-training samples include k rounds of dialogue text content labeled with different dialogue roles, where k is an integer greater than 1, and one question-and-answer process between different dialogue roles constitutes one round of dialogue; performing feature extraction on the pre-training samples to obtain feature vectors, wherein the feature extraction includes role feature extraction; obtaining a loss value based on the feature vectors, and pre-training an initial model based on the loss value to obtain a pre-trained model; and fine-tuning the parameters of the pre-trained model to obtain a text classification model.

[0006] In one embodiment of the present invention, the first t rounds of dialogue in the k-round dialogue text content are context text information, and the last kt rounds of dialogue in the k-round dialogue text content are response text information, wherein the response text information is the response text corresponding to the context relationship of the context text information; t is an integer less than k, and the loss value includes a first loss value. The above-mentioned method of obtaining the loss value based on the feature vector and pre-training the initial model based on the loss value to obtain a pre-trained model includes: obtaining the probability that the response text information is the response text information of the context text information based on the feature vector, and obtaining the first loss value based on the probability; and pre-training the initial model based on the first loss value to obtain a pre-trained model.

[0007] In one embodiment of the present invention, the above-mentioned method of obtaining the probability that the response text information is the response text information of the context text information based on the feature vector, and obtaining a first loss value based on the probability, includes: inputting the feature vector into an initial model, using the initial model to extract context text features from the context text information to obtain a context feature matrix, and using the initial model to extract response text features from the response text information to obtain a response feature matrix; obtaining the probability that the response text information is the response text information corresponding to the context relationship of the context text information based on the context feature matrix and the response feature matrix; and obtaining a first loss value based on the probability using a loss function.

[0008] In one embodiment of the present invention, the loss value includes a second loss value. The method further includes: masking a portion of the words in the pre-training samples to obtain masked text information; wherein the above-mentioned feature extraction of the pre-training samples to obtain a feature vector includes: performing feature extraction on the masked text information to obtain a feature vector; wherein the above-mentioned obtaining a loss value based on the feature vector and pre-training the initial model based on the loss value to obtain a pre-trained model includes: obtaining a prediction result of the mask position based on the feature vector and obtaining a second loss value; and pre-training the initial model based on the second loss value to obtain a pre-trained model.

[0009] In one embodiment of the present invention, the above-mentioned fine-tuning of the parameters of the pre-trained model to obtain a text classification model includes: acquiring training samples, wherein the training samples include dialogue text information and classification labels, and the dialogue text information includes at least one round of dialogue text content labeled with different dialogue roles; using the pre-trained model to extract features from the dialogue text information to obtain feature vectors, wherein the feature extraction includes role feature extraction; classifying the text based on the feature vectors to obtain classification prediction results, and obtaining a third loss value based on the classification prediction results and classification labels; and fine-tuning the parameters of the pre-trained model based on the third loss value to obtain a text classification model.

[0010] In one embodiment of the present invention, feature extraction further includes dialogue turn number feature extraction, wherein the dialogue turn number feature is used to identify the turn in which each turn of dialogue content in the k-turn dialogue text content is located.

[0011] In one embodiment of the present invention, the pre-trained model includes a transformer-based bidirectional encoding representation BERT model.

[0012] In one embodiment of the present invention, the text classification model includes an intent classification model.

[0013] According to a second aspect of the present invention, a text classification method is provided, comprising: acquiring dialogue text information, wherein the dialogue text information includes at least one round of dialogue text content annotated with different dialogue roles; performing feature extraction on the dialogue text information to obtain a feature vector, wherein the feature extraction includes role feature extraction; and classifying the text based on the feature vector.

[0014] In one embodiment of the present invention, at least one round of dialogue text content includes multi-round dialogue text content, wherein feature extraction further includes extracting dialogue round number features, which are used to identify the round in which each round of dialogue content in the multi-round dialogue text content is located.

[0015] In one embodiment of the invention, text classification includes intent classification.

[0016] According to a third aspect of the present invention, a training apparatus for a text classification model is provided, comprising: an acquisition module for acquiring pre-training samples, wherein the pre-training samples include k rounds of dialogue text content labeled with different dialogue roles, where k is an integer greater than 1, and one question-and-answer process between different dialogue roles constitutes one round of dialogue; a feature extraction module for extracting features from the pre-training samples to obtain feature vectors, wherein the feature extraction includes role feature extraction; a pre-training module for obtaining a loss value based on the feature vectors and pre-training an initial model based on the loss value to obtain a pre-trained model; and a fine-tuning module for fine-tuning the parameters of the pre-trained model to obtain a text classification model.

[0017] According to a fourth aspect of the present invention, a text classification apparatus is provided, comprising: an acquisition module for acquiring dialogue text information, wherein the dialogue text information includes at least one round of dialogue text content annotated with different dialogue roles; a feature extraction module for extracting features from the dialogue text information to obtain feature vectors, wherein the feature extraction includes role feature extraction; and a classification module for classifying the text based on the feature vectors.

[0018] According to a fifth aspect of the present invention, a computer-readable storage medium is provided, having stored thereon computer-executable instructions, wherein the executable instructions, when executed by a processor, implement the method as described in any of the preceding claims.

[0019] According to a sixth aspect of the present invention, an electronic device is provided, characterized in that the electronic device includes: a processor; a memory for storing processor-executable instructions; the processor being configured to perform the method described in any of the preceding embodiments.

[0020] According to the technical solution provided in the embodiments of the present invention, by adding special markers to the input pre-training samples to distinguish dialogue roles and extracting role features (such as role embedding), the model can learn the influence of different identity speech on the final text classification, thereby improving the accuracy of text classification. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 The diagram shown is a flowchart illustrating the training method of a text classification model provided in an embodiment of the present invention.

[0023] Figure 2 The diagram shown is a schematic representation of the structure of a pre-trained model provided in an embodiment of the present invention.

[0024] Figure 3 The diagram shown is a schematic diagram of the embedded layer provided in an embodiment of the present invention.

[0025] Figure 4 The diagram shown is a schematic diagram of the embedded layer provided in another embodiment of the present invention.

[0026] Figure 5 The diagram shown is a schematic representation of a processing method for obtaining a pre-trained model according to an embodiment of the present invention.

[0027] Figure 6 The diagram shows a processing method for fine-tuning a pre-trained model to obtain a text classification model, according to an embodiment of the present invention.

[0028] Figure 7 The diagram shown is a flowchart illustrating the training method of a text classification model provided in another embodiment of the present invention.

[0029] Figure 8 The diagram shown is a flowchart of a text classification method provided in an embodiment of the present invention.

[0030] Figure 9The diagram shown is a block diagram of a training device for a text classification model provided in an embodiment of the present invention.

[0031] Figure 10 The diagram shown is a block diagram of a text classification device provided in an embodiment of the present invention.

[0032] Figure 11 The diagram shown is a block diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0033] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0034] In recent years, pre-trained models have been widely used in the field of natural language processing. These models are first pre-trained on a large number of unlabeled samples using a self-supervised approach to learn a general text representation. Then, in application, the model only needs to be fine-tuned for specific tasks (such as text classification). Compared to training from scratch, fine-tuning based on a pre-trained model requires fewer training samples and the model has stronger generalization ability.

[0035] Currently, commonly used pre-trained models include the BERT (Bidirectional Encoder Representation from Transformers) model. The training corpus for BERT models typically consists of directly concatenating dialogue text, such as: "Hi, it's my pleasure to assist you. How can I help you? I haven't borrowed any money, but it shows I have uncashed checks in my account, and the billing date is already shown. Let me confirm your question; you need to inquire about repayment, right? No, it's not about repayment. I haven't borrowed any money from your platform, but I have a repayment date, and the bill is already shown. Hi, you're asking why you have a billing date and repayment date even though you haven't borrowed any money, right? Where did you see that? My loan." This training data is then input into the BERT model for pre-training.

[0036] Specifically, the pre-training tasks of the BERT model include MLM (Masked Language Model) multi-class classification and NSP (Next Sentence Prediction) two-class classification.

[0037] The MLM task involves masking portions of the input sentence and then having the model predict the masked words. The MLM loss function is shown below:

[0038]

[0039] Where M is the number of words masked, P(x m ) is the word x m The predicted probability.

[0040] The NSP task is used to determine whether sentence A is the next sentence of sentence B. The NSP loss function is shown below:

[0041] L nsp = -logP(t|x,y)

[0042] If x and y are two consecutive sentences in the original corpus, then t = 1; otherwise, t = 0.

[0043] The loss function of the BERT model consists of two parts: the MLM loss function and the NSP loss function, i.e., Final Loss = MLM Loss + NSP Loss. Then, the BERT model can be pre-trained based on the obtained Final Loss.

[0044] In summary, the training corpora for commonly used BERT models are usually just concatenated dialogue texts, without specifically designing training tasks for dialogue-type data to learn the contextual structure of the dialogue. Therefore, it is essential to design pre-trained models specifically based on dialogue-type corpora.

[0045] Figure 1 The diagram shows a flowchart of a text classification model training method according to an embodiment of the present invention. This method can be executed by a computer device (e.g., a server). Figure 1 As shown, the method includes the following:

[0046] S110: Obtain pre-training samples, where the pre-training samples include k rounds of dialogue text labeled with different dialogue roles, where k is an integer greater than 1.

[0047] For example, it can obtain k-round (one question and one answer per round) dialogue content between a company's actual customer service representatives and customers, and add corresponding tokens to distinguish the dialogue roles during input. For example, such as Figure 2 As shown, customer service is represented by [SEAT] and customers by [USER], thus obtaining pre-training samples.

[0048] By using the conversational data provided by this invention to pre-train the initial model, a pre-trained model can be obtained, which can learn rich conversational knowledge. Then, the pre-trained model is fine-tuned based on the target text classification task, and a text classification model with good performance can be obtained with only a small number of labeled training samples.

[0049] In one embodiment of the present invention, the pre-trained model may include a Bidirectional Encoder Representation from Transformers (BERT) model. The BERT model may employ 12 Transformer Blocks, also known as Transformer Encoders. The structure of the BERT model will not be described in detail in this invention.

[0050] Specifically, after obtaining the dialogue data between customer service representatives and customers, in order to improve the quality of the training data, the dialogue data can be preprocessed, for example:

[0051] 1. Remove special characters from the data, such as HTML tags, emoticons, image links, etc.

[0052] 2. Filter out non-human dialogue in the conversation by using the system scripts maintained in the database, such as "The guest has left" or "Entering human service".

[0053] 3. Remove dialogues with fewer than 3 rounds (one question and one answer counts as one round). It should be understood that this description is merely illustrative, and the present invention does not impose a specific limitation on the number of dialogue rounds.

[0054] 4. If consecutive sentences contain the same identity dialogue, they can be merged to ensure that each sample in the same dialogue after processing represents different identity dialogue.

[0055] The training corpus examples are shown in Table 1.

[0056] Table 1

[0057]

[0058] S120: Perform feature extraction on the pre-trained samples to obtain feature vectors, where feature extraction includes role feature extraction.

[0059] Role characteristics represent the dialogue role corresponding to each sentence of dialogue text in each round of dialogue text.

[0060] like Figure 2As shown, taking the BERT model as an example, the vectorized representation of the pre-trained samples can be obtained through the embedding layer of the BERT model, that is, the aforementioned feature vector can be obtained.

[0061] Specifically, such as Figure 3 As shown, the token embedding and position embedding of each word in the text information of the pre-trained sample can be obtained through the token embedding layer and position embedding layer of the BERT model. In this embodiment of the invention, the role embedding representation of the pre-trained sample can also be obtained through the role embedding layer of the BERT model.

[0062] The token embedding layer can convert each word into a vector of fixed dimensions.

[0063] Position Embedding layer: can represent the positional features of each word in a sentence.

[0064] Role Embedding layer: can distinguish the dialogue role characteristics of each sentence.

[0065] Specifically, the word vectors obtained from these three embedding results (i.e., TokenEmbedding, Position Embedding, and Role Embedding) can be obtained in sequence and concatenated to obtain the final BERT word vector, i.e., Token Embedding + Role Embedding + Position Embedding.

[0066] By adding special markers to the input pre-training samples to distinguish dialogue roles and extracting role features (such as role embedding), the model can learn the impact of different role-based speech on the final text classification, thereby improving the accuracy of text classification.

[0067] In another embodiment of the invention, dialogue turn number feature extraction may also be included. The dialogue turn number feature represents the turn in which each turn of dialogue text content exists within k turns of dialogue text content.

[0068] like Figure 4As shown, in addition to the three embedding layers mentioned above, the vector TurnEmbedding can also be obtained through the round embedding layer. The word vectors obtained from these four embedding results (i.e., Token Embedding, PositionEmbedding, Role Embedding, and Turn Embedding) are then concatenated to obtain the final BERT word vector, namely TokenEmbedding+Role Embedding+Turn Embedding+Position Embedding.

[0069] By adding Turn Embedding, the model can capture potential dialogue order and dialogue structure information. In addition, dialogue data is generally very long, making it difficult for the model to learn the entire dialogue information. By combining Turn Embedding and Role Embedding, the model can be more inclined to learn the important parts of the dialogue information.

[0070] It should be noted that the above-described method of obtaining role features and round number features through the embedding layer is only an exemplary description, and the present invention does not specifically limit the feature extraction method.

[0071] S130: Obtain the loss value based on the feature vector, and pre-train the initial model based on the loss value to obtain the pre-trained model.

[0072] like Figure 2 As shown, the word vectors output from the embedding layer can be input into the 12-layer Transformer Blocks of the BERT model to obtain the loss value. Adjusting the parameters of the initial model based on this loss value yields the pre-trained model.

[0073] S140: Fine-tune the parameters of the pre-trained model to obtain a text classification model.

[0074] For example, a text classification model may include an intent classification model; it should be understood that this invention does not specifically limit this.

[0075] According to the technical solution provided in the embodiments of the present invention, by adding special markers to the input pre-training samples to distinguish dialogue roles and extracting role features (such as role embedding), the model can learn the influence of different identity speech on the final text classification, thereby improving the accuracy of text classification.

[0076] In one embodiment of the present invention, the first t rounds of dialogue in the k-round dialogue text content are context text information, and the last kt rounds of dialogue in the k-round dialogue text content are response text information, wherein the response text information is the response text corresponding to the context relationship of the context text information; t is an integer less than k, and the loss value includes a first loss value. The above-mentioned method of obtaining the loss value based on the feature vector and pre-training the initial model based on the loss value to obtain a pre-trained model includes: obtaining the probability that the response text information is the response text information of the context text information based on the feature vector, and obtaining the first loss value based on the probability; and pre-training the initial model based on the first loss value to obtain a pre-trained model.

[0077] In other words, we can randomly select the number of rounds *t* to divide each dialogue into two parts: the first *t* rounds serve as the context (hereinafter referred to as *C*), and the subsequent rounds serve as the corresponding responses (hereinafter referred to as *R*), thus obtaining sample pairs. For example, the sample pair corresponding to dialogue 1 is (C1, R1), where C1 represents the first *t* rounds of dialogue 1, and R1 represents the subsequent rounds of dialogue 1. For instance, if the content of dialogue C1 is about bank loans, and the content of dialogue R1 is about ordering takeout, then the context relationship between R1 and C1 does not correspond, and the probability that R1 is the response text corresponding to C1 is low; if the content of dialogues C1 and R1 is both about bank loans, then the probability that R1 is the response text corresponding to C1 is high.

[0078] It should be understood that multiple dialogues can be selected as pre-training samples, i.e., batch data can be constructed to pre-train the model. Similarly, if the sample pair corresponding to dialogue 2 is (C2, R2), and the sample pair corresponding to dialogue 3 is (C3, R3), then the sample pairs corresponding to n dialogues are (C1, R1, ..., C2). n ,R n For example, the batch size is 128, i.e., n = 128. It should be understood that the sample size is only an illustrative description, and this invention does not specifically limit it. To facilitate the construction of pre-training samples, n dialogues with consistent context can be selected first, that is, the context and the dialogue content of the responses in each dialogue are consistent. For example, the dialogue content of C1 and R1 is about bank loans, the dialogue content of C2 and R2 is about bank loans, and so on, to obtain n sample pairs (C1, R1, ... C1) corresponding to the dialogues. n ,R n Then, the order of the above sample pairs is randomly shuffled, for example, to obtain pre-training samples (C2, R). m (C1, R2). Each batch contains 128 dialogue sample pairs.

[0079] Based on the pre-trained samples mentioned above, a multi-classification task, namely the NRS (Next Response Select) task, can be constructed. The number of categories is the batch size, which is to determine whether the response sample information R is the response corresponding to the context information C.

[0080] For example, we can determine the response text information R1, R2, R3, ..., R... n Let R1 be the probability of the response text corresponding to the contextual relationship of the contextual text information C1, for example (0.7, 0.1, 0, ..., 0.1). Among them, R1 has the highest probability of being the response text corresponding to C1, which is 0.7. Therefore, R1 can be predicted to be the response text corresponding to C1. Based on this probability, the first loss value, i.e., the NRS loss value, is obtained. Then, the parameters of the initial model are adjusted based on this loss value to obtain the pre-trained model, i.e., the pre-trained model.

[0081] Specifically, the aforementioned feature vectors can be input into the initial model, and the initial model can be used to extract context text features from the context text information to obtain the context feature matrix. The initial model can also be used to extract response text features from the response text information to obtain the response feature matrix. Based on the context feature matrix and the response feature matrix, the probability that the response text information is the response text corresponding to the context relationship of the context text information can be obtained. Based on this probability, the first loss value can be obtained using the loss function.

[0082] Specifically, feature extraction can be performed on the context text and response text separately using a 12-layer transformer block to obtain the context representation matrix C∈R. b*h The response represents the matrix R ∈ R b*h Where b is the batch size (batch_size), for example, it can be 128, and h is the hidden layer dimension (hidden_size), for example, it can be 768.

[0083] In one embodiment of the present invention, the formula for the loss function can be as follows:

[0084]

[0085] Where M is the probability, M = Softmax(CR) T ), C∈R b*h Let R be the context feature matrix, and R ∈ R b*h The response is the feature matrix, where b is the batch size and h is the hidden layer dimension.

[0086] According to the technical solution provided by the embodiments of the present invention, by constructing context text and response text from conversational corpus and calculating the correlation between context and response within the same batch, the model can learn the complex dialogue relationship between different responses and conversational context text.

[0087] In one embodiment of the present invention, the loss value includes a second loss value. The method further includes: masking a portion of the words in the pre-training samples to obtain masked text information; wherein the above-mentioned feature extraction of the pre-training samples to obtain a feature vector includes: performing feature extraction on the masked text information to obtain a feature vector; wherein the above-mentioned obtaining a loss value based on the feature vector and pre-training the initial model based on the loss value to obtain a pre-trained model includes: obtaining a prediction result of the mask position based on the feature vector and obtaining a second loss value; and pre-training the initial model based on the second loss value to obtain a pre-trained model.

[0088] In other words, a multi-class classification task can be constructed, namely the MLM (Masked Language Model) task, a classification task with the number of categories equal to the number of words in the dictionary. That is, a portion of the words in the input sentence is masked, and the model predicts the masked words. Then, a second loss value (i.e., the MLM loss value) is obtained based on the prediction results, and the parameters of the initial model are adjusted based on this loss value.

[0089] The loss function formula can be shown below:

[0090]

[0091] Where M is the number of words being masked, P(x m ) for word x m The predicted probability.

[0092] In another embodiment of the invention, such as Figure 5 As shown, the initial model can be trained simultaneously based on both the NRS and MLM tasks, with the final loss function being L. mlm +L nrs According to the technical solution provided in the embodiments of the present invention, training the initial model through NRS and MLM tasks can improve the performance of the pre-trained model.

[0093] In one embodiment of the present invention, the above-mentioned fine-tuning of the parameters of the pre-trained model to obtain a text classification model includes: acquiring training samples, wherein the training samples include dialogue text information and classification labels, and the dialogue text information includes at least one round of dialogue text content labeled with different dialogue roles; using the pre-trained model to extract features from the dialogue text information to obtain feature vectors, wherein the feature extraction includes role feature extraction; classifying the text based on the feature vectors to obtain classification prediction results, and obtaining a third loss value based on the classification prediction results and classification labels; and fine-tuning the parameters of the pre-trained model based on the third loss value to obtain a text classification model.

[0094] Specifically, such as Figure 6 As shown, the input to the pre-trained model is the dialogue context text information. A special role token can be added before each sentence of text to distinguish the dialogue roles. For example, [SEAT] represents customer service and [USER] represents customer service.

[0095] By inputting contextual text information into a pre-trained model for feature extraction, the [CLS] vector output by the pre-trained model can be used as the sentence vector representation of the training samples.

[0096] Then, a softmax classifier is added to classify and predict the [CLS] vector (e.g., intent classification); a third loss value is obtained based on the classification label and the classification prediction result, and the parameters of the pre-trained model are fine-tuned using the third loss value to obtain the text classification model.

[0097] The loss function can be cross-entropy. It should be understood that this invention does not impose specific limitations on the form of the loss function.

[0098] According to the technical solution provided by the embodiments of the present invention, the present invention is based on a pre-trained model designed specifically for dialogue corpora, which is fine-tuned. The pre-trained model has learned rich dialogue knowledge on a large amount of dialogue corpora in this scenario in advance. Therefore, when applied in the scenario, only a small number of labeled samples are needed to achieve good results.

[0099] Figure 7 The diagram shows a flowchart of a text classification model training method according to an embodiment of the present invention. This method can be executed by a computer device (e.g., a server). Figure 7 The example is as follows Figure 1 Specific examples of the embodiments are described below. Figure 5 right Figure 7 The embodiments are described in detail below. Figure 7 As shown, the method includes the following:

[0100] S710: Obtain pre-training samples, wherein the pre-training samples include k rounds of dialogue text content labeled with different dialogue roles, the first t rounds of dialogue in the k rounds of dialogue text content are context text information, the last kt rounds of dialogue in the k rounds of dialogue text content are response text information, k is an integer greater than 1, and t is an integer less than k.

[0101] Specifically, this invention can be applied to natural language understanding tasks using conversational corpora. Therefore, the training corpus uses real conversations between customer service representatives and customers of a company. To improve the quality of the training data, the following preprocessing was performed:

[0102] 1. Remove special characters from the data, such as HTML tags, emoticons, image links, etc.

[0103] 2. Filter out non-human dialogue in the conversation by using the system scripts maintained in the database, such as "The guest has left" or "Entering human service".

[0104] 3. Remove dialogues with fewer than 3 rounds (one question and one answer counts as one round).

[0105] 4. If consecutive sentences contain the same identity dialogue, they can be merged to ensure that each sample in the same dialogue after processing represents different identity dialogue.

[0106] Unlike the NSP (Next Sentence Prediction) task in the traditional BERT model (i.e., determining whether sentence A is the next sentence of sentence B), this invention constructs training samples by building context and corresponding response sample pairs:

[0107] Randomly select a number of rounds *t* to divide each dialogue into two parts: the first *t* rounds serve as the context (hereinafter referred to as *C*), and the subsequent rounds serve as the corresponding responses (hereinafter referred to as *R*), resulting in sample pairs. For example, the sample pair corresponding to dialogue 1 is (C1, R1), the sample pair corresponding to dialogue 2 is (C2, R2), and the sample pairs corresponding to *n* dialogues are (C1, R1, ..., C...). n ,R n For example, n=128. It should be understood that the sample size is only an exemplary description and the present invention does not make any specific limitation on it.

[0108] Secondly, batch data can be constructed, that is, the order of the above sample pairs is randomly shuffled, for example, to obtain pre-training samples (C2, R). m ,…C1,R2).

[0109] Conventional sentence pairing tasks use single sentences, that is, directly splicing together multiple sentences of dialogue text, which is not suitable for the multi-turn structure of dialogue scenarios. In contrast, the sentence pairing construction in this invention uses multi-turn contexts, which is more suitable for text classification tasks in multi-turn dialogues.

[0110] S720: Mask some words in the pre-trained samples to obtain the masked text information.

[0111] S730: Perform feature extraction on the masked text information to obtain a feature vector. Feature extraction includes character feature extraction and dialogue turn number feature extraction.

[0112] The role feature represents the dialogue role corresponding to each sentence in each round of dialogue text. The dialogue round number feature is used to identify the round in which each round of dialogue text exists within k rounds of dialogue text.

[0113] Specifically, special tokens can be added to the input to distinguish dialogue roles. For example, customer service representatives are represented by [SEAT], and customers by [USER]. Figure 4 As shown, role features can be obtained through the role embedding layer of the BERT model, and round number features can be obtained through the round number embedding layer.

[0114] Then, the word vectors obtained from the four embedding results (i.e., Token Embedding, Position Embedding, Role Embedding, and Turn Embedding) can be concatenated to obtain the final BERT word vector, which is TokenEmbedding+Role Embedding+Turn Embedding+Position Embedding.

[0115] Compared to traditional BERT embedding layers that obtain word vectors (i.e., Token Embedding, Position Embedding, and Segment Embedding), this invention replaces Segment Embedding with Turn Embedding and Role Embedding. Turn Embedding is used to record the number of dialogue rounds, and Role Embedding is used to distinguish dialogue roles.

[0116] This invention improves the accuracy of text classification by adding special markers to the input pre-training samples to distinguish dialogue roles and extracting role features (such as role embedding). This allows the model to learn the impact of different role-based speech on the final text classification.

[0117] In addition, by adding Turn Embedding, the model can capture potential dialogue order and dialogue structure information. Furthermore, dialogue data is generally very long, making it difficult for the model to learn the entire dialogue information. By combining Turn Embedding and Role Embedding, the model can be more inclined to learn the important parts of the dialogue.

[0118] S740: Input the feature vector into the initial model, use the initial model to extract context text features from the context text information to obtain the context feature matrix, and use the initial model to extract response text features from the response text information to obtain the response feature matrix; based on the context feature matrix and the response feature matrix, use the loss function to obtain the NRS loss value.

[0119] Specifically, feature extraction can be performed on the context text and response text separately using a 12-layer transformer block to obtain the context representation matrix C∈R. b*h The response represents the matrix R ∈ R b*h Where b is the batch size (batch_size), for example, it can be 128, and h is the hidden layer dimension (hidden_size), for example, it can be 768.

[0120] In one embodiment of the present invention, the formula for the NRS loss function can be as follows:

[0121]

[0122] Where M = Softmax(CR) T ), C∈R b*h Let R be the context feature matrix, and R ∈ R b*h The response is the feature matrix, where b is the batch size and h is the hidden layer dimension.

[0123] Conventional NRS training tasks use binary classification tasks that are too simple and cannot learn complex dialogue structure information. This invention uses a multi-classification task (i.e., the NRS task mentioned above) within the same batch, which is more difficult than binary classification tasks, and the model is better able to learn the complex dialogue relationships between different response texts and dialogue context texts.

[0124] S750: Obtain the prediction result of the mask position based on the feature vector, and obtain the MLM loss value.

[0125] The MLM (Masked Language Model) task is a classification task where the number of categories is equal to the number of words in the dictionary. Specifically, it involves masking a portion of the words in an input sentence and having the model predict the masked words. The MLM loss value is then obtained based on the prediction results, and the initial model parameters are adjusted accordingly.

[0126] The MLM loss function formula can be shown below:

[0127]

[0128] Where M is the number of words being masked, P(x m ) for word x m The predicted probability.

[0129] S760: Pre-train the initial model based on the NRS loss value and MLM loss value to obtain a pre-trained model.

[0130] The final loss function is L mlm +L nrs That is, the initial model can be pre-trained based on the sum of the NRS loss value and the MLM loss value to obtain a pre-trained model.

[0131] S770: Fine-tune the parameters of the pre-trained model to obtain an intent classification model.

[0132] The input to the pre-trained model is the dialogue context text information. A special role token can be added before each sentence of text to distinguish the dialogue roles. For example, [SEAT] represents customer service and [USER] represents customer service.

[0133] By inputting contextual text information into a pre-trained model for feature extraction, the [CLS] vector output by the pre-trained model can be used as the sentence vector representation of the training samples.

[0134] Then, a softmax classifier is added to classify and predict the [CLS] vector (e.g., intent classification); a third loss value is obtained based on the classification label and classification prediction, and the parameters of the pre-trained model are fine-tuned using the third loss value to obtain the text classification model.

[0135] The loss function can be cross-entropy. It should be understood that this invention does not impose specific limitations on the form of the loss function.

[0136] According to the technical solution provided in this embodiment of the invention, by adding special markers to the input pre-training samples to distinguish dialogue roles and extracting role features (e.g., Role Embedding), the model can learn the influence of different role-related speech on the final text classification, thereby improving the accuracy of text classification. Adding Turn Embedding helps the model capture potential dialogue order and structure information; furthermore, dialogue data is generally very long, making it difficult for the model to learn the entire dialogue information. Combining Turn Embedding and Role Embedding allows the model to focus on learning the important parts of the dialogue. Additionally, by constructing context text and response text from the dialogue corpus, the model can learn the complex dialogue relationships between the context text and the response text. Moreover, this invention is based on a pre-trained model specifically designed for dialogue corpora, which has already learned rich dialogue knowledge on a large amount of dialogue corpus in this scenario. Therefore, only a small number of labeled samples are needed to achieve good results when applied in the scenario.

[0137] Figure 8 The diagram shown is a schematic flowchart of a text classification method provided by an embodiment of the present invention. This method can be executed by a computer device (e.g., a server). Figure 8 As shown, the method includes the following:

[0138] S810: Obtain dialogue text information, wherein the dialogue text information includes at least one round of dialogue text content annotated with different dialogue roles.

[0139] For example, this text classification method can be implemented using a text classification model. Specifically, the input to the text classification model is the content of at least one round of dialogue. Each sentence of text can be preceded by a corresponding role token to distinguish the dialogue roles. For example, [SEAT] represents customer service and [USER] represents customer service.

[0140] S820: Extract features from the dialogue text information to obtain feature vectors, including character feature extraction.

[0141] By inputting the dialogue text information into a text classification model for feature extraction, the [CLS] vector output by the model can be used as the sentence vector representation of the dialogue text information.

[0142] S830: Text classification based on feature vectors.

[0143] Then, the softmax classifier is used to classify the [CLS] vector into text.

[0144] It should be noted that text classification can include intent classification, but this invention does not specifically limit it.

[0145] According to the technical solution provided by the embodiments of the present invention, by adding special markers to the dialogue text information to distinguish the dialogue roles and extracting role features (such as role embedding), the model can classify text according to the influence of different role speech, thereby improving the accuracy of text classification.

[0146] In one embodiment of the present invention, at least one round of dialogue text content includes multi-round dialogue text content, wherein feature extraction further includes extracting dialogue round number features, which are used to identify the round in which each round of dialogue content in the multi-round dialogue text content is located.

[0147] For example, the number of dialogue turns can be used as a turn embedding feature. In a customer service conversation, one question and one answer constitute one turn. When the dialogue text includes multi-turn dialogue content, by extracting the number of dialogue turns feature, the text classification model can classify the dialogue based on the captured dialogue order and structure information, especially for longer dialogue data, where it can achieve better classification results.

[0148] All the above-mentioned optional technical solutions can be combined in any way to form optional embodiments of the present invention, and will not be described in detail here.

[0149] The following are embodiments of the apparatus of the present invention, which can be used to execute embodiments of the method of the present invention. For details not disclosed in the embodiments of the apparatus of the present invention, please refer to the embodiments of the method of the present invention.

[0150] Figure 9 The diagram shown is a block diagram of a training device for a text classification model provided in an embodiment of the present invention. Figure 9 As shown, the training device 900 for the text classification model includes:

[0151] The acquisition module 910 is used to acquire pre-training samples, which include k rounds of dialogue text labeled with different dialogue roles, where k is an integer greater than 1, and one question-and-answer process between different dialogue roles is one round of dialogue.

[0152] The feature extraction module 920 is used to extract features from the pre-training samples to obtain feature vectors, including character feature extraction.

[0153] The pre-training module 930 is used to obtain the loss value based on the feature vector and to pre-train the initial model based on the loss value to obtain the pre-trained model.

[0154] The fine-tuning module 940 is used to fine-tune the parameters of the pre-trained model to obtain a text classification model.

[0155] According to the technical solution provided in the embodiments of the present invention, by adding special markers to the input pre-training samples to distinguish dialogue roles and extracting role features (such as role embedding), the model can learn the influence of different identity speech on the final text classification, thereby improving the accuracy of text classification.

[0156] In one embodiment of the present invention, the first t rounds of dialogue in the k-round dialogue text content are context text information, and the last kt rounds of dialogue in the k-round dialogue text content are response text information. An initial model is used; t is an integer less than k, and the loss value includes a first loss value. The pre-training module 930 is used to obtain the probability that the response text information is the context text information based on the feature vector, and obtain the first loss value based on the probability. The initial model is pre-trained based on the first loss value to obtain a pre-trained model.

[0157] In one embodiment of the present invention, the pre-training module 930 is used to input the feature vector into the initial model, use the initial model to extract context text features from the context text information to obtain a context feature matrix, and use the initial model to extract response text features from the response text information to obtain a response feature matrix; based on the context feature matrix and the response feature matrix, obtain the probability that the response text information is the response text corresponding to the context relationship of the context text information; and based on the probability, use a loss function to obtain a first loss value.

[0158] In one embodiment of the present invention, the formula for the loss function is as follows:

[0159]

[0160] Where M = Softmax(CR) T ), C∈R b*h Let R be the context feature matrix, and R ∈ R b*h The response is the feature matrix, where b is the batch size and h is the hidden layer dimension.

[0161] In one embodiment of the present invention, the loss value includes a second loss value. The device further includes a masking module 950, which is used to mask some words in the pre-training samples to obtain masked text information. The feature extraction module 920 is used to extract features from the masked text information to obtain a feature vector. The pre-training module 930 is used to obtain the prediction result of the mask position based on the feature vector and obtain the second loss value. The initial model is pre-trained based on the second loss value to obtain a pre-trained model.

[0162] In one embodiment of the present invention, the fine-tuning module 940 is used to acquire training samples, wherein the training samples include dialogue text information and classification labels, the dialogue text information includes at least one round of dialogue text content labeled with different dialogue roles; feature extraction is performed on the dialogue text information using a pre-trained model to obtain feature vectors, wherein feature extraction includes role feature extraction; text classification is performed based on the feature vectors to obtain classification prediction results, and a third loss value is obtained based on the classification prediction results and classification labels; the parameters of the pre-trained model are fine-tuned based on the third loss value to obtain a text classification model.

[0163] In one embodiment of the present invention, feature extraction further includes dialogue turn number feature extraction, wherein the dialogue turn number feature is used to identify the turn in which each turn of dialogue content in the k-turn dialogue text content is located.

[0164] In one embodiment of the present invention, the pre-trained model includes a transformer-based bidirectional encoding representation BERT model.

[0165] In one embodiment of the present invention, the text classification model includes an intent classification model.

[0166] Figure 10 The diagram shown is a block diagram of a text classification device according to an embodiment of the present invention. Figure 10 As shown, the text classification device 1000 includes:

[0167] The acquisition module 1010 is used to acquire dialogue text information, wherein the dialogue text information includes at least one round of dialogue text content labeled with different dialogue roles;

[0168] The feature extraction module 1020 is used to extract features from the dialogue text information to obtain feature vectors, including character feature extraction.

[0169] Classification module 1030 is used to classify text based on feature vectors.

[0170] According to the technical solution provided by the embodiments of the present invention, by adding special markers to the dialogue text information to distinguish the dialogue roles and extracting role features (such as role embedding), the model can classify text according to the influence of different role speech, thereby improving the accuracy of text classification.

[0171] In one embodiment of the present invention, at least one round of dialogue text content includes multi-round dialogue text content, wherein feature extraction further includes extracting dialogue round number features, which are used to identify the round in which each round of dialogue content in the multi-round dialogue text content is located.

[0172] In one embodiment of the invention, text classification includes intent classification.

[0173] The specific implementation process of the functions and roles of each module in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0174] Figure 11 The diagram shown is a block diagram of an electronic device 1100 provided in an embodiment of the present invention.

[0175] Reference Figure 11 The electronic device 1100 includes a processing component 1110, which further includes one or more processors, and memory resources represented by memory 1120 for storing instructions executable by the processing component 1110, such as application programs. The application programs stored in memory 1120 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1110 is configured to execute instructions to perform the training method or text classification method of the aforementioned text classification model.

[0176] Electronic device 1100 may also include a power supply component configured to perform power management of electronic device 1100, a wired or wireless network interface configured to connect electronic device 1100 to a network, and an input / output (I / O) interface. Electronic device 1100 can operate on an operating system, such as Windows Server, stored in memory 1120. TM macOS X TM Unix TM Linux TM FreeBSD TM Or similar.

[0177] A non-transitory computer-readable storage medium, wherein when the instructions in the storage medium are executed by the processor of the aforementioned electronic device 1100, the aforementioned electronic device 1100 is able to execute a training method for a text classification model or a text classification method.

[0178] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0179] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0180] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0181] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0182] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0183] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program verification codes.

[0184] Furthermore, it should be noted that the combination of the various technical features in this case is not limited to the combination methods described in the claims of this case or the combination methods described in the specific embodiments. All technical features described in this case can be freely combined or combined in any way, unless they contradict each other.

[0185] It should be noted that the above examples are merely specific embodiments of the present invention, and the present invention is obviously not limited to the above embodiments, with many similar variations. All modifications that can be directly derived or conceived by those skilled in the art from the content disclosed in this invention should fall within the protection scope of this invention.

[0186] It should be understood that the terms "first," "second," etc., mentioned in the embodiments of the present invention are merely for the purpose of more clearly describing the use of the technical solutions in the embodiments of the present invention, and are not intended to limit the scope of protection of the present invention.

[0187] The above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A training method for a text classification model, characterized in that, include: Obtain pre-training samples, wherein the pre-training samples include k rounds of dialogue text content labeled with different dialogue roles, wherein the first t rounds of dialogue in the k rounds of dialogue text content are context text information, and the last kt rounds of dialogue in the k rounds of dialogue text content are response text information, wherein the response text information is the response text corresponding to the context relationship of the context text information; k is an integer greater than 1, and t is an integer less than k; one question-and-answer process between different dialogue roles is one round of dialogue; Feature extraction is performed on the pre-trained samples to obtain feature vectors, wherein the feature extraction includes role feature extraction; The loss value is obtained based on the feature vector, and the initial model is pre-trained based on the loss value to obtain a pre-trained model; The parameters of the pre-trained model are fine-tuned to obtain a text classification model; The step of obtaining a loss value based on the feature vector and pre-training the initial model based on the loss value to obtain a pre-trained model includes: Based on the feature vector, obtain the probability that the response text information is the response text information of the context text information, and obtain a first loss value based on the probability; and The initial model is pre-trained based on the first loss value to obtain the pre-trained model.

2. The method according to claim 1, characterized in that, The step of obtaining the probability that the response text information is the response text information of the context text information based on the feature vector, and obtaining the first loss value based on the probability, includes: The feature vector is input into the initial model, and the initial model is used to extract context text features from the context text information to obtain a context feature matrix. The initial model is also used to extract response text features from the response text information to obtain a response feature matrix. Based on the context feature matrix and the response feature matrix, the probability that the response text information is a response text corresponding to the context relationship of the context text information is obtained; The first loss value is obtained using the loss function based on the probability.

3. The method according to claim 1, characterized in that, The loss value includes a second loss value, wherein the method further includes: Masking is performed on some words in the pre-trained samples to obtain the masked text information; The step of extracting features from the pre-trained samples to obtain feature vectors includes: The text information behind the mask is subjected to feature extraction to obtain the feature vector; The step of obtaining a loss value based on the feature vector and pre-training the initial model based on the loss value to obtain a pre-trained model includes: The predicted mask position is obtained based on the feature vector, and a second loss value is obtained. The initial model is pre-trained based on the second loss value to obtain the pre-trained model.

4. The method according to claim 1, characterized in that, The step of fine-tuning the parameters of the pre-trained model to obtain a text classification model includes: Obtain training samples, wherein the training samples include dialogue text information and classification labels, and the dialogue text information includes at least one round of dialogue text content labeled with different dialogue roles; The pre-trained model is used to extract features from the dialogue text information to obtain a feature vector, wherein the feature extraction includes role feature extraction; Text classification is performed based on the feature vector to obtain a classification prediction result, and a third loss value is obtained based on the classification prediction result and the classification label; The parameters of the pre-trained model are fine-tuned based on the third loss value to obtain the text classification model.

5. The method according to any one of claims 1 to 4, characterized in that, The feature extraction also includes dialogue turn number feature extraction, which is used to identify the turn number of each dialogue content in the k-turn dialogue text content.

6. A text classification method, characterized in that, include: Obtain dialogue text information, wherein the dialogue text information includes at least one round of dialogue text content annotated with different dialogue roles; Feature extraction is performed on the dialogue text information to obtain a feature vector, wherein the feature extraction includes role feature extraction; The feature vector is input into a text classification model to obtain a classification result; wherein the text classification model is trained according to the method described in any one of claims 1-5.

7. The method according to claim 6, characterized in that, The at least one round of dialogue text content includes multi-round dialogue text content, wherein the feature extraction further includes extracting dialogue round number features, which are used to identify the round in which each round of dialogue content is located in the multi-round dialogue text content.

8. A text classification device, characterized in that, include: The acquisition module is used to acquire dialogue text information, wherein the dialogue text information includes at least one round of dialogue text content labeled with different dialogue roles; The feature extraction module is used to extract features from the dialogue text information to obtain a feature vector, wherein the feature extraction includes role feature extraction; A classification module is used to input the feature vector into a text classification model to obtain a classification result; wherein the text classification model is trained according to the method described in any one of claims 1-5.

9. A computer-readable storage medium having computer-executable instructions stored thereon, characterized in that, When the executable instructions are executed by the processor, they implement the method as described in any one of claims 1 to 7.

10. An electronic device, characterized in that, The electronic device includes: processor; Memory used to store the processor's executable instructions; The processor is used to perform the method according to any one of claims 1 to 7.

Images