Chinese text classification method fusing label confusion and pinyin information

Abstract

The invention discloses a Chinese text classification method fusing label confusion and pinyin information. The method comprises the following three main parts: carrying out label confusion operation on sample labels to generate label simulation distribution to replace traditional one-hot representation, fusing pinyin information of samples into character representation, and constructing comparative learning to deal with the noise problem; the label confusion distribution is obtained by calculating the similarity value of the labels and the samples, and the learned simulated label distribution is beneficial for the model to better represent instances with different labels; the pronunciation of the Chinese character is represented by embedding the pinyin of the sample, and the pronunciation is fused into the character embedding of the sample, so that the highly popular polyphone phenomenon in the Chinese is solved; then, proper positive and negative sample data are constructed for comparative learning, in the process of zooming the distance between the positive and negative samples, features of real classification are learned, meanwhile, interference of noise features is reduced, and the robustness of the model is improved.

Description

Technical Field

[0001] The present invention belongs to the field of text classification, and specifically relates to Chinese text classification, label confusion, neural network and contrastive learning method. Background Art

[0002] In text classification, the feature representation of text is to convert text into a digital sequence that can be understood by computers. Before the emergence of word vector technology, common text feature representations included: word frequency features, N-gram features, features selected by feature selection methods, etc. These methods have a single information extraction capability, and the information contained in the obtained text representation is limited. Later, one-hot encoding appeared, and this method was widely used in the industry, but as the amount of data increases, the problem of dimensionality explosion will occur. Nowadays, pre-trained models based on deep networks such as GPT, Bert series, ERNIE, and XLM series are widely popular. In Chinese text, the same character may have very different meanings depending on the different pinyins. How to better integrate the characteristics of Chinese text into traditional text feature representation is a challenging task.

[0003] In text classification tasks, labels are usually represented by one-hot, that is, the corresponding category is 1 and other categories are 0. However, when calculating the loss, it is found that only the dimension with a value of 1 is involved in the loss calculation. This causes the relationship between the true label and other labels to be ignored, and many useful features cannot be learned. This situation is more likely to be affected in easily confused classification tasks. Summary of the invention

[0004] In order to solve the above technical problems, the present invention designs a Chinese text classification method that integrates label confusion and pinyin information. The method firstly represents the text through the BERT pre-training model; and by integrating pinyin embedding, the embedding can obtain more information to better deal with the situation of polyphones; then, the label confusion module can make the label closer to the real label, avoid the model being more "arbitrary", and prevent the overfitting problem.

[0005] In order to achieve the above technical effects, the present invention discloses a Chinese text classification method integrating label confusion and pinyin information, which specifically includes the following steps:

[0006] S1: Use the BERT pre-trained model to represent sentences;

[0007] S2: Represent the phonetic information of the sentence and integrate it into the representation of the sentence;

[0008] S3: Add denoising processing: Design a contrastive learning task and use appropriate positive and negative sample data for contrastive learning. In the process of increasing the distance between positive and negative samples and increasing the distance between positive samples, the features that truly divide the categories can be learned.

[0009] S4: Use the label confusion module to calculate the similarity between the instance and the label to generate a better label distribution to replace the original one-hot label vector;

[0010] S5: Design a feature extraction network to obtain the predicted label distribution;

[0011] S6: Construct supervision tasks and contrastive learning for joint optimization;

[0012] S7: Get the classification result.

[0013] Furthermore, the pre-trained word vector model selected in S1 is the BERT-Base-Chinese model in the BERT pre-trained deep language model; it learns language representation through a bidirectional Transformer architecture, including a 12-layer Transformer encoder, each layer has 12 self-attention heads, a total of 110M parameters, pre-trained on Chinese text, and can learn the semantic and grammatical information of Chinese text.

[0014] Furthermore, the pinyin embedding in S2 uses the open source pypinyin package to generate a pinyin sequence for each character, uses special markers to represent tones, and then appends these tones to the end of the pinyin sequence; a CNN model with a width of 2 is applied to the pinyin sequence, and then the generated pinyin embedding is derived using maximum pooling, which makes the output dimension unaffected by the length of the input pinyin sequence; the length of the input pinyin sequence is fixed to 8, and when the actual length of the pinyin sequence does not reach 8, the remaining slots are filled with a special letter "-"; the two D-dimensional embeddings of the character embedding and the pinyin embedding after contrastive learning are input into the fusion layer; the two embeddings are first concatenated and then mapped to the D-dimensional embedding through a fully connected layer to form a fused embedding.

[0015] Furthermore, the contrastive learning strategy adopted in step 3 is: in traditional contrastive learning, it assumes a set of paired instances where x i and are semantically related; use standard dropout as noise, pass the same sentence twice to the pre-trained encoder, apply standard dropout twice to obtain two different embeddings, and then integrate the pinyin information to make the two fused embeddings as "positive pairs", and take the fused embeddings of sentences with different labels in the same batch as "negative pairs"; take a sentence set as and make The key to getting the same alignment is to use x i and The independent sampling dropout mask of where z is the random mask of dropout; we only need to feed the same input to the encoder twice to get two fused embeddings z and z' with different dropout masks; the training objective of contrastive learning is:

[0016]

[0017] Where τ is the temperature hyperparameter and sim(h1,h2) is the cosine similarity N is the number of sentences in the batch that have different labels than the input sentence.

[0018] Further, in S4, a label representation matrix is ​​generated by a label encoder, which is a deep neural network (DNN), and then a label simulation distribution calculation block (SLD Block) is used to calculate the similarity between the label and the instance to obtain the label confusion distribution; the SLD block consists of a similarity layer and an SLD calculation layer; the similarity layer takes the fusion embedding of the label representation and the current instance as input, and calculates their similarity value through dot product, and then applies a neural network with softmax activation to obtain the label confusion distribution (LCD); finally, the one-hot vector of the original label is added to the LCD, the parameter α is controlled, and then normalized by the softmax function to generate the simulated label distribution SLD; the control parameter α determines how much the LCD will change the one-hot vector; the above process can be expressed as:

[0019]

[0020] y (c) =softmax(v (i)T V (l) W+b)

[0021] y (s) =softmax(αy (t) +y (c) )

[0022] where f L is the label encoding function used to convert the label l = [l1,l2,…,l C ] is converted to a label representation matrix V (l) , C is the number of categories; f L Implemented by an embedding lookup layer and a DNN, which can be a multi-layer perceptron (MLP), LSTM, or attention network; the order of the label sequence input to LCM should be the same as the one-hot target. (c) It is LCD,(s) is SLD, y (t) is the label one-hot distribution.

[0023] Furthermore, the feature extraction network in step 5 can directly input the fused embedding into the fully connected layer to obtain a C-dimensional fused embedding, and then obtain the predicted label distribution through the softmax classifier. TextCNN, TextRCNN, TextRNN, DPCNN models can be used as feature extraction network layers to obtain predicted label distribution.

[0024] Furthermore, in S6, the simulated label distribution SLD obtained in S4 is regarded as a new training target to replace the original label distribution one-hot vector and supervise the model training; since SLD y (s) and the predicted label vector y (p) They are all probability distributions, so we use Kullback–Leibler divergence (KL-divergence) as the loss function to measure their differences:

[0025]

[0026] Where C is the number of label categories, y c (S) To simulate the probability of label distribution on category c, y c (p) Represents the probability of predicting the label distribution on category c;

[0027] Joint loss function L Loss It is the KL loss plus the contrastive learning loss, using the following formula:

[0028] L Loss =L KL +βL CL

[0029] Here, β is a hyperparameter that controls the trade-off.

[0030] Furthermore, after the predicted label distribution is obtained in step 7, the dimension with the largest value is taken as the classification result.

[0031] The beneficial effects of the present invention are:

[0032] The present invention provides a Chinese text classification method that integrates label confusion and pinyin information. First, after BERT pre-training, character embedding is integrated with pinyin embedding to form a fusion embedding, which enhances the representation ability of Chinese corpus and can better handle the phenomenon that the same character in Chinese text has different pronunciations and different meanings (polyphones).

[0033] Secondly, this method obtains two different embeddings as “positive pairs” for comparative learning by applying two standard dropouts. The method is simple and effective, which can reduce the interference of noise features while improving the robustness of the model.

[0034] Finally, the simulated label distribution is obtained through the label confusion module to replace the one-hot vector and supervise the model. It is better than the distribution that only considers the similarity between labels, and the uniform noise distribution in the label smoothing LS. Especially for those samples that are easily confused. SLD is also more robust when facing noisy data, because the probability of wrong labels is assigned to similar labels, so the model can still learn some useful information from the wrongly labeled data. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solution of the embodiment of the present invention, the drawings required for describing the embodiment are briefly introduced below.

[0036] Figure 1 It is a flowchart of a Chinese text classification method that integrates label confusion and pinyin information;

[0037] Figure 2 It is a schematic diagram of the overall model framework of the Chinese text classification method that integrates label confusion and pinyin information;

[0038] Figure 3 It is a character embedding diagram of the Chinese text classification method that integrates label confusion and pinyin information;

[0039] Figure 4 It is a schematic diagram of pinyin embedding of a Chinese text classification method that integrates label confusion and pinyin information;

[0040] Figure 5 It is the architecture of the label confusion module of the Chinese text classification method that integrates label confusion and pinyin information. DETAILED DESCRIPTION

[0041] The present invention discloses a Chinese text classification method integrating label confusion and pinyin information. The overall process of the method is as follows: Figure 1 As shown in the figure, the overall model framework is as follows: Figure 2 As shown. Specifically including the following steps:

[0042] S1: Use the BERT pre-trained model to represent sentences. The sentence character embedding process is as follows Figure 3As shown, the selected pre-trained word vector model is the BERT-Base-Chinese model in the BERT pre-trained deep language model. It learns language representations through a bidirectional Transformer architecture, including 12 layers of Transformer encoders, with 12 self-attention heads in each layer, and a total of 110M parameters. It has been pre-trained on Chinese texts and can learn semantic and syntactic information of Chinese texts.

[0043] S2: Characterize the pinyin information of the sentence and integrate it into the sentence representation. The specific pinyin embedding process is as Figure 4 shown, which includes:

[0044] S2.1: Use the open-source pypinyin package to generate a pinyin sequence for each character, use special markers to represent tones, and then append these tones to the end of the pinyin sequence. Use the numbers 1, 2, 3, 4 to represent tones; for example, when a pinyin is in the first tone, then append the number 1 to the end of the sequence.

[0045] S2.2: Apply a CNN model with a width of 2 to the pinyin sequence, and then use max pooling to derive the generated pinyin embedding. The length of the input pinyin sequence is fixed at 8. When the actual length of the pinyin sequence does not reach 8, the remaining slots are filled with a special letter "-". As Figure 4 shown in, the input pinyin sequence of the character "新" does not reach the fixed length, and the remaining dimensions are filled with "-" as {x, i, n, 1, -, -, -, -}.

[0046] S2.3: Input these two D-dimensional embeddings, the character embedding and the pinyin embedding after contrastive learning, into the fusion layer. These two embeddings are first concatenated and then mapped to a D-dimensional embedding through a fully connected layer to form a fused embedding.

[0047] S3: Add denoising processing: Design a contrastive learning task and perform contrastive learning using appropriate positive and negative sample data. It assumes a set of paired instances where x i and are semantically related. In the process of widening the distance between positive and negative samples and narrowing the distance between positive samples, the features that truly divide the categories can be learned. The contrastive learning strategy adopted is: only use standard dropout as noise, pass the same sentence twice to the pre-trained encoder, and obtain two different embeddings by applying two standard dropouts. Then integrate the pinyin information to make the two fused embeddings as "positive pairs", and take the fused embeddings of sentences with different labels in the same batch as "negative pairs". This contrastive learning strategy is simple and easy to implement while also achieving good results. At the same time, this change slightly perturbs or enhances the input data, allowing the model to learn richer feature representations, improve the generalization ability of the model, and reduce the risk of overfitting. Take a sentence set as and make The key to getting the same alignment is to use x i and The independent sampling dropout mask of where z is the random mask of dropout. Simply feed the encoder with the same input twice to get two fused embeddings z and z' with different dropout masks. Then the training objective of contrastive learning is:

[0048]

[0049] Among them, τ is the temperature hyperparameter, sim(h1,h2) is the cosine similarity N is the number of sentences in the batch that have different labels than the input sentence.

[0050] S4: In Chinese datasets, many texts have features of multiple labels. For example, in an instance of misprediction, the news "Loan dispute leads to a case where a real estate developer used false information to develop a property" was labeled as "society", while the model predicted it as "real estate". This is because the news has both the features of "society" and "real estate" news. The traditional one-hot label vector is only 1 in the "society" dimension and 0 in other categories, so the model does not consider information from other categories when calculating the loss function, which makes our model too arbitrary. Therefore, the label confusion module is used to calculate the similarity between the instance and the label to generate a better label distribution to replace the original one-hot label vector. The specific process is as follows: Figure 5 As shown, the steps include:

[0051] S4.1: Generate a label representation matrix through a label encoder, which is a deep neural network (DNN).

[0052] S4.2: The label confusion distribution is obtained by calculating the similarity between the label and the instance through the label simulation distribution calculation block (SLD Block). The SLD block consists of a similarity layer and an SLD calculation layer. The similarity layer takes the fused embedding of the label representation and the current instance as input, calculates their similarity value through dot product, and then applies a neural network with softmax activation to obtain the label confusion distribution (LCD).

[0053] S4.3: Add the one-hot vector of the original label to LCD, control parameter α, and then normalize it through the softmax function to generate the simulated label distribution SLD. The control parameter α determines how much LCD will change the one-hot vector.

[0054] The above process can be expressed as:

[0055]

[0056] y (c) =softmax(v (i)T V (l) W+b)

[0057] y (s) =softmax(αy (t) +y (c) )

[0058] where f L is the label encoding function used to convert the label l = [l1,l2,…,l C ] is converted to a label representation matrix V (l) , C is the number of categories. L It is implemented by an embedding lookup layer and a DNN, which can be a multi-layer perceptron (MLP), LSTM, or attention network. The order of the label sequence input to LCM should be the same as the one-hot target. (c) It is LCD, (s) is SLD, y (t) is the label one-hot distribution.

[0059] Step 5: Design a feature extraction network to obtain the predicted label distribution. The feature extraction network can directly input the fused embedding into the fully connected layer to obtain the C-dimensional fused embedding, and then obtain the predicted label distribution through the softmax classifier. You can also use TextCNN, TextRCNN, TextRNN, DPCNN and other models as the feature extraction network layer to obtain the predicted label distribution.

[0060] Step 6: Construct a supervised task and perform joint optimization with contrastive learning. We treat the simulated label distribution SLD obtained in step 4 as a new training target to replace the original label distribution one-hot vector and supervise model training. (s) and the predicted label vector y (p) They are all probability distributions, so we use Kullback–Leibler divergence (KL-divergence) as the loss function to measure their differences:

[0061]

[0062] Joint loss function L Loss It is the KL loss plus the contrastive learning loss, using the following formula:

[0063] L Loss =L KL +βL CL

[0064] where β is a hyperparameter that controls the trade-off.

[0065] Step 7: After obtaining the classification result and the predicted label distribution, the dimension with the largest value is taken as the classification result.

[0066] The preferred embodiments of the present invention disclosed above are only used to help explain the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described.

Claims

1. A Chinese text classification method integrating label confusion and pinyin information, characterized in that: The specific steps include: S1: Use the BERT pre-trained model to represent sentences; S2: Represent the phonetic information of the sentence and integrate it into the representation of the sentence; S3: Add denoising processing: Design a contrastive learning task and use appropriate positive and negative sample data for contrastive learning. In the process of increasing the distance between positive and negative samples and increasing the distance between positive samples, the features that truly divide the categories can be learned. S4: Use the label confusion module to calculate the similarity between the instance and the label to generate a better label distribution to replace the original one-hot label vector; S5: Design a feature extraction network to obtain the predicted label distribution; S6: Construct supervision tasks and contrastive learning for joint optimization; S7: Get the classification result.

2. The Chinese text classification method integrating label confusion and pinyin information according to claim 1 is characterized in that: The pre-trained model selected in S1 is the BERT-Base-Chinese model in the BERT pre-trained deep language model; it learns language representation through a bidirectional Transformer architecture, including a 12-layer Transformer encoder, each layer has 12 self-attention heads, and a total of 110M parameters. It is pre-trained on Chinese text and can learn the semantic and grammatical information of Chinese text.

3. The Chinese text classification method integrating label confusion and pinyin information according to claim 1 is characterized in that: In S2, S2.1: Use the open source pypinyin package to generate pinyin sequences for each character, use special markers to represent tones, and then append these tones to the end of the pinyin sequence; S2.2: Apply a CNN model with a width of 2 on the pinyin sequence, and then use max pooling to derive the generated pinyin embedding. The length of the input pinyin sequence is fixed to 8. When the actual length of the pinyin sequence does not reach 8, the remaining slots are filled with a special letter "-"; S2.3: The two D-dimensional embeddings, the character embedding and the pinyin embedding after contrastive learning, are input into the fusion layer; the two embeddings are first concatenated and then mapped to the D-dimensional embedding through the fully connected layer to form the fused embedding.

4. The Chinese text classification method integrating label confusion and pinyin information according to claim 1 is characterized in that: The contrastive learning strategy adopted in step 3 is: In traditional contrastive learning, it assumes a set of paired instances where x i and are semantically related; use standard dropout as noise, pass the same sentence twice to the pre-trained encoder, apply standard dropout twice to obtain two different embeddings, and then integrate the pinyin information to make the two fused embeddings as "positive pairs", and take the fused embeddings of sentences with different labels in the same batch as "negative pairs"; take a sentence set as and make The key to getting the same alignment is to use x i and The independent sampling dropout mask of where z is the random mask of dropout; we only need to feed the same input to the encoder twice to get two fused embeddings z and z' with different dropout masks; the training objective of contrastive learning is: Where τ is the temperature hyperparameter and sim(h1,h2) is the cosine similarity N is the number of sentences in the batch that have different labels than the input sentence.

5. The Chinese text classification method integrating label confusion and pinyin information according to claim 1 is characterized in that: In S4, S4.1: Generate a label representation matrix through a label encoder, which is a deep neural network; S4.2: The label confusion distribution is obtained by calculating the similarity between the label and the instance through the label simulation distribution calculation block SLD Block; the SLD block consists of a similarity layer and an SLD calculation layer; the similarity layer takes the fused embedding of the label representation and the current instance as input, and calculates their similarity value through dot product, and then applies a neural network with softmax activation to obtain the label confusion distribution LCD; S4.3: Add the one-hot vector of the original label to the LCD, Control parameter α , and then normalized by the softmax function to generate the simulated label distribution SLD; the control parameter α determines how much the LCD will change the one-hot vector; The above process can be expressed as: y (c) =softmax(v (i)T V (l) W+b) and (s) =softmax(αy (t) +y (c) ) Among them, f L is the label encoding function used to convert the label l = [l1,l2,…,l C ] is converted to a label representation matrix V (l) , C is the number of categories; y (c) It is LCD, (s) is SLD, y (t) is the label one-hot distribution.

6. The Chinese text classification method integrating label confusion and pinyin information according to claim 1 is characterized in that: The feature extraction network in step 5 directly inputs the fused embedding into the fully connected layer to obtain a C-dimensional fused embedding, and then obtains the predicted label distribution through a softmax classifier.

7. The Chinese text classification method integrating label confusion and pinyin information according to claim 1 is characterized in that: In S6, the simulated label distribution SLD obtained in S4 is regarded as a new training target to replace the original label distribution one-hot vector and supervise the model training; since SLD y (s) and the predicted label vector y (p) They are all probability distributions, so we use Kullback–Leibler divergence (KL-divergence) as the loss function to measure their differences: In the formula, C is the number of label categories, y c (S) To simulate the probability of label distribution on category c, y c (p) Represents the probability of predicting the label distribution on category c; Joint loss function L Loss It is the KL loss plus the contrastive learning loss, using the following formula: L Loss =L KL +βL CL Here, β is a hyperparameter that controls the trade-off.

8. The Chinese text classification method integrating label confusion and pinyin information according to claim 1 is characterized in that: After the predicted label distribution is obtained in step 7, the dimension with the largest value is taken as the classification result.

Images