Chinese text classification method integrating label confusion and pinyin information

Abstract

The present invention discloses a Chinese text classification method that integrates label confusion and pinyin information. The method includes three main parts: performing label confusion on sample labels to generate label simulation distribution to replace traditional one-hot representation, incorporating the pinyin information of the sample into the character representation, and constructing contrastive learning to deal with the noise problem; obtaining the label confusion distribution by calculating the similarity value between the label and the sample, and the learned simulated label distribution helps the model better represent instances with different labels; representing the pronunciation of Chinese characters by embedding the sample pinyin and incorporating it into the character embedding of the sample, thereby solving the highly prevalent phenomenon of polyphones in Chinese; then constructing appropriate positive and negative sample data for contrastive learning. In the process of widening the distance between the positive and negative samples, the features that truly divide the categories are learned, while reducing the interference of noise features and improving the robustness of the model.

Description

Technical Field

[0001] The present invention belongs to the field of text classification, and specifically relates to Chinese text classification, label confusion, neural networks and contrastive learning methods. Background Art

[0002] In text classification, text feature representation converts text into a numerical sequence that computers can understand. Before the advent of word embedding technology, common text feature representations included word frequency features, N-gram features, and features selected using feature selection methods. These methods had limited information extraction capabilities, and the resulting text representations contained limited information. Later, one-hot encoding emerged and was initially widely used in industry. However, as data volumes increased, it faced the problem of dimensionality explosion. Currently, deep network-based pre-trained models such as GPT, the BERT series, ERNIE, and the XLM series are widely popular. However, in Chinese text, the meaning of the same character can vary significantly depending on the pinyin. Therefore, integrating the characteristics of Chinese text into traditional text feature representations is a challenging task.

[0003] In text classification tasks, labels are usually represented in one-hot format, that is, the corresponding category is 1 and other categories are 0. However, when calculating the loss, it is found that only the dimension with a value of 1 is involved in the loss calculation. This results in the relationship between the true label and other labels being ignored, and many useful features cannot be learned. This situation is more likely to be affected in easily confused classification tasks. Summary of the Invention

[0004] In order to solve the above technical problems, the present invention designs a Chinese text classification method that integrates label confusion and pinyin information. This method first characterizes the text through the BERT pre-training model; and by integrating pinyin embedding, the embedding can obtain more information to better deal with the situation of polyphones; then, the label confusion module can make the label closer to the real label, avoid the model being more "arbitrary", and prevent overfitting problems.

[0005] In order to achieve the above technical effects, the present invention discloses a Chinese text classification method that integrates label confusion and pinyin information, which specifically includes the following steps:

[0006] S1: Use the BERT pre-trained model to represent sentences;

[0007] S2: Represent the phonetic information of the sentence and integrate it into the representation of the sentence;

[0008] S3: Add denoising: Design a contrastive learning task and use appropriate positive and negative sample data for contrastive learning. By increasing the distance between positive and negative samples and closing the distance between positive samples, we can learn the features that truly divide the categories.

[0009] S4: Use the label confusion module to calculate the similarity between the instance and the label to generate a better label distribution to replace the original one-hot label vector;

[0010] S5: Design a feature extraction network to obtain the predicted label distribution;

[0011] S6: Constructing supervision tasks and contrastive learning for joint optimization;

[0012] S7: Get the classification result.

[0013] Furthermore, the pre-trained word vector model selected in S1 is the BERT-Base-Chinese model in the BERT pre-trained deep language model; it learns language representation through a bidirectional Transformer architecture, including a 12-layer Transformer encoder, each with 12 self-attention heads, with a total of 110M parameters. It is pre-trained on Chinese text and can learn the semantic and grammatical information of Chinese text.

[0014] Furthermore, the pinyin embedding in S2 uses the open source pypinyin package to generate a pinyin sequence for each character, uses special markers to represent tones, and then appends these tones to the end of the pinyin sequence; a CNN model with a width of 2 is applied to the pinyin sequence, and then maximum pooling is used to derive the generated pinyin embedding, which makes the output dimension unaffected by the length of the input pinyin sequence; the length of the input pinyin sequence is fixed to 8, and when the actual length of the pinyin sequence does not reach 8, the remaining slot is filled with a special letter "-"; the two D-dimensional embeddings of the character embedding and the pinyin embedding after contrastive learning are input into the fusion layer; the two embeddings are first concatenated and then mapped to the D-dimensional embedding through a fully connected layer to form a fused embedding.

[0015] Furthermore, the contrastive learning strategy adopted in step 3 is: In traditional contrastive learning, it assumes a set of paired instances where x i and are semantically related; use standard dropout as noise, pass the same sentence twice to the pre-trained encoder, obtain two different embeddings by applying standard dropout twice, and then integrate the pinyin information to make the two fused embeddings as "positive pairs", and take the fused embeddings of sentences with different labels in the same batch as "negative pairs"; take a sentence set as and make The key to getting the same alignment is to use x i and The independent sampling dropout mask of where z is the random mask of dropout; we only need to feed the same input to the encoder twice to obtain two fused embeddings z and z' with different dropout masks; the training objective of contrastive learning is:

[0016]

[0017] Where τ is the temperature hyperparameter and sim(h1,h2) is the cosine similarity N is the number of sentences in the batch that have different labels than the input sentence.

[0018] Furthermore, in S4, a label representation matrix is ​​generated by a label encoder, which is a deep neural network (DNN). Then, a label simulation distribution calculation block (SLD Block) is used to calculate the similarity between the label and the instance to obtain the label confusion distribution; the SLD block consists of a similarity layer and an SLD calculation layer; the similarity layer takes the fused embedding of the label representation and the current instance as input, and calculates their similarity value through dot product, and then applies a neural network with softmax activation to obtain the label confusion distribution (LCD); finally, the one-hot vector of the original label is added to the LCD, the parameter α is controlled, and then normalized by the softmax function to generate the simulated label distribution SLD; the control parameter α determines how much the LCD will change the one-hot vector; the above process can be expressed as:

[0019]

[0020] y (c) =softmax(v (i)T V (l) W+b)

[0021] y (s) =softmax(αy (t) +y (c) )

[0022] where f L It is the label encoding function used to convert the label l=[l1,l2,…,l C ] is converted to a label representation matrix V (l) , C is the number of categories; f L Implemented by an embedding lookup layer and a DNN, which can be a multi-layer perceptron (MLP), LSTM, or attention network; the order of the label sequence input to LCM should be the same as the one-hot target. (c) It is LCD,(s) It is SLD, y (t) is the label one-hot distribution.

[0023] Furthermore, the feature extraction network in step 5 can directly input the fusion embedding into the fully connected layer to obtain a C-dimensional fusion embedding, and then obtain the predicted label distribution through the softmax classifier. The TextCNN, TextRCNN, TextRNN, and DPCNN models can be used as the feature extraction network layer to obtain the predicted label distribution.

[0024] Furthermore, in S6, the simulated label distribution SLD obtained in S4 is regarded as a new training target to replace the original label distribution one-hot vector and supervise the model training; since SLD y (s) and the predicted label vector y (p) They are all probability distributions, so we use Kullback–Leibler divergence (KL-divergence) as the loss function to measure their differences:

[0025]

[0026] Where C is the number of label categories, y c (S) To simulate the probability of label distribution on category c, y c (p) Represents the probability of predicting the label distribution on category c;

[0027] Joint loss function L Loss The KL loss is added to the contrastive learning loss, using the following formula:

[0028] L Loss =L KL +βL CL

[0029] Here, β is a hyperparameter that controls the trade-off.

[0030] Furthermore, after the predicted label distribution is obtained in step 7, the dimension with the largest value is taken as the classification result.

[0031] The beneficial effects of the present invention are:

[0032] This paper provides a Chinese text classification method that integrates label obfuscation and pinyin information. First, after BERT pre-training, character embeddings are integrated with pinyin embeddings to form a fused embedding, which enhances the representation ability of Chinese corpus and better handles the phenomenon that the same character in Chinese text has different pronunciations and different meanings (polyphones).

[0033] Secondly, this method obtains two different embeddings as "positive pairs" for comparative learning by applying two standard dropouts. This method is simple and effective, while reducing the interference of noise features and improving the robustness of the model.

[0034] Finally, a simulated label distribution is generated through the label confusion module to replace the one-hot vectors and supervise the model. This is superior to distributions that only consider the similarity between labels, as well as the uniform noise distribution in label smoothing (LS). This is particularly true for samples that are easily confused. SLD is also more robust to noisy data because the probability of incorrect labels is assigned to similar labels, allowing the model to still learn useful information from incorrectly labeled data. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments.

[0036] Figure 1 This is a flowchart of the Chinese text classification method that integrates label confusion and pinyin information;

[0037] Figure 2 This is a schematic diagram of the overall model framework of the Chinese text classification method that integrates label confusion and pinyin information;

[0038] Figure 3 This is a character embedding diagram of the Chinese text classification method that integrates label confusion and pinyin information;

[0039] Figure 4 This is a schematic diagram of pinyin embedding for the Chinese text classification method that integrates label confusion and pinyin information;

[0040] Figure 5 It is the architecture of the label confusion module of the Chinese text classification method that integrates label confusion and pinyin information. DETAILED DESCRIPTION

[0041] The present invention discloses a Chinese text classification method that integrates label confusion and pinyin information. The overall process of the method is as follows: Figure 1 As shown in the figure, the overall model framework is as follows: Figure 2 The specific steps include:

[0042] S1: Use the BERT pre-trained model to represent the sentence. The sentence character embedding process is as follows Figure 3As shown, the selected pre-trained word vector model is the BERT-Base-Chinese model in the BERT pre-trained deep language model. It learns language representations through a bidirectional Transformer architecture, including 12 layers of Transformer encoders, with 12 self-attention heads in each layer, and a total of 110M parameters. It has been pre-trained on Chinese texts and can learn semantic and syntactic information of Chinese texts.

[0043] S2: Characterize the pinyin information of the sentence and incorporate it into the sentence representation. The specific pinyin embedding process is as Figure 4 shown, which includes:

[0044] S2.1: Use the open-source pypinyin package to generate a pinyin sequence for each character, use special tokens to represent tones, and then append these tones to the end of the pinyin sequence. Use the numbers 1, 2, 3, 4 to represent tones; for example, when a pinyin is in the first tone, then append the number 1 to the end of the sequence.

[0045] S2.2: Apply a CNN model with a width of 2 to the pinyin sequence, and then use max pooling to derive the generated pinyin embedding. The length of the input pinyin sequence is fixed at 8. When the actual length of the pinyin sequence does not reach 8, the remaining slots are filled with a special letter "-". As Figure 4 shown in, the input pinyin sequence of the character "新" does not reach the fixed length, so the remaining dimensions are filled with "-" as {x, i, n, 1, -, -, -, -}.

[0046] S2.3: Input these two D-dimensional embeddings, the character embedding and the pinyin embedding after contrastive learning, into the fusion layer. These two embeddings are first concatenated and then mapped to a D-dimensional embedding through a fully connected layer to form a fused embedding.

[0047] S3: Add denoising processing: Design a contrastive learning task and perform contrastive learning using appropriate positive and negative sample data. It assumes a set of paired instances where x i and Is semantically related. In the process of widening the distance between positive and negative samples and narrowing the distance between positive samples, the features that truly divide the categories can be learned. The contrastive learning strategy adopted is: only use standard dropout as noise, pass the same sentence twice to the pre-trained encoder, and obtain two different embeddings by applying two standard dropouts. Then, incorporate the pinyin information to make the two fused embeddings as "positive pairs", and take the fused embeddings of sentences with different labels in the same batch as "negative pairs". This contrastive learning strategy is simple and easy to implement while also achieving good results. At the same time, this change slightly perturbs or enhances the input data, allowing the model to learn richer feature representations, improve the generalization ability of the model, and reduce the risk of overfitting. Take a sentence set as and make The key to getting the same alignment is to use x i and The independent sampling dropout mask of Where z is the random mask of dropout. Simply feed the encoder the same input twice to obtain two fused embeddings z and z' with different dropout masks. The training objective of contrastive learning is then:

[0048]

[0049] Where τ is the temperature hyperparameter and sim(h1,h2) is the cosine similarity N is the number of sentences in the batch that have different labels than the input sentence.

[0050] S4: In Chinese datasets, many texts have features of multiple labels. For example, in an instance of a prediction error, the news "Loan dispute leads to a case of real estate developers using false information to develop real estate projects" was labeled as "social", while the model predicted the news as "real estate". This is because the news has both the features of "social" news and the features of "real estate" news. The traditional one-hot label vector is only 1 in the "social" category and 0 in other categories, so that the model does not consider information from other categories when calculating the loss function, which will cause our model to become too arbitrary. Therefore, the label confusion module is used to calculate the similarity between the instance and the label to generate a better label distribution to replace the original one-hot label vector. The specific process is as follows: Figure 5 As shown, the steps include:

[0051] S4.1: Generate a label representation matrix through a label encoder, which is a deep neural network (DNN).

[0052] S4.2: The Label Confusion Distribution (LCD) block is used to calculate the similarity between labels and instances. The SLD block consists of a similarity layer and an SLD calculation layer. The similarity layer takes the fused embedding of the label representation and the current instance as input and calculates their similarity via a dot product. A neural network with softmax activation is then applied to obtain the LCD.

[0053] S4.3: Add the original label's one-hot vector to the LCD, control the parameter α, and then normalize it through the softmax function to generate the simulated label distribution SLD. The control parameter α determines how much the LCD will change the one-hot vector.

[0054] The above process can be expressed as:

[0055]

[0056] y (c) =softmax(v (i)T V (l) W+b)

[0057] y (s) =softmax(αy (t) +y (c) )

[0058] where f L It is the label encoding function used to convert the label l=[l1,l2,…,l C ] is converted to a label representation matrix V (l) , C is the number of categories. f L It is implemented by an embedding lookup layer and a DNN, which can be a multi-layer perceptron (MLP), LSTM, or attention network. The order of the label sequence input to LCM should be the same as the one-hot target. (c) It is LCD, (s) It is SLD, y (t) is the label one-hot distribution.

[0059] Step 5: Design a feature extraction network to obtain the predicted label distribution. The feature extraction network can directly input the fused embedding into the fully connected layer to obtain a C-dimensional fused embedding, and then obtain the predicted label distribution through the softmax classifier. Alternatively, models such as TextCNN, TextRCNN, TextRNN, and DPCNN can be used as the feature extraction network layer to obtain the predicted label distribution.

[0060] Step 6: Construct a supervision task and perform joint optimization with contrastive learning. We regard the simulated label distribution SLD obtained in step 4 as a new training target to replace the original label distribution one-hot vector and supervise model training. (s) and the predicted label vector y (p) Both are probability distributions, so we use Kullback–Leibler divergence (KL-divergence) as the loss function to measure their differences:

[0061]

[0062] Joint loss function L Loss The KL loss is added to the contrastive learning loss, using the following formula:

[0063] L Loss =L KL +βL CL

[0064] where β is a hyperparameter that controls the trade-off.

[0065] Step 7: Get the classification result. After obtaining the predicted label distribution, take the dimension with the largest value as the classification result.

[0066] The preferred embodiments of the present invention disclosed above are only used to help illustrate the present invention. The preferred embodiments do not describe all details in detail, nor do they limit the invention to only the specific implementation methods described.

Claims

1. A Chinese text classification method integrating label confusion and pinyin information, characterized in that: The specific steps include: S1: Use the BERT pre-trained model to represent sentences; S2: Represent the phonetic information of the sentence and integrate it into the representation of the sentence; S3: Add denoising: Design a contrastive learning task and use appropriate positive and negative sample data for contrastive learning. By increasing the distance between positive and negative samples and closing the distance between positive samples, we can learn the features that truly divide the categories. The contrastive learning strategy adopted is: In traditional contrastive learning, it assumes a set of paired instances ,in and are semantically related; use standard dropout as noise, pass the same sentence twice to the pre-trained encoder, apply standard dropout twice to obtain two different embeddings, and then integrate the pinyin information to make the two fused embeddings as "positive pairs", and take the fused embeddings of sentences with different labels in the same batch as "negative pairs"; take a sentence set as and make ; The key to achieving the same alignment is to use and The independent sampling dropout mask of ,in is a random mask for dropout; simply feed the same input to the encoder twice to get two fused embeddings with different dropout masks and ; The training objectives of contrastive learning are: ， in is the temperature hyperparameter, is the cosine similarity , is the number of sentences in the batch that have different labels from the input sentence; S4: Use the label confusion module to calculate the similarity between the instance and the label to generate a better label distribution to replace the original one-hot label vector; S4.1: Generate a label representation matrix using a label encoder, which is a deep neural network. S4.2: The Label Confusion Distribution (LCD) is obtained by calculating the similarity between the label and the instance using the Label Simulation Distribution (SLD) Block. The SLD block consists of a similarity layer and an SLD calculation layer. The similarity layer takes the fused embedding of the label representation and the current instance as input and calculates their similarity via a dot product. A neural network with softmax activation is then applied to obtain the Label Confusion Distribution (LCD). S4.3: Add the original label's one-hot vector to the LCD, control parameter α, and then normalize it through the softmax function to generate the simulated label distribution SLD. The control parameter α determines how much the LCD will change the one-hot vector. The above process can be expressed as: ， ， ， in, The label encoding function is used to encode the label Convert to label representation matrix , is the number of categories; It's LCD. It is SLD, is the label one-hot distribution; S5: Design a feature extraction network to obtain the predicted label distribution; S6: Constructing supervision tasks and contrastive learning for joint optimization; S7: Get the classification result.

2. The Chinese text classification method integrating label confusion and pinyin information according to claim 1 is characterized in that: The pre-trained model selected in S1 is the BERT-Base-Chinese model in the BERT pre-trained deep language model; it learns language representation through a bidirectional Transformer architecture, including a 12-layer Transformer encoder, each with 12 self-attention heads, with a total of 110M parameters. It is pre-trained on Chinese text and can learn the semantic and grammatical information of Chinese text.

3. The Chinese text classification method integrating label confusion and pinyin information according to claim 1 is characterized in that: In the S2, S2.1: Use the open-source pypinyin package to generate a pinyin sequence for each character, using special markers to represent tones, and then append these tones to the end of the pinyin sequence; S2.2: Apply a CNN model with a width of 2 to the pinyin sequence, and then use max pooling to derive the generated pinyin embedding. The length of the input pinyin sequence is fixed to 8. When the actual length of the pinyin sequence does not reach 8, the remaining slots are filled with a special letter "-"; S2.3: The two D-dimensional embeddings, the character embedding and the pinyin embedding after contrastive learning, are input into the fusion layer; the two embeddings are first concatenated and then mapped to the D-dimensional embedding through a fully connected layer to form a fused embedding.

4. The Chinese text classification method integrating label confusion and pinyin information according to claim 1 is characterized in that: The feature extraction network in step 5 directly inputs the fused embedding into the fully connected layer to obtain a C-dimensional fused embedding, and then obtains the predicted label distribution through the softmax classifier.

5. The Chinese text classification method integrating label confusion and pinyin information according to claim 1 is characterized in that: In S6, the simulated label distribution SLD obtained in S4 is regarded as a new training target to replace the original label distribution one-hot vector and supervise the model training; since SLD and the predicted label vector They are all probability distributions, so we use Kullback–Leibler divergence (KL-divergence) as the loss function to measure their differences: ， Where C is the number of label categories, y c (S) To simulate the probability of label distribution on category c, y c (p) Represents the probability of predicting the label distribution on category c; Joint loss function The KL loss is added to the contrastive learning loss, using the following formula: ， in, is a hyperparameter that controls the trade-off.

6. The Chinese text classification method integrating label confusion and pinyin information according to claim 1 is characterized in that: After the predicted label distribution is obtained in step 7, the dimension with the largest value is taken as the classification result.

Images