A text classification method based on vocabulary projection hybrid label learning

Abstract

The application discloses a text classification method based on a vocabulary projection mixed label learning, which first enriches label information of a feature word vocabulary of a label based on labels and texts of all samples by using TF-IDF and attention, and carries out projection purification to reserve independence between each label; then, semantic information of a sample model is learned by using a semantic learning model; next, on the basis of one-hot coding, mixed learning between sample texts and each label is established by using semantic information and the label feature word vocabulary to obtain mixed learning labels; finally, semantic information is classified in an output layer, and KL divergence between the semantic information and the mixed learning labels is calculated as a final cross-entropy loss; since the label mixed learning model does not participate in the process of the semantic learning model and classification, the performance of model classification is not affected, and the feature extractor in the semantic learning model can be replaced by any model, so that the method has higher freedom and flexibility.

Description

Technical Field

[0001] This invention relates to the field of natural language text processing technology, and in particular to a text classification method based on lexical projection hybrid label learning. Background Technology

[0002] Text classification is a classic task in the field of natural language processing. Machine models can learn to infer labels or sets of labels for given text (sentences, documents, etc.). Its applications are widespread, including binary classification, multi-class classification, and multi-label classification. Text classification has broad applications in many fields, such as news classification, sentiment classification, and topic classification. For input text, network models such as recurrent neural networks (RNNs) and convolutional neural networks (CNNs) are used to learn the dense features of the text, and then this representation learning is used for the final classification. Existing research includes neural networks and embedding techniques, such as TextCNN (Convolutional Neural Networks for Sentence Classification), RNNs (Recurrent Neural Networks), Transformers, and BERT (Bidirectional Encoder Representations from Transformer).

[0003] In recent years, many methods have been proposed to improve the accuracy of text classification, such as Label Distribution Learning (LDL) and Label Smoothing (LS) in label enhancement techniques. LDL distribution labels consist of a one-hot encoded vector and label distribution noise. The noise data is generated by the number of samples distributed across each label, and the loss is obtained by calculating the distance similarity between the model's predicted label and the distributed label. LS label smoothing adds random noise to each dimension of the original one-hot encoded representation. Although simple and brute-force, it can prevent overfitting and is currently used in many image classification models. These methods can alleviate the problem of models relying too heavily on a single label to some extent and also have a certain degree of noise resistance, but they still have the following shortcomings:

[0004] (1) Using one-hot encoding as the final true label can easily lead to the model overfitting to a certain label class. This is especially true in confusing datasets with very similar labels and noisy datasets with mislabeled labels, which may reduce the classification accuracy of the model.

[0005] (2) Since labels are usually not completely independent, there may be relationships between labels, and samples may also be associated with multiple labels. One-hot encoding cannot fully reflect the relationship between samples and each label.

[0006] (3) The label distribution noise used in LDL is based on the distribution of samples on the labels. However, the real label distribution is often difficult to obtain, or even impossible to obtain, and can only be simulated. Therefore, LDL cannot be used directly. Label smoothing LS simply adds random noise to the labels, which cannot reflect the relationship between the labels, cannot effectively improve the classification accuracy of the model, and may have the risk of underfitting. Summary of the Invention

[0007] To address the aforementioned technical problems, the following technical solution is adopted: A text classification method based on lexical projection hybrid label learning is designed, including a training phase and a testing phase. The training phase includes the following steps:

[0008] S1 generates relevant word vectors based on all samples and divides them into training, validation, and test sets according to a ratio;

[0009] S2 uses the relevant word vectors to encode the labels and text of the samples, and constructs a purified vocabulary of label feature words;

[0010] S3 inputs the training set into the semantic learning model and uses a feature extractor to extract the semantic information of the samples;

[0011] S4 uses the purified tag feature word vocabulary and the semantic information of the sample text to perform hybrid learning of tags and samples, generating hybrid learning tags;

[0012] S4 inputs the semantic information of the sample into a one-layer fully connected neural network as a classifier for classification;

[0013] S5 uses the KL divergence calculation to calculate the class obtained by the classifier and the result of the mixed learning label as the cross-entropy loss. The parameters are updated by backpropagation through the Adam optimizer. After each parameter update, the value of the loss function on the validation set is calculated, and the parameters are continuously adjusted to make F1 optimal.

[0014] The testing phase includes the following steps:

[0015] S6 obtains the semantic information of the corresponding samples in the test set.

[0016] S7 inputs the test set into the semantic learning model and uses a feature extractor to extract the semantic information of the samples;

[0017] S8 inputs the semantic information of the sample into the trained classifier to obtain the text classification result.

[0018] Furthermore, including:

[0019] In step S1, generating relevant word vectors based on all samples includes:

[0020] All S11 samples used Stanford GloVe open-source word vector code to create word vectors, and added special characters to assist in classification;

[0021] S12 uses the format [label, content] for each sample and visualizes the length of each sample to obtain the maximum sentence length processed by the label hybrid learning model and the semantic learning model.

[0022] Furthermore, including:

[0023] In step S2, constructing the purified tag feature word vocabulary includes:

[0024] The original sample text for S21 is represented as follows:

[0025] , including tags and text The sentence length is All samples are encoded using word vectors, resulting in samples with the following dimensions: ,in, For the number of tags, The number of samples. The dimension is the word vector.

[0026] S22 calculates the attention coefficients between each label. ;

[0027] S23 uses a combination of TF-IDF and attention mechanisms to obtain feature word entries for each label based on all samples, and constructs a feature word vocabulary;

[0028] S24 uses attention coefficients to project and refine the feature word vocabulary for two-label feature word entries. and The purified feature word vocabulary is obtained by using the projection purification method to eliminate commonalities. The formula is shown below:

[0029]

[0030]

[0031] in, , This is the formula for vector projection. , Two vectors, through orthogonal projection

[0032]

[0033] Represented as Vector in The projection vector onto a vector;

[0034] Used to count, for The number of vector , It comes from The subsequent vector.

[0035] Furthermore, including:

[0036] Step S23 includes the following steps:

[0037] S231 For each sample statement The following formula is used to obtain the feature words with high TF-IDF frequency for the samples in the training set:

[0038]

[0039] For feature words with low frequency, an attention mechanism is used to acquire them, and the formula is as follows:

[0040] For low-frequency words, the vector encoded from the sentence characters... The formula for calculating self-attention is as follows:

[0041]

[0042] Obtain the attention coefficient of the word in the sentence, and take the word with the largest attention coefficient as the feature word;

[0043]

[0044] The feature words of each label obtained by combining TF-IDF and attention mechanism are merged to form the final label feature word entries, thus constructing the feature word entries. This feature term is based on the above. The calculated words are combined to obtain feature entries;

[0045] S232 constructs feature word entries for each type of label. This leads to the construction of a vocabulary of feature words; .

[0046] Furthermore, including:

[0047] Step S3 includes the following steps:

[0048] S31 inputs the dataset processed in step S12 into the semantic learning model to learn the semantic information of the samples. Specifically, position encoding is performed using the following formula:

[0049]

[0050]

[0051] in, This refers to the position of a word in a sentence. It refers to the word position at an even-numbered position. It refers to the position of the character in an odd-numbered position. It is the dimension of the word vector, and The combined encoding vector , dimension ;

[0052] S32 inputs the obtained encoded vector into the Transformer feature extractor model to learn semantic information, thus obtaining a semantic vector. Dimensions .

[0053] Furthermore, including:

[0054] Step S4 includes the following steps:

[0055] S41 calculates the semantic vector of the sample text obtained in S32. The similarity between each tag feature word entry in the feature word vocabulary obtained from S24 is calculated as follows:

[0056]

[0057] This allows us to obtain the semantic information of the sample text and the noise label for each tag. Its dimensions are ;

[0058] S42 will label the noise The original one-hot encoded label is fused with the original one-hot encoded label, and the fused result is used... function The normalization process is calculated using the following formula:

[0059]

[0060] This leads to the final hybrid learning labels. The original one-hot encoded tag is the encoding of each tag. There are 4 tags. The one-hot encoding of the first tag is {1,0,0,0}, the one-hot encoding of the second tag is {0,1,0,0}, the one-hot encoding of the third tag is {0,0,1,0}, and the one-hot encoding of the fourth tag is {0,0,0,1}.

[0061] Furthermore, including:

[0062] Step S5 includes the following steps:

[0063] In the output layer, the semantic vector obtained in step S32 is... A fully connected neural network with one layer is input as a classifier to classify data and obtain predicted labels. As the output of the model classification, its dimension is The calculation formula is as follows:

[0064]

[0065] Will Two dimensions become The method is to Become ; This represents a fully connected layer;

[0066] The test labels obtained using the KL divergence classifier and the mixed learning labels obtained in step S42 As the cross-entropy loss, its calculation formula is as follows:

[0067]

[0068] The parameters are updated via backpropagation using the Adam optimizer, and the value of the loss function on the validation set is calculated after each parameter update.

[0069] Compared with the prior art, the significant advantages of the present invention are:

[0070] (1) When the labels learn the sample information, a label feature word vocabulary is constructed. Two methods are proposed to extract feature words using attention relationship and TD-IDF, and feature word entries are used to enrich the information of each label. In order to eliminate the similarity between the labels, projection purification is used in the constructed label feature word vocabulary to save the independent information of each label.

[0071] (2) An innovation was made based on the final calculation of cross-entropy loss. The absoluteness of the traditional real label using one-hot encoded vectors was improved. A noisy label with the semantic information of the sample text and the similarity between each label was added to the one-hot encoded vector, so that the real label can learn the association information with the sample text. When calculating the loss, the KL divergence between the label after learning the sample text and the label predicted by the model was used to calculate the loss, so that the classification result predicted by the model is closer to the label of the actual sample.

[0072] (3) Since the label hybrid learning model is only to make the label closer to the mixed information of the label and the sample itself, the label hybrid learning model does not participate in the semantic learning model process and has no impact on the performance of the model classification. That is, the feature extractor can be replaced with any model that is needed, with a high degree of freedom and flexibility. Attached Figure Description

[0073] Figure 1 This is a flowchart of the text classification method based on lexical projection hybrid label learning described in this invention;

[0074] Figure 2 This is the overall model structure diagram described in this invention;

[0075] Figure 3 This is a schematic diagram of the feature word vocabulary projection purification method described in this invention.

[0076] Figure 4 This is a schematic diagram illustrating the application of the Transformer model described in this invention. Detailed Implementation

[0077] The present invention will be further described in detail through the following specific embodiments.

[0078] This invention designs a text classification method based on lexical projection hybrid label learning. In practical applications, such as... Figure 1 As shown, specific steps 1 to 6 are performed to obtain the classification model of the sample text, and steps A to C are performed to classify the target sample text.

[0079] Step 1: For all samples, generate relevant word vectors and divide them into training, validation, and test sets according to proportions.

[0080] Step 2: Use the word vectors from Step 1 to encode the sample labels and text, and construct a vocabulary of label feature words.

[0081] Step 3: Input the dataset processed in Step 1 into the semantic learning model and feature extractor to extract the semantic information of the sample text.

[0082] Step 4: In the label hybrid learning model, the label feature word vocabulary constructed in Step 2 and the semantic information of the sample text obtained in Step 3 are used to perform hybrid learning of labels and samples to obtain hybrid learning labels.

[0083] Step 5: Input the semantic information from Step 3 into a fully connected neural network in the output layer as a classifier for classification. Use KL divergence to calculate the class obtained by the classifier and the result of the mixed learning label obtained in Step 4 as the cross-entropy loss. Update the parameters through backpropagation of the Adam optimizer. Calculate the value of the loss function on the validation set after each parameter update.

[0084] Step 6: Train the model and adjust parameters such as the learning rate and the number of hidden layers to optimize F1.

[0085] Step A: Obtain the sample text as in Step 1.

[0086] Step B: Follow step 3 to obtain the semantic information of the sample text from step A.

[0087] Step C: Using the semantic information of the sample text obtained in Step B, classify the target sample text according to Step 5.

[0088] In practical applications, step 1 includes the following steps 1.1 to 1.2:

[0089] Step 1.1: All samples were generated using the Stanford GloVe open-source word vector code, with dimensions of [missing information]. The word vectors are used to add special characters as needed, such as "#PAD#" and "#UNK#", to assist in classification. You can add "#PAD#", "#UNK#", "#CLS#", "#SEP#", "#MASK#", "#NUM#", etc., as needed. In practical applications, because computers are ineffective at classifying and recognizing numbers, a symbol "#NUM#" is defined to replace numbers encountered during classification. The dictionary range 0-19 is reserved for user-defined special symbols; the actual dictionary encoding starts from 20. The dictionary is constructed through code, and the defined symbols are shown in Table 1 below.

[0090] Table 1. Explanation of Special Symbols

[0091] 0 #PAD# Used to fill in length and blank space 1 #UNK# The word used to express the unknown, if it cannot be found in the dictionary. 2 #CLS# The identifier at the beginning of the sentence 3 #SEP# Identifiers that separate the two input sentences 4 #MASK# Used to make a mask for Masked LM 5 #NUM# Used to replace all numbers in a statement, for example, replacing "23.9" directly with #NUM#.

[0092] Step 1.2: The format of each data point is [label, content]. Visualize the length of each sample to obtain the maximum sentence length for model processing. Divide the training set, test set, and validation set in a 6:2:2 ratio, and then proceed to Step 2.1.

[0093] In practical applications, such as Figure 2As shown, step 2 includes the following steps 2.1 to 2.4:

[0094] Step 2.1: Original Sample Text Including tags and text Its input dimension is All samples are encoded using the word vectors from step 1.1, resulting in samples with the following dimensions: ,in, For the number of tags, The number of samples. Sentence length The word vector dimension is then used, and then proceed to step 2.2;

[0095] Step 2.2: Use the sample labels encoded in Step 2.1 Calculate the attention coefficient between each label. Then proceed to step 2.3;

[0096] Step 2.3: Based on all samples, use a combination of TF-IDF and attention mechanisms to obtain the feature word entries for each label, and construct a feature word vocabulary.

[0097] The preferred technical solution, step 2.3 includes the following steps 2.3.1 to 2.3.2:

[0098] Step 2.3.1: The feature words of the labels obtained using the attention mechanism are for each sentence. For samples with existing labels, the following formula is used to obtain feature words with high TF-IDF frequencies:

[0099]

[0100] For feature words with low frequency, an attention mechanism is used to acquire them, and the formula is as follows:

[0101]

[0102] The feature words of each label obtained by combining TF-IDF and the attention mechanism are merged to form the final label feature word segment. This feature word segment is obtained using the above... The calculated words are combined together to form feature terms.

[0103] For example, a word frequency coefficient of 0.7-1 can be defined as a high-frequency word, and 0.4-0.7 as a low-frequency word.

[0104] First, TF-IDF is used to obtain the word frequency of each word, identifying all high-frequency words. Then, attention is calculated for low-frequency words to extract those with higher attention. These two methods are combined to obtain the label feature words.

[0105] Step 2.3.2: After obtaining the feature words using the two methods described in Step 2.3.1, construct the feature word entries. .

[0106] Step 2.3.3: Using the feature word entries obtained in Step 2.3.2, construct feature word entries for each type of label. The feature word vocabulary is constructed as shown in Table 2 below, and then proceed to step 2.4;

[0107] Table 2. Vocabulary of Featured Words

[0108] World {Iraqi, hostage, Allawi, iyad, minister, prime}, {officials,…}, …, {egypts, …} Sports {yankees, championship, sox, league, game, series}, {beating, …}, …, {foutfielder, …} Business {mae, fannie, mortgage, finance, company, accounting}, {stocks, …}, …, {profits, …} Science {concern, software, threats, cyber, viruses}, {system, …}, …, {inetwork, …}

[0109] Step 2.4: As Figure 3 As shown, the attention coefficients between labels obtained in step 2.2 are used to project and refine the feature word vocabulary in step 2.3. For label feature word entries... and The projection purification method is used to eliminate commonalities, resulting in a purified feature word vocabulary. The formula is shown below. Then, proceed to step 3.1:

[0110]

[0111]

[0112]

[0113] It comes from The vectors that follow, here In order to Depend on .

[0114] In practical applications, step 3 includes the following steps 3.1 to 3.2:

[0115] Step 3.1: Input the dataset processed in Step 1.2 into the semantic learning model to learn the semantic information of the sample text. Taking Transformer as an example, first, the result of word encoding in Step 1.2 is used... and The function is position-coded according to the following formula:

[0116]

[0117]

[0118] In the above formula, This refers to the position of words in a sentence. and It refers to the position of the character in the odd / even position. It is the dimension of the word vector, resulting in the encoded vector. Dimensions Then proceed to step 3.2. The above indicates... That formula It is the encoding calculation of the word at the corresponding even-numbered position. That formula This involves calculating the encoding of characters at odd-numbered positions. The PE (Position Equation) is calculated using the positional encoding of characters at each odd and even position; combining these encodings yields the encoded vector for the entire sentence.

[0119] Step 3.2: Input the encoded vector obtained in Step 3.1 into the Transformer feature extractor model to learn semantic information and obtain the semantic vector. Dimensions Then proceed to step 4.1. For example... Figure 4 The diagram shown is a structural diagram of the Transformer model.

[0120] In practical applications, step 4 includes the following steps 4.1 to 4.2:

[0121] Step 4.1: In the label fusion learning model, calculate the semantic vector of the sample text obtained in Step 3.2. The similarity between the feature words and each tag feature word in the feature word vocabulary obtained in step 2.4 is calculated as follows:

[0122]

[0123] This allows us to obtain the semantic information of the sample text and the noise label for each tag. Its dimensions are Then proceed to step 4.2;

[0124] Step 4.2: Transfer the noise labels obtained in Step 4.1 The original one-hot encoded label is fused with the original one-hot encoded label, and the fused result is used... The normalization formula for the function is shown below:

[0125]

[0126] This leads to the final hybrid learning labels. .

[0127] Step 5: In the output layer, combine the semantic vectors from step 3.2. A fully connected neural network with one layer is input as a classifier to classify and obtain predicted labels. As the output of the model classification, its dimension is The calculation formula is as follows:

[0128]

[0129] The test labels obtained using the KL divergence classifier and the hybrid learning labels obtained in step 4.2 As the cross-entropy loss, its calculation formula is as follows:

[0130]

[0131] The parameters are updated via backpropagation using the Adam optimizer, and the value of the loss function on the validation set is calculated after each parameter update.

Claims

1. A text classification method based on lexical projection hybrid label learning, characterized in that, It includes a training phase and a testing phase, wherein the training phase includes the following steps: S1 generates relevant word vectors based on all samples and divides them into training, validation, and test sets according to proportions; S2 uses the relevant word vectors to encode the labels and text of the samples, and constructs a refined vocabulary of label feature words; including: The original sample S21 is represented by the following text: , including tags and text The sentence length is All samples are encoded using word vectors, resulting in samples with the following dimensions: ,in, For the number of tags, The number of samples. The dimension is the word vector. S22 calculates the attention coefficients between each label. ; S23 uses a combination of TF-IDF and attention mechanisms to obtain feature word entries for each label based on all samples, and constructs a feature word vocabulary; S24 uses attention coefficients to project and refine the feature word vocabulary for two-label feature word entries. and The purified feature word vocabulary is obtained by using the projection purification method to eliminate commonalities. The formula is shown below: ； ； in, , This is the formula for vector projection. , Two vectors, through orthogonal projection ； Represented as Vector in The projection vector onto a vector; Used to count, for The number of vector , It comes from The subsequent vector; S3 inputs the training set into the semantic learning model and uses a feature extractor to extract the semantic information of the samples; S4 uses the purified label feature word vocabulary and the semantic information of the sample text to perform hybrid learning of labels and samples, generating hybrid learned labels. S5 inputs the semantic information of the sample into a one-layer fully connected neural network as a classifier for classification; uses KL divergence to calculate the class obtained by the classifier and the result of the mixed learning label as the cross-entropy loss; updates the parameters through backpropagation of the Adam optimizer; calculates the value of the loss function on the validation set after each parameter update; and continuously adjusts the parameters to make F1 optimal. The testing phase includes the following steps: S6 obtains the semantic information of the corresponding samples in the test set; S7 inputs the test set into the semantic learning model and uses a feature extractor to extract the semantic information of the samples; S8 inputs the semantic information of the sample into the trained classifier to obtain the text classification result.

2. The text classification method based on lexical projection hybrid label learning according to claim 1, characterized in that, In step S1, generating relevant word vectors based on all samples includes: All S11 samples used Stanford GloVe open-source word vector code to create word vectors, and added special characters to assist in classification; S12 uses the format [label, content] for each sample and visualizes the length of each sample to obtain the maximum sentence length processed by the label hybrid learning model and the semantic learning model.

3. The text classification method based on lexical projection hybrid label learning according to claim 1, characterized in that, Step S23 includes the following steps: S231 For each sample statement The following formula is used to obtain the feature words with high TF-IDF frequency for the samples in the training set: ； For feature words with low frequency, an attention mechanism is used to acquire them, and the formula is as follows: For low-frequency words, the vector encoded from the sentence characters... The formula for calculating self-attention is as follows: ； Obtain the attention coefficient of the word in the sentence, and take the word with the largest attention coefficient as the feature word; ； The feature words of each label obtained by combining TF-IDF and attention mechanism are merged to form the final label feature word entries, thus constructing the feature word entries. This feature term is based on the above. The calculated words are combined to obtain feature entries; S232 constructs feature word entries for each type of label. This leads to the construction of a vocabulary of feature words; .

4. The text classification method based on lexical projection hybrid label learning according to claim 3, characterized in that, Step S3 includes the following steps: S31 inputs the dataset processed in step S12 into the semantic learning model to learn the semantic information of the samples. Specifically, position encoding is performed using the following formula: ； ； in, This refers to the position of a word in a sentence. It refers to the word position at an even-numbered position. It refers to the position of the character in an odd-numbered position. It is the dimension of the word vector, and The combined encoding vector , dimension ; S32 inputs the obtained encoded vector into the Transformer feature extractor model to learn semantic information, thus obtaining a semantic vector. Dimensions .

5. The text classification method based on lexical projection hybrid label learning according to claim 4, characterized in that, Step S4 includes the following steps: S41 calculates the semantic vector of the sample text obtained in S32. The similarity between each tag feature word entry in the feature word vocabulary obtained from S24 is calculated as follows: ； This allows us to obtain the semantic information of the sample text and the noise label for each tag. Its dimensions are ; S42 will label the noise The original one-hot encoded label is fused with the original one-hot encoded label, and the fused result is used... The normalization formula for the function is shown below: ； This leads to the final hybrid learning labels. The original one-hot encoded tag is the encoding of each tag. There are four tags. The one-hot encoding of the first tag is {1,0,0,0}, the one-hot encoding of the second tag is {0,1,0,0}, the one-hot encoding of the third tag is {0,0,1,0}, and the one-hot encoding of the fourth tag is {0,0,0,1}. This is the sigmoid function.

6. The text classification method based on lexical projection hybrid label learning according to claim 5, characterized in that, Step S5 includes the following steps: In the output layer, the semantic vector obtained in step S32 is... A fully connected neural network with one layer is input as a classifier to classify data and obtain predicted labels. As the output of the model classification, its dimension is The calculation formula is as follows: ； Will Two dimensions become The method is to Become ; This represents a fully connected layer; Predicted labels obtained using the KL divergence classifier and the mixed learning labels obtained in step S42 As the cross-entropy loss, its calculation formula is as follows: ； The parameters are updated via backpropagation using the Adam optimizer, and the value of the loss function on the validation set is calculated after each parameter update.

Images