A data enhancement method for multi-label text classification based on active feature selection

Abstract

The present invention discloses a multi-label text classification data enhancement method based on active feature selection, relating to the field of natural language processing technology. This method uses a label prompt template to obtain its score for each word in the text. From each batch of sentences with the same label, the top K words with the highest scores are extracted. The features of these words are then randomly swapped. The features of the high-scoring words and the randomly swapped words are mixed according to λ and 1-λ, and then rewritten into the sentence, achieving the effect of data enhancement. Compared with traditional baseline models, the experimental group's evaluation indicators of this method are significantly improved compared to all baseline models.

Description

Technical Field

[0001] The invention discloses a multi-label text classification data enhancement method based on active feature selection, and relates to the technical field of natural language processing. Background Art

[0002] Text classification is an important and classic problem in natural language processing, which involves categorizing text according to specific rules. Multi-label text classification involves classifying text, where a single text item may be assigned to multiple categories, necessitating the consideration of multiple labels. With the continuous development of the internet, platforms like Weibo and Twitter have accumulated a vast amount of labeled text data, requiring more granular classification of categories. Therefore, multi-label classification is more suitable for people's needs. For example, a text about a disaster on social media often covers multiple topics, including the type of disaster, the victim's condition, rescue operations, and donation information. This information allows people to take timely action.

[0003] Nowadays, the data augmentation strategy of Mixup feature mixing has been widely used in single-label and multi-label text classification problems. For single-label text, each text feature is associated with a label. However, for multi-label text classification, a text has multiple labels. Therefore, the relationship between text features and labels is unclear during the feature mixing process. Therefore, it is proposed to mix features at the sequence level. During the mixing process, the importance of each word is judged by the score of each word for the label, and a new mixed feature vector is generated. This smoothes the boundaries between different samples in the feature space, thereby improving the model's understanding of the correlation between labels and its classification ability. Summary of the Invention

[0004] The present invention provides a multi-label text classification data enhancement method based on active feature selection, which can enhance data and thus improve the classification effect.

[0005] The technical solution of the present invention is: a multi-label text classification data enhancement method based on active feature selection, comprising the following steps:

[0006] S1. Use the embedding layer of the pre-trained language model to embed the segmented text to obtain a word embedding vector. At the same time, use the pre-trained language model to encode the label category length to obtain the initialized label prompt template PL, and embed PL to obtain a label vector. Then, concatenate the label vector and the word embedding vector to obtain a label prompt template vector with text information.

[0007] S2. Use the attention structure in the pre-trained language model to encode the concatenated label prompt template vector with text information to generate a new vector sequence, and then reduce the dimension of the vector through a single linear layer;

[0008] S3. Use the inner product similarity between the reduced-dimensional label vector and the word embedding vector to calculate the score of each word for each template;

[0009] S4. Take the K words with higher scores from each template in each sentence of each batch, re-embed these words, and then randomly shuffle the K words with higher scores and embed them again; then compare the features obtained after re-embedding and the features obtained after shuffling the re-embedding according to and 1- mix, Obeying the Beta distribution, the original word embedding layer is rewritten after mixing to obtain the data-enhanced word embedding vector, thereby achieving the effect of data enhancement;

[0010] S5. Concatenate the data-enhanced word embedding vector with the label vector in S1 again to obtain a new label prompt template vector with text information; repeat S2-S3 to obtain the score of each word for each template after data enhancement. The word with the highest score for each template after data enhancement in the current sentence represents the category of this sentence. Therefore, the maximum value of the scores of each template among several templates is used for classification.

[0011] As a further solution of the present invention, the specific implementation of S1 includes:

[0012] Define the batch number as B, N as the label type, M as the vector dimension, and pass the text sequence of length L through the embedding layer of the pre-trained language model to obtain the word embedding vector E W = [ w 1 , … , w L ] ; Use the pre-trained language model to encode the label length to obtain the initialized label prompt template PL, and use the embedding layer to embed the label prompt template PL to obtain the following label vector E p = [ p 0 ,..., p N ] , then concatenate the label vector and the word embedding vector to obtain the label prompt template vector with text information .

[0013] As a further solution of the present invention, the specific implementation of S2 includes:

[0014] Label prompt template vector with text information After encoding with the pre-trained language model, a vector sequence is obtained [ e p , e 1 ,... e L ] ,in is the label vector encoded by the pre-trained language model, [e 1 ,..., e L ]∈ R B*L*M It is a word embedding vector encoded by a pre-trained language model. The bidirectional Transformer encoder and attention interaction in the pre-trained language model enable the label prompt template to carry the semantic information of the text;

[0015] The vector sequence is reduced in dimension using a single linear layer, and the dimension reduction is U = [ u p , u 1 ,..., u L ] ∈ R B*(L+N)* m , m is the dimension after dimensionality reduction, B is the number of batches, N is the type of label, M is the vector dimension, [u 1 ,..., u L ] is the word embedding vector after dimensionality reduction, is the label vector after dimensionality reduction.

[0016] As a further solution of the present invention, the specific implementation of S4 includes:

[0017] From the score of each word for each template ] Select K groups of token values ​​and their indexes with higher scores according to Take out a total of B*K words and re-embed them to get the embedding representation E topk = [ W K ] ∈ R B*K*M , then randomly shuffle the order of the words and re-embed them to get the shuffled embedding representation ; Then according to the Beta distribution, we get value, will and according to and 1- Mixed, mixed results dimension ; Then according to the index Will Rewrite the original word embedding layer In the data enhancement process, we get the word embedding vector after data enhancement. , B is the number of batches, L is the length of the text sequence, and M is the vector dimension.

[0018] As a further solution of the present invention, the specific implementation of S5 includes:

[0019] Data-enhanced word embedding vectors and the label vector in S1 Splice again and repeat S2-S3 to get the score of each word for each template after data enhancement. The word with the highest score for each template in the current sentence represents the category of this sentence. Therefore, the maximum score of each word for each template after data enhancement in several templates is used for classification, and the maximum value is taken for classification: , and then complete multi-label text classification.

[0020] in, The score of each word in each template after data augmentation in several templates is obtained after encoding by the S2 pre-trained language model. The vector sequence is as follows: in is the label vector after data enhancement, [e mix ,..., e mix_L ] ∈ R B*L*M is the word vector of data enhancement, B is the number of batches, N is the type of label, L is the length of the text sequence, and M is the vector dimension.

[0021] Beneficial effects of the present invention:

[0022] 1. The present invention obtains the score of each word in the text through the label prompt template, takes out the top K words in each batch of sentences with the same label, and then randomly swaps the features of these words. The features of the words with high scores and the features of the randomly swapped words are combined according to and 1- After mixing, rewrite the sentence to achieve data enhancement;

[0023] 2. The present invention was tested on five datasets: Russia Meteor, Cyclone Pam, Philippines Flood, Mixed Disasters, and the publicly available AAPD. Compared with traditional baseline models, the evaluation indicators of the experimental group were significantly improved compared with all baseline models. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 This is an overall framework diagram of a multi-label text classification data enhancement method based on active feature selection provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The embodiments of the present invention are described below in conjunction with the accompanying drawings. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0026] like Figure 1 FIG. 1 is a diagram showing the overall framework of a multi-label text classification data enhancement method based on active feature selection provided by an embodiment of the present invention. The multi-label text classification data enhancement method based on active feature selection includes the following steps:

[0027] S1. Use the embedding layer of the pre-trained language model to embed the segmented text to obtain a word embedding vector. At the same time, use the pre-trained language model to encode the label category length to obtain an initialized label prompt template PL, and embed PL to obtain a label vector. Then, the label vector and the word embedding vector are concatenated to obtain a label prompt template vector with text information. In some feasible implementations, the specific implementation of S1 includes:

[0028] Define the batch number as B, N as the label type, M as the vector dimension, and pass the text sequence of length L through the embedding layer of the pre-trained language model to obtain the word embedding vector E W = [ w 1 , … , w L ] ; Use the pre-trained language model to encode the label length to obtain the initialized label prompt template PL, and use the embedding layer to embed the label prompt template PL to obtain the following label vector E p = [ p 0 ,..., p N ] , then concatenate the label vector and the word embedding vector to obtain the label prompt template vector with text information .

[0029] S2. Use the attention structure in the pre-trained language model to encode the concatenated label prompt template vector with text information to generate a new vector sequence, and then reduce the dimension of the vector through a single linear layer;

[0030] In some feasible implementations, the specific implementation of S2 includes: After encoding with the pre-trained language model, a vector sequence is obtained [ e p , e 1 ,... e L ] ,in is the label vector encoded by the pre-trained language model, [e 1 ,..., e L ]∈ R B*L*M It is a word embedding vector encoded by a pre-trained language model. The bidirectional Transformer encoder and attention interaction in the pre-trained language model enable the label prompt template to carry the semantic information of the text;

[0031] The vector sequence is reduced in dimension using a single linear layer, and the dimension reduction is U = [ u p , u 1 ,..., u L ] ∈ R B*(L+N)* m , m is the dimension after dimensionality reduction, B is the number of batches, N is the type of label, M is the vector dimension, [u 1 ,..., u L ] is the word embedding vector after dimensionality reduction, is the label vector after dimensionality reduction.

[0032] S3. Use the inner product similarity between the reduced-dimensional label vector and the word embedding vector to calculate the score of each word for each template;

[0033] S4. Take the K words with higher scores from each template in each sentence of each batch, re-embed these words, and then randomly shuffle the K words with higher scores and embed them again; then compare the features obtained after re-embedding and the features obtained after shuffling the re-embedding according to and 1- mix, Following the Beta distribution, the original word embedding layer is rewritten after mixing to obtain a data-enhanced word embedding vector, thereby achieving the effect of data enhancement; in some feasible implementations, the specific implementation of S4 includes:

[0034] From the score of each word for each template ] Select K groups of token values ​​and their indexes with higher scores according to Take out a total of B*K words and re-embed them to get the embedding representation E topk = [ W K ] ∈ R B*K*M , then randomly shuffle the order of the words and re-embed them to get the shuffled embedding representation ; Then according to the Beta distribution, we get value, will and according to and 1- Mixed, mixed results dimension ; Then according to the index Will Rewrite the original word embedding layer In the data enhancement process, we get the word embedding vector after data enhancement. , B is the number of batches, L is the length of the text sequence, and M is the vector dimension.

[0035] S5. The data-enhanced word embedding vector is concatenated with the label vector in S1 again to obtain a new label hint template vector with text information; S2-S3 are repeated to obtain the score of each word for each template after data enhancement. The word with the highest score for each template after data enhancement in the current sentence represents the category of this sentence. Therefore, the maximum score of each template among several templates is used for classification. In some feasible implementations, the specific implementation of S5 includes:

[0036] Data-enhanced word embedding vectors and the label vector in S1 Splice again and repeat S2-S3 to get the score of each word for each template after data enhancement. The word with the highest score for each template in the current sentence represents the category of this sentence. Therefore, the maximum score of each word for each template after data enhancement in several templates is used for classification, and the maximum value is taken for classification: ;

[0037] in, The score of each word in each template after data augmentation in several templates is obtained after encoding by the S2 pre-trained language model. The vector sequence is as follows: in is the label vector after data enhancement, [e mix ,..., e mix_L ] ∈ R B*L*M is the word vector of data enhancement, B is the number of batches, N is the type of label, L is the length of the text sequence, and M is the vector dimension.

[0038] To illustrate the effectiveness of the present invention, we compared it with existing methods using several aggregated resources from CrisisNLP and CrisisLex, including Resource #1, #4, #5, and #7, as well as CrisisLexT6 and T26. These resources contain four labels: Casualties & Damage, Caution & Advice, Donation & Volunteering, and Informative, along with 108,973 English data items. 98,973 data items were used for training, 10,000 for validation, and four datasets were used for testing: Meteor, Cyclone, Flood, and Mixed, containing 802, 2,473, 684, and 10,000 English data items, respectively. The publicly available AAPD dataset was also used, containing 55,840 samples and 54 labels. 53,840 samples were used for training, 1,000 for validation, and 1,000 for testing. The evaluation index is consistent with the comparison method, and F1 is used as the index to evaluate the classification results. The experimental results of the dataset used are shown in Table 1:

[0039] Table 1 Experimental results

[0040]

[0041] As shown in Table 1, the evaluation indicators of the experimental group are significantly improved compared to all baseline models. It is believed that by actively selecting features and mixing features in proportion, data enhancement is achieved, thereby improving the classification performance of the model.

[0042] Throughout this specification, references to terms such as "one embodiment," "example," or "specific example" indicate that the specific features, structures, materials, or characteristics described in conjunction with that embodiment or example are included in at least one embodiment or example of the present invention. In this specification, schematic representations of these terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0043] The preferred embodiments of the present invention disclosed above are intended only to help illustrate the present invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the present invention to the specific embodiments described. Obviously, many modifications and variations are possible based on the content of this specification. These embodiments are selected and described in detail in this specification to better explain the principles and practical applications of the present invention, thereby enabling those skilled in the art to better understand and utilize the present invention. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A multi-label text classification data enhancement method based on active feature selection, characterized in that: The steps include: S1. Use the embedding layer of the pre-trained language model to embed the segmented text to obtain a word embedding vector. At the same time, use the pre-trained language model to encode the label category to obtain the initialized label prompt template PL, and embed PL to obtain a label vector. Then, concatenate the label vector and the word embedding vector to obtain a label prompt template vector with text information. S2. Use the attention structure in the pre-trained language model to encode the concatenated label prompt template vector with text information to generate a new vector sequence, and then reduce the dimension of the vector through a single linear layer; S3. Use the inner product similarity between the reduced-dimensional label vector and the word embedding vector to calculate the score of each word for each template; S4. Take the top K words in each sentence of each batch, re-embed them, and then randomly shuffle the top K words and embed them again. Then, compare the features obtained after re-embedding and the features obtained after shuffling the re-embedding according to and 1- mix, Obeying the Beta distribution, the original word embedding layer is rewritten after mixing to obtain the data-enhanced word embedding vector, thereby achieving the effect of data enhancement; S5. Concatenate the data-enhanced word embedding vector with the label vector in S1 again to obtain a new label prompt template vector with text information; repeat S2-S3 to obtain the score of each word for each template after data enhancement. The word with the highest score for each template after data enhancement in the current sentence represents the category of this sentence. Therefore, the maximum value of the scores of each template among several templates is used for classification.

2. The multi-label text classification data enhancement method based on active feature selection according to claim 1 is characterized in that The specific implementation of S1 includes: Define the batch number as B, N as the label type, M as the vector dimension, and pass the text sequence of length L through the embedding layer of the pre-trained language model to obtain the word embedding vector ; Use the pre-trained language model to encode the label category to obtain the initialized label prompt template PL, and use the embedding layer to embed the label prompt template PL to obtain the following label vector , then concatenate the label vector and the word embedding vector to obtain the label prompt template vector with text information .

3. The multi-label text classification data enhancement method based on active feature selection according to claim 1 is characterized in that The specific implementation of S2 includes: Label prompt template vector with text information After encoding with the pre-trained language model, a vector sequence is obtained ,in is the label vector encoded by the pre-trained language model, It is a word embedding vector encoded by a pre-trained language model. The bidirectional Transformer encoder and attention interaction in the pre-trained language model enable the label prompt template to carry the semantic information of the text; The vector sequence is reduced in dimension using a single linear layer, and the dimension reduction is , m is the dimension after dimensionality reduction, B is the number of batches, N is the type of label, M is the vector dimension, is the word embedding vector after dimensionality reduction, is the label vector after dimensionality reduction, and L is the length of the text sequence.

4. The multi-label text classification data enhancement method based on active feature selection according to claim 1 is characterized in that The specific implementation of S4 includes: From the score of each word for each template ] Select K groups of token values ​​and their indexes with the highest scores according to Take out a total of B*K words and re-embed them to get the embedding representation , then randomly shuffle the order of the words and re-embed them to get the shuffled embedding representation ; Then according to the Beta distribution, we get value, will and according to and 1- Mix, the mixed result is ; Then according to the index Will Rewrite the original word embedding layer In the data enhancement process, we get the word embedding vector after data enhancement. , B is the number of batches, L is the length of the text sequence, and M is the vector dimension.

5. The multi-label text classification data enhancement method based on active feature selection according to claim 1 is characterized in that The specific implementation of S5 includes: Data-enhanced word embedding vectors and the label vector in S1 Splice again and repeat S2-S3 to get the score of each word for each template after data enhancement. The word with the highest score for each template in the current sentence represents the category of this sentence. Therefore, the maximum score of each word for each template after data enhancement in several templates is used for classification, and the maximum value is taken for classification: ; in, The score of each word in each template after data augmentation in several templates is obtained after encoding by the S2 pre-trained language model. The vector sequence is as follows: in is the label vector after data enhancement, is the word vector of data enhancement, B is the number of batches, N is the type of label, L is the length of the text sequence, and M is the vector dimension.

Images