A multi-label text classification method based on long tail problem
By constructing a text semantic extraction module and a weight transfer module, and utilizing BERT, BILSTM, and attention mechanisms, the long-tail problem in multi-label text classification is solved, the generalization ability of tail labels and model performance are improved, and more accurate classification results are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-11-30
- Publication Date
- 2026-03-17
AI Technical Summary
Existing multi-label text classification methods have weak generalization ability for tail labels when dealing with long-tail problems, and the correlation between labels affects the classification effect, causing the classification to be biased towards head labels, resulting in incorrect classification results.
A text classification model is constructed, which includes a text semantic extraction module, a weight transfer module, a fusion module, and a classification module. The text representations of the head and tail labels are extracted by BERT, BILSTM, and attention mechanisms, respectively. The head label classifier is trained to obtain the head label weights, and the weights are transferred to the tail labels through transfer learning. The final classifier is obtained by fusion.
It significantly improves the accuracy and cumulative gain of multi-label text classification, alleviates the long-tail problem, enhances the generalization ability of tail labels, takes into account the correlation between labels, and improves the overall performance of the model.
Smart Images
Figure CN115757795B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of natural language processing and multi-label text classification, and specifically relates to a multi-label text classification method based on the long-tail problem. Background Technology
[0002] The long-tail problem, a hot topic in text classification, has attracted considerable attention. How to solve the long-tail problem is a key research point in the field of multi-label text classification. For general single-label text classification, the impact is relatively low. However, for multi-label text, some labels are closely related while others are not, making classification more complex. Head labels are those that appear frequently in a few documents, while tail labels are those that appear less frequently in most documents. This leads to classification being biased towards head labels, causing the long-tail problem, and in severe cases, even resulting in classification errors.
[0003] Multi-label text classification is a branch of natural language processing. The main approaches to multi-label text classification fall into two categories: traditional machine learning methods and deep learning-based methods. Traditional machine learning methods have many drawbacks, particularly in handling label relevance and long-tail problems. Deep learning-based methods are currently the mainstream approach, utilizing various neural network models to address multi-label text classification. This paper also uses BERT and BILSTM, currently mainstream models. Although there has been considerable research on multi-label text classification, several issues still remain:
[0004] 1. The long-tail problem of labels: Due to the long-tail problem, the generalization ability of the tail labels is relatively weak. When training the classifier, more classification tends to be based on the head labels, causing labels that should be tail labels to be head labels. Therefore, solving the long-tail problem is urgent.
[0005] 2. There are interrelationships between labels, so there is an influence between labels. This is also a problem we need to address, and solving this problem is one of the keys to improving model performance. Summary of the Invention
[0006] To address the aforementioned problems, this invention provides a multi-label text classification method based on the long-tail problem. It constructs a text classification model comprising a text semantic extraction module, a weight transfer module, a fusion module, and a classification module. The classification process includes the following steps:
[0007] S1. The text semantic extraction module extracts the text semantics of the header and tail labels using BERT, BILSTM, and attention mechanisms, and obtains the text representations of the header and tail labels respectively.
[0008] S2. Sample the header label text representation and the footer label text representation respectively to obtain the header label prototype and the footer label prototype;
[0009] S3. Train the header label classifier using the header label text representation to obtain the header label classifier. Obtain the header label weights by minimizing the loss function of the header label classifier.
[0010] S4. The weight transfer module maps the header label prototype to the header label weight and obtains the transfer weight by minimizing transfer learning.
[0011] S4. The weight transfer module transfers the weights of the head labels according to the transfer weights to obtain the tail label classifier.
[0012] S5. The fusion module merges the tail label classifier and the head label classifier to obtain the classification module;
[0013] S6. Input the text representation of the data to be classified into the classification module to obtain the classification result.
[0014] Furthermore, the process of obtaining the text representation includes:
[0015] The text is pre-trained using the BERT model to obtain a pre-trained text representation;
[0016] Obtain the text context dependencies of the pre-trained text representation using BILSTM;
[0017] By using an attention mechanism to enhance the text context dependencies, a text representation can be obtained.
[0018] Furthermore, the text is pre-trained using BERT, and the output is {V1, V2, ..., V...} p , ..., V n}, where n represents the maximum word length, and V p Let be the text embedding vector corresponding to the p-th word; the BERT dimension used is 768, the number of multi-head attention mechanisms is 12, and the number of Transformer layers is 12.
[0019] Furthermore, obtaining the text context dependencies of the pre-trained text representation using BILSTM includes:
[0020]
[0021]
[0022]
[0023]
[0024]
[0025] Among them, V p Let H be the text embedding vector corresponding to the p-th word; H is the text context dependency, H∈R 2k×n , Forward text context dependency, For backward text context dependencies, This represents the forward hidden state at point p. This represents the backward hidden state at point p.
[0026] Furthermore, attention mechanisms are used to enhance the text context dependencies, resulting in text representations including:
[0027] e = softmax(tanh(W1H))
[0028] o=eH T W2
[0029] Where e represents the contribution of all words to the text, o is the text representation, and W1∈R 1×2k W2∈R 2k×d It is a trainable matrix.
[0030] Furthermore, the head label classifier includes a sigmoid activation function layer. The text representation is input to the head label classifier, and the cross-entropy loss function is used as the loss function of the head label classifier. The weight W of the head label is obtained by minimizing the loss function. head , is represented as:
[0031]
[0032]
[0033] in, The classification result predicted by the head label classifier, o represents the text representation, and W represents the text. head The weights of the head classifier are represented by l. head Indicates the number of tags corresponding to the header tag, x i D represents the i-th text. head This represents the collection of text corresponding to the header tags. Let y represent the predicted probability of the i-th text corresponding to the j-th header tag. ij Let represent the actual probability that the i-th text corresponds to the j-th header label, and loss represent the loss function.
[0034] Furthermore, the header and footer label text representations are sampled separately to obtain header and footer label prototypes, including:
[0035] In x samples, for the i-th header tag, the document is represented as Then the prototype of the i-th header tag is:
[0036] In x samples, for the j-th tail tag, the document is represented as Then the prototype of the j-th tail tag is:
[0037] in, This represents the vector representation of the x-th sample for the i-th header label; represents the vector representation of the x-th sample with respect to the j-th tail label; avg represents the mean of the vector.
[0038] Furthermore, the weight transfer module maps the header label prototypes to header label weights, and the process of obtaining the transfer weights by minimizing transfer learning includes:
[0039]
[0040] Where τ represents the transfer learner function, l head This indicates the number of tags corresponding to the header tag. Let Wt represent the prototype of the i-th header label, and Wt represent the transition weight matrix. Whead represents the weight of the i-th header label, and Whead is the set of weights for the left and right header labels.
[0041] Furthermore, the process of building a tail label classifier includes the following steps:
[0042] The linear combination of the relevance of the i-th header label to the j-th tail label is expressed as:
[0043]
[0044] An attention mechanism is used to process the linear combination of the relevance between the i-th head label and the j-th tail label, and the prototype of the i-th head label is then processed. After weighting, we obtain the new prototype of the j-th tail label, represented as:
[0045]
[0046] a ji =softmax(e ji )
[0047]
[0048]
[0049] The new prototype of the j-th tail label is used to obtain its corresponding tail label classifier through the transition matrix, denoted as:
[0050]
[0051] in, This represents a linear prototypical combination of header and footer tags. This represents the prototype of the i-th header tag. Represents the prototype of the j-th tail tag; e ji W3 represents the correlation between the head and tail labels after additive attention processing, and W3 represents the attention weight. ji This represents the attention score. Let p represent the attention prototype of the j-th header label, avg represent the mean of the attention quantity, and p j W represents the new prototype of the j-th tail tag; t Represents the transition weight matrix. This represents the tail label classifier weight for the j-th label.
[0052] Furthermore, the text representation of the data to be classified is input into the classification module, and the classification results include:
[0053]
[0054] W = [W head :W tail ]
[0055] in, The classification result is represented by r, which represents the text representation of the data to be classified; W head Indicates the weight of the header label; W tail This indicates the weight of the tail label.
[0056] This invention utilizes BERT, BILSTM, and attention mechanisms for text semantic extraction. By processing head and tail labels separately, a head label classifier is trained, and head and tail label prototypes are obtained. A weight transferor is then constructed to transfer the generalization ability of the head labels to the tail labels. During the transfer process, not only is the long-tail problem alleviated, but the correlation between head and tail labels is also considered, resulting in a more comprehensive tail label classifier. Finally, a complete classifier is synthesized for classification. This classification method can improve the performance of the classification model, significantly increasing classification accuracy and normalized loss cumulative gain. Attached Figure Description
[0057] Figure 1This is a flowchart illustrating the overall process of a multi-label text classification method based on the long-tail problem of the present invention.
[0058] Figure 2 This is a diagram of the semantic extraction module in a multi-label text classification method based on the long-tail problem of the present invention;
[0059] Figure 3 This is a flowchart of the multi-label text classification method based on the long-tail problem of the present invention. Detailed Implementation
[0060] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0061] This invention provides a multi-label text classification method based on the long-tail problem, constructing a text classification model including a text semantic extraction module, a weight transfer module, a fusion module, and a classification module, such as... Figure 1 The classification process includes the following steps:
[0062] S1. The text semantic extraction module extracts the text semantics of the header and tail labels using BERT, BILSTM, and attention mechanisms, and obtains the text representations of the header and tail labels respectively.
[0063] S2. Sample the header label text representation and the footer label text representation respectively to obtain the header label prototype and the footer label prototype;
[0064] S3. Train the header label classifier using the header label text representation to obtain the header label classifier. Obtain the header label weights by minimizing the loss function of the header label classifier.
[0065] S4. The weight transfer module maps the header label prototype to the header label weight and obtains the transfer weight by minimizing transfer learning.
[0066] S4. The weight transfer module transfers the weights of the head labels according to the transfer weights to obtain the tail label classifier.
[0067] S5. The fusion module merges the tail label classifier and the head label classifier to obtain the classification module;
[0068] S6. Input the text representation of the data to be classified into the classification module to obtain the classification result.
[0069] The overall block diagram of a multi-label text classification method based on the long-tail problem in this embodiment is as follows: Figure 3 As shown, it mainly includes a text semantic extraction module, a head label classifier acquisition module and head label and tail label prototypes, a weight transfer module, a head label classifier and tail label classifier fusion module, and a classification module.
[0070] This embodiment implements a multi-label text classification method based on the long-tail problem. It uses the BERT model for text preprocessing to obtain preliminary text representation vectors. First, the text is pre-trained using BERT, and the output result is {V1, V2, ..., V...} p ,...,V n In the BERT model, n represents the maximum word length (n=200), k=768 dimensions, there are 12 multi-head attention mechanisms, and 12 Transformer layers. During implementation, text and labels are fed into the BERT model separately. The semantic extraction process involves passing the text and label data through word embedding layers, sentence embedding layers, and position embedding layers to obtain a vector representation containing words, sentences, and positions.
[0071] In this embodiment, the text data and label data are obtained from an existing text dataset. The text dataset in this embodiment mainly includes one of three datasets, and a dataset composed of these datasets is obtained. The three datasets are AAPD, RCV1, and EUR-Lex datasets. The training text used in this embodiment includes text data with multiple labels; that is, one text data includes multiple labels. Labels with higher weights are designated as head labels, and labels with lower weights are designated as tail labels. Those skilled in the art can set these settings according to actual conditions, such as setting head label thresholds and tail label thresholds. When the weight of a label is greater than the head label threshold, the label is a head label; when the weight of a label is less than the tail label threshold, the label is a tail label. Those skilled in the art can select the training dataset according to actual conditions.
[0072] The obtained text label embedding vectors are then used to extract context dependencies using BILSTM. The specific steps are as follows:
[0073] The structure diagram of LSTM is as follows Figure 2 The calculation formula is as follows:
[0074]
[0075]
[0076]
[0077]
[0078]
[0079] Among them, V p Let H be the text embedding vector corresponding to the p-th word, and H be the text representation. Forward text representation, For backward text representation, This represents the forward hidden state at point p. H represents the backward hidden state at point p, where H belongs to R. 2k×n .
[0080] The context dependencies extracted by BILSTM are enhanced through an attention mechanism to obtain a sufficient text vector representation:
[0081] e = softmax(tanh(W1H))
[0082] o=eH T W2
[0083] Where e represents the contribution of each word to the text, o represents the final text vector representation, and W1∈R 1×2k W2∈R 2k×d d represents the dimension of the document vector, typically d = 128.
[0084] In this embodiment, the process of obtaining the head classifier, the head label prototype, and the tail label prototype specifically includes:
[0085] First, the text representation is passed through a Sigmoid activation function to obtain a head label classifier. Then, the head label weights are obtained by minimizing the loss function.
[0086] The head classifier is constructed using a single-layer sigmoid activation function, and the cross-entropy loss function is used as the loss function. The weight W of the head label is obtained by minimizing the loss function. head The formula is as follows:
[0087]
[0088]
[0089] in, For the classification probability, W head The weights of the head classifier are represented by l. head Indicates the number of tags corresponding to the header tag, x i D represents the i-th text. head This indicates the document corresponding to the header tag. y represents the predicted probability. ijdenoted by , where represents the actual probability, and loss represents the loss function.
[0090] Then, using the concept of prototype networks, the text is sampled to obtain the prototypes of the header and footer tags, i.e.:
[0091] If, in x samples, for the i-th header tag, the document is represented as... Then the prototype of the i-th header tag is:
[0092]
[0093] Similarly, in x samples, for the j-th tail tag, the document is represented as Then the prototype of the j-th tail tag is:
[0094]
[0095] Here, avg represents the mean of the vector.
[0096] In this embodiment, a transfer learner is constructed in the weight transfer module. A mapping method is used to map the header prototype to the header label weights. The transfer weights are obtained by minimizing the transfer learner, and are expressed as follows:
[0097]
[0098] Where τ represents the transfer learner function, Let Wt represent the prototype of the i-th header label, and Wt represent the transition weight. This represents the weight of the i-th header label.
[0099] When fusing the head classifier and the tail classifier to obtain the final classifier, the specific steps include:
[0100] By leveraging the concept of self-attention, the correlation between head and tail labels is considered. Furthermore, by utilizing the self-attention mechanism, while obtaining the correlation between head and tail labels, a linear combination of the correlation between head and tail labels is also obtained, expressed as:
[0101]
[0102] in, This represents a linear prototypical combination of header and footer tags. This represents the prototype of the i-th header tag. This represents the prototype of the j-th tail tag.
[0103] Using an attention mechanism, the correlation between the header and tail label prototypes is considered to obtain the attention prototype. An additive attention mechanism is used here, resulting in superior performance. Further normalization yields an attention score, which, combined with the header label, gives the attention prototype, leading to a new prototype representation. The specific processing is as follows:
[0104]
[0105] a ji =softmax(e ji )
[0106]
[0107]
[0108] Among them, e ji W3 represents the correlation between the head and tail labels after additive attention processing, and W3 represents the attention weight. ji This represents the attention score. p represents the attention prototype of the j-th header label. j This represents the latest prototype representation, which is a prototype with improved generalization ability.
[0109] Prototype fusion is performed to obtain the latest tail label classifier. This is then combined with the head label classifier to obtain the final classifier. By incorporating transfer weights and a prototype with improved generalization ability, a tail label prototype is obtained. Finally, the head label classifier and the tail label classifier are concatenated to obtain the final text classifier. The specific processing is as follows:
[0110]
[0111] W = [W head :W tail ]
[0112] in, W represents the weight of the j-th tail label. tail represents the tail label weight, and W represents the final text classifier.
[0113] After the above training, a classifier for classification is obtained. The text to be classified is then classified by the classifier to obtain the final classification result, which is specifically represented as follows:
[0114]
[0115] in, The classification result is represented by r, where r represents the text to be classified. Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.
Claims
1. A multi-label text classification method based on long-tail problems, characterized by, The text classification model is constructed to include a text semantic extraction module, a weight transfer module, a fusion module and a classification module, and the classification process includes the following steps: S1, the text semantic extraction module extracts the text semantics of the head label and the tail label through Bert, BILSTM and attention mechanism, and obtains the head label text representation and the tail label text representation respectively; the process of obtaining the text representation includes: The text is pre-trained by using the Bert model to obtain the pre-trained text representation; The text context dependency of the pre-trained text representation is obtained by using BILSTM; The text context dependency is strengthened by using the attention mechanism to obtain the text representation; S2, the head label prototype and the tail label prototype are obtained by sampling the head label text representation and the tail label text representation respectively; S3, training the head label classifier by using the head label text representation, obtaining the head label classifier, and obtaining the head label weight by minimizing the loss function of the head label classifier; the head label classifier comprises a layer of Sigmoid activation function, the text representation is input into the head label classifier, a cross-entropy loss function is used as the loss function of the head label classifier, and the head label weight is obtained by minimizing the loss function , is represented as: wherein, a classification result predicted by the head tag classifier, a text representation, a weight representing the head classifier, a label number corresponding to the head tag, an i-th text, a set of texts corresponding to the head tag, a predicted probability that the i-th text corresponds to a j-th head tag, an actual probability that the i-th text corresponds to the j-th head tag, and loss represents a loss function. S4, the weight transfer module maps the head label prototype to the head label weight, and obtains the transfer weight by minimizing the transfer learning; S4, the weight transfer module transfers the head label weight according to the transfer weight to obtain the tail label classifier; the construction process of the tail label classifier includes the following steps: The linear combination of the correlation between the ith head label and the jth tail label is obtained, which is represented as: ； The attention mechanism is used to process a linear combination of the relevance of the i-th head label to the j-th tail label, and the i-th head label prototype is weighted to obtain a new prototype of the j-th tail label, denoted as: The new prototype of the jth tail label is obtained by using the transfer matrix to obtain the corresponding tail label classifier, which is represented as: ； wherein, denotes a linear combination of head tags to tail tags, denotes the i-th head tag prototype, denotes the j-th tail tag prototype; denotes the head tag and tail tag correlation after additive attention mechanism processing, denotes the attention weight, denotes the attention score, denotes the j-th head tag attention prototype, avg denotes the mean of the vector, denotes the new prototype of the j-th tail tag; denotes the transition weight matrix, denotes the tail tag classifier weight of the j-th tag; S5, the fusion module fuses the tail label classifier and the head label classifier to obtain the classification module; S6, the text representation of the data to be classified is input into the classification module to obtain the classification result.
2. The multi-label text classification method based on long tail problem according to claim 1, characterized in that, The text is pre-trained by Bert, and the output result is , n represents the maximum word length, is the text embedding vector corresponding to the pth word; the adopted Bert dimension is 768 dimensions, the number of multi-head attention mechanism is 12, and the number of Transformer layers is 12. 3.The method of claim 2, wherein, The text context dependency of the pre-trained text representation obtained by BILSTM includes: wherein, is a text embedding vector corresponding to the pth word; is a text context dependency, , is a forward text context dependency, is a backward text context dependency, denotes a forward hidden state at p, denotes a backward hidden state at p.
4. The multi-label text classification method based on long tail problem according to claim 3, characterized in that, The text context dependency is strengthened by using the attention mechanism to obtain the text representation, which includes: where e represents the contribution of all words to the text, o is the text representation, , is a trainable matrix, d represents the dimension of the document representation.
5. The method of claim 1, wherein the method is based on a long-tail problem. The head label prototype and the tail label prototype are obtained by sampling the head label text representation and the tail label text representation respectively, which includes: In x samples, for the i-th head label, the document representation is , then the i-th head label prototype is: ; In x samples, for the jth tail label, the document representation is , then the jth tail label prototype is: ; wherein, represents the vector representation of the x-th sample for the i-th head label; represents the vector representation of the x-th sample for the j-th tail label; avg{} represents taking the average over the vectors.
6. The method of claim 1, wherein the method is based on a long-tail problem. The process of the weight transfer module mapping the head label prototype to the head label weight and obtaining the transfer weight by minimizing the transfer learning includes: ； wherein, represents a transfer learner function, represents a number of labels corresponding to a head label, represents an i-th head label prototype, represents a transfer weight matrix, represents a weight of an i-th head label, is a set of left and right head label weights; l head is a number of head labels.
7. The method of claim 1, wherein the method is based on a long-tail problem. The text representation of the data to be classified is input into the classification module to obtain the classification result, which includes: wherein, represents a classification result, represents a textual representation of the data to be classified; represents a head tag weight; represents a tail tag weight.