Large-scale multi-label text classification method based on label-adaptive text representation

Abstract

The application discloses a large-scale multi-label text classification method based on label adaptive text representation. The application firstly explores label adaptive representation of the text to effectively process classification performance of head labels and tail labels under large-scale multi-label classification; a pre-trained language model is used to learn a representation pool for the text, so that different labels can focus on different representations to complete correlation discrimination. Considering the characteristics of deep models and long texts, text representation enhancement is proposed to ensure the difference and comprehensiveness of the representations in the pool. Therefore, the application can provide effective discriminative text features for large-scale labels to improve the prediction performance. Compared with current large-scale multi-label algorithms, the application can guarantee the overall classification performance of large-scale multi-labels on the one hand, and guarantee that tail labels can better focus on detailed text features on the other hand, and the performance is better than that of the current most advanced large-scale multi-label algorithm.

Description

TECHNICAL FIELD

[0001] The present application relates to a large-scale multi-label text classification technology, and in particular to a label adaptive representation based large-scale multi-label text classification (LAR-LMTC) method. BACKGROUND

[0002] The purpose of large-scale multi-label text classification (LMTC) is to assign the most relevant label subset to each text from a large-scale label pool. Nowadays, LMTC appears in all aspects of production and life due to its wide range of applications, such as e-commerce dynamic search recommendation, legal judgment and medical diagnosis, etc., and has received extensive attention in the academic field.

[0003] Due to the distribution characteristics of data, the labels in LMTC often exhibit high sparsity, diversity and bias, which seriously affect the classification performance. In order to solve this problem, a large number of studies have proposed to solve these problems by introducing new optimization objectives or utilizing label hierarchy. However, few people pay attention to how to learn effective text representation in LMTC. The existing text representation learning methods mainly have the following two kinds: (1) A simple method is to learn holistic text representation (HTR), that is, to learn a shared text representation for all labels. However, the head label (a label that exists universally in the data set) is easy to dominate the learning process, so that the learning of the tail label (a label with sparse samples) is inhibited, and the prediction performance is damaged. (2) Another method is to decompose the text into multiple label-specific representations (LSR) based on attention networks. However, due to the label sparsity, the learning of the tail label is prone to overfitting. In addition, excessive attention to the tail label will also affect the prediction performance of the head label. Therefore, the existing research has not conducted sufficient research on representation learning in LMTC, especially the representation learning of the tail label. SUMMARY

[0004] In view of the deficiencies of the prior art, the purpose of the present application is to provide a large-scale multi-label text classification method based on label adaptive representation algorithm.

[0005] The technical solution adopted in this invention is as follows: A large-scale multi-label text classification method based on label adaptive text representation, comprising the following steps:

[0006] (1) For each original text x in the dataset i Using a pre-trained Transformer-based language model as the feature encoder E(·), the original text x is processed. i Process into a tokenized text sequence and extract the maximum length L. ent sequence q ent The subscript ent represents the entire text; feature enhancement is performed from both feature depth and text granularity to generate text base features;

[0007] (2) Construct a text representation pool using the basic features of this paper, and make the representations label-agnostic when constructing the representation pool;

[0008] (3) Achieve adaptive matching between large-scale labels and text representation pools: In constructing the text representation pool M l With M f Then, match the most relevant representation in the pool for each label and predict its relevance score;: match each label with representations in both pools at the same time; (4) use the learned model to predict the input text x and the relevant label.

[0009] The beneficial effects of this invention are:

[0010] 1. The text representation pool generated by enhancing features in two aspects can better cover the discriminative line information in the text;

[0011] 2. Labels can adaptively match the most relevant text representations to themselves, thus each representation no longer needs to focus on learning all labels, and it is also more conducive to associating tail labels with detailed text features. Extensive experiments have demonstrated that the prediction performance of the LAR-LMTC proposed in this invention is superior to the current state-of-the-art LMTC algorithm, and it can effectively handle LMTC tasks. Detailed Implementation

[0012] The technical solution of the present invention will now be further described with reference to specific embodiments.

[0013] Given a training dataset Where x is the input text, y is an L-dimensional multi-hot label vector, and N represents the number of samples in the dataset. For each label l, the label y... l =1 indicates that label l is related to text x, y l =0 indicates no correlation. The goal of the LMTC task is to learn a fractional function f: To measure the relevance between input text x and label l, a simple and direct approach is to learn a one-vs-all model: Among them, w l Φ represents the parameters of the label classifier. l (.) represents the feature embedding process for the d-dimensional vectorized encoding of the text for label l. After obtaining the predicted scores for all labels, the top k labels are taken as the final relevant labels.

[0014] The specific embodiments of the present invention and their implementation process are as follows:

[0015] Step 1: For each original text x in the dataset i We use a pre-trained Transformer-based language model (such as BERT) as the feature encoder E(·) and apply two aspects of feature enhancement to generate text base features.

[0016] First, the original text x i Process into a tokenized text sequence and truncate to a maximum length of L. ent sequence q ent The subscript ent indicates the entire text.

[0017] Then, feature enhancement is performed from the aspects of feature depth and text granularity to generate text base features.

[0018] Aspect 1: Feature enhancement in model depth. Utilizing multi-layered features from pre-trained language models to capture more semantic information from the text. In fact, LMTC is not a standard planar multi-label classification; large-scale labels often hide label hierarchies, which encourages labels to focus on different semantics.

[0019] Let the length be L ent text sequence q ent Input a pre-trained language model (Transformer-based model) and use the feature vectors from the model's last multiple layers to generate multi-level basic features for this paper. Where d is the dimension of the feature, n l This indicates the number of layers from which model features are selected.

[0020] Aspect Two: Feature Enhancement at the Text Granularity Level. Because Transformer-based models utilize a self-attention mechanism to maintain long-distance dependencies within long texts, feature learning remains within the global receptive field, weakening the representation of a significant amount of local information. Since tail labels tend to focus on detailed information, single global features will lose a large amount of discriminative information. This is addressed by using fragmented text sequences to supplement fine-grained text features. First, a sequence of length L... ent The original text sequence is uniformly divided into segments of length L. frag And non-overlapping n frag A series of equal-length segmented text sequences The subscript frag indicates fragmented text.

[0021] Then, the original sequence q ent Together with the segmented text sequence Q frag As input to a pre-trained language model Obtaining multi-granularity text base features in, One represents the global features of the entire sequence, while the others represent the local features of different sequence segments.

[0022] Step Two: Construct a text representation pool using two aspects of the text's base features. The goal is to enable large-scale labels to adaptively match representations within the pool. During pool construction, the representations are made label-agnostic. Learning a representation pool for text has two advantages: First, multiple representations can describe different aspects of the text; different representations emphasize different discriminative features, allowing labels to adaptively match their relevant representations. Second, the representation pool reduces the risk of acquiring incorrect text representations, enabling all representations to collaboratively complete the full-label classification task.

[0023] Using text base features H respectively l With H f Each of these is used to generate its own representation pool to avoid overfitting. This allows for the use of text-based base features H... l For example, specifically:

[0024] H l Input Multi-Attention Network MA l (·), calculate H l Multiple linear combinations between them are used to obtain an attention representation. Among them, P l This represents the size of the text representation pool, which is also the number of attention maps. M l The j-th representation It can be obtained through the following:

[0025]

[0026] Among them, w j It is the parameter of the j-th attention unit, α ij yes The normalized coefficients. It is worth noting that no constraints are imposed on the labels and representations to limit the correlation, so constructing all representations together reveals critical information about all labels.

[0027] Similarly, using text base features H f The same text representation pool can be obtained.

[0028] Step 3: Achieve adaptive matching between large-scale labels and the text representation pool. This involves constructing the text representation pool M. l With M f Then, for each label, the most relevant representation in the pool is matched and its relevance score is predicted. To avoid overfitting, classification is still performed on M. l With M f This allows each label to match representations in both representation pools simultaneously. This enables the use of representation pool M. l For example, specifically:

[0029] For each attention representation It is fed into a fully connected layer and activated by the Sigmoid function. l (·), to obtain the correlation score between this representation and the full label. in b cls ∈R L The parameters of the fully connected layer are shared across all representations to reduce model size while avoiding overfitting.

[0030] Subsequently, the relevance scores of all attention annotations and full labels are integrated to obtain a matrix.

[0031] Will Through the maximum pooling layer MP l (·) Select the maximum value for each label k to represent its final relevance score. Therefore, for each text, in M l Its final full-label prediction score is

[0032] Finally, the loss is calculated using the binary cross-entropy loss (BCE Loss) as the objective function, as follows:

[0033]

[0034] Similarly, using the characterization pool M f The corresponding full-label prediction score can be obtained in the same way. And calculate the loss:

[0035]

[0036] By combining the two losses mentioned above with a multi-task learning model, the final loss can be obtained. Where γ is the balancing weight coefficient. The model parameters are updated using stochastic gradient descent.

[0037] Step 4: Use the learned model to predict the input text x and related labels. First, obtain the serialized input q. ent With Q f Subsequently, a correlation score based on two-sided feature enhancement was calculated:

[0038]

[0039]

[0040] Combine the two to obtain the final score. Where γ is the balancing weight coefficient. This score is a robust prediction score obtained by finding the most relevant representations for each label from two aspects.

[0041] Finally, we get The top-k labels are used as predicted labels for text x.

Claims

1. A large-scale multi-label text classification method based on label adaptive text representation, characterized in that, The steps include: (1) For each original text x in the dataset i Using a pre-trained Transformer-based language model as the feature encoder E(·), the original text x is processed. i Process into a tokenized text sequence and extract the maximum length L. ent sequence q ent The subscript ent represents the entire text; feature enhancement is performed from both feature depth and text granularity to generate text base features; (2) Construct a text representation pool using text base features, making the representations label-agnostic during the construction of the representation pool; in step (2), the text base features H l Input Multi-Attention Network MA l (·), calculate the text base features H l Multiple linear combinations between them are used to obtain attention representations. Among them, P l M represents the size of the text representation pool, i.e., the number of attention maps; l The j-th representation Obtained through the following formula: Among them, w j It is the parameter of the j-th attention unit, α ij yes The normalized coefficient; Using text base features H f Obtain the text representation pool (3) Achieve adaptive matching between large-scale labels and text representation pools: In constructing the text representation pool M l With M f Then, predict the relevance score of the most relevant representation in the matching pool for each label; and let each label match the representation in both representation pools simultaneously. (4) Using the learned model, predict the input text x and related labels.

2. The large-scale multi-label text classification method based on label adaptive text representation according to claim 1, characterized in that, In step (1), the specific steps of generating text base features by enhancing features from both feature depth and text granularity are as follows: (1.1) Feature enhancement in model depth: Use multi-layer features from a pre-trained language model to obtain more semantic information of the text; extend the length of L... ent text sequence q ent Input a pre-trained Transformer-based language model and use the feature vectors from the model's last multilayer layers to generate text base features. Where d is the dimension of the feature, n l Indicates the number of layers from which model features are selected; (1.2) Feature enhancement at the text granularity level: First, the text granularity of length L is enhanced. ent The original text sequence is uniformly divided into segments of length L. frag And non-overlapping n frag A series of equal-length segmented text sequences The subscript frag indicates fragmented text; subsequently, the original sequence q is... ent Together with the segmented text sequence Q frag As input to a pre-trained language model Obtaining multi-granularity text base features n f =n frag +1, where, One represents the global features of the entire sequence, while the others represent the local features of different sequence segments.

3. The large-scale multi-label text classification method based on label adaptive text representation according to claim 2, characterized in that, In step (3), for each attention representation It is fed into a fully connected layer and activated by the Sigmoid function. l (·), to obtain the correlation score between this representation and the full label. in Parameters for the fully connected layer are shared across all representations. Subsequently, the relevance scores of all attention annotations and full labels are integrated to obtain a matrix. Will Through the maximum pooling layer MP l (·) Select the maximum value for each label k to represent its final relevance score. Therefore, for each text, in M l Its final full-label prediction score is Finally, the loss is calculated using the binary cross-entropy loss (BCE Loss) as the objective function, as follows: Using characterization pool M f Obtain the corresponding full-label prediction score And calculate the loss: By combining the two losses mentioned above with a multi-task learning model, the final loss can be obtained. Where γ is the balancing weight coefficient; the model parameters are updated using stochastic gradient descent.

4. The large-scale multi-label text classification method based on label adaptive text representation according to claim 2, characterized in that, In step (4), firstly, the serialized input q is obtained. ent With Q f Subsequently, a correlation score based on two-sided feature enhancement was calculated: Combine the two to obtain the final score. Where γ is the balancing weight coefficient; finally, we obtain The top-k labels are used as predicted labels for text x.

Images