Small-sample multi-label text classification method based on machine reading comprehension

Abstract

This invention relates to the field of multi-label text classification technology and discloses a few-sample multi-label text classification method based on machine reading comprehension, comprising the following steps: First, the multi-label text dataset is preprocessed, its format is standardized, and labels are added. Next, auxiliary questions are designed based on the labels and combined with text instances to form input data. A RoBERTa model is used as the backbone of the joint encoder to learn the semantic relevance between the text and the questions, and a label graph representing the label relationships is constructed. Through feature fusion, the model is trained to minimize the loss function, and hyperparameters are tuned using a validation set. Finally, the model performance is evaluated on a test set and compared with traditional methods to verify its advantages and component contributions. By transforming multi-label text classification into a machine reading comprehension task, it helps to better capture the semantic relevance between text and labels, significantly improving the accuracy of multi-label text classification.

Description

Technical Field

[0001] This invention belongs to the field of multi-label text classification technology, specifically relating to a few-sample multi-label text classification method based on machine reading comprehension. Background Technology

[0002] Multi-Label Text Classification (MLTC) is an important research topic in the field of natural language processing, aiming to associate text instances with a set of relevant labels. This technology plays a crucial role in various practical applications such as text classification, information retrieval, and dialogue systems. With the explosive growth of internet information, the demand for multi-label text classification technology is becoming increasingly urgent.

[0003] In existing technologies, multi-label text classification methods mainly focus on two research directions: first, constructing the correlation between labels; and second, constructing the semantic correlation between text instances and labels. These methods include graph-based models, co-attention neural networks with label embeddings, and Seq2Seq architectures that combine pre-trained language models.

[0004] Graph-based models, such as MAGNET, incorporate label relevance information into text embeddings through graph-structured label representations.

[0005] Models based on the co-attention mechanism, such as CNLE, further construct tag-specific document representations by building mutual attention between text and tags.

[0006] Hybrid models based on sequence generation models, such as BERT+SGM, combine a pre-trained BERT encoder and a sequence generation model for multi-label text classification tasks.

[0007] Existing methods typically require large, precisely labeled datasets to improve model generalization ability, which is often expensive and time-consuming in practical applications. Real-world classification data often exhibits a long-tailed label distribution, making low- and mid-frequency labels difficult to learn, and existing techniques perform poorly when handling this type of data. Some methods fail to fully consider natural language grammar rules in their model input structure, making it difficult for pre-trained language models to effectively capture the semantic information related to text instances and labels. While some existing models focus on the semantic association between text and labels, they neglect the modeling of label correlations, impacting model performance. Seq2Seq-based methods are sensitive to label order when handling multi-label text classification tasks, heavily relying on manually predefined label ordering, which easily leads to error propagation. Some methods do not fully utilize the semantic understanding capabilities of pre-trained language models, instead relying on large amounts of training samples, making them difficult to transfer to other real-world application scenarios where data is difficult to label. Therefore, existing techniques suffer from strong data dependence, long-tailed label problems, neglect of grammar rules, insufficient label correlation modeling, difficulty in transferability, and sensitivity to label order.

[0008] To address the problems in existing technologies, this invention proposes a few-sample multi-label text classification method based on machine reading comprehension. This method designs three different types of label-assisted questions based on the grammatical rules of natural language. A pre-trained model learns the semantic correlation features between text instances and label-assisted questions to build a label-specific document representation, used to answer these natural language-assisted questions about whether a certain category of label is contained. A label graph relationship construction task is introduced to establish the association between label predictions within the same text instance. By transforming multi-label text classification into a machine reading comprehension task, it helps to better capture the semantic correlation between text and labels, significantly improving the accuracy of multi-label text classification, especially in real-world scenarios with scarce samples and long-tailed label distributions, demonstrating a significant improvement in learning ability for small-sample data and low-to-medium frequency label data. Summary of the Invention

[0009] This invention addresses the shortcomings of existing multi-label text classification techniques, which heavily rely on large-scale labeled datasets, neglect text grammatical structure, and fail to adequately model the correlations between labels. Furthermore, these techniques are sensitive to label order and pose risks of error propagation. These issues collectively lead to insufficient model performance in scenarios with scarce samples and long-tailed label distributions.

[0010] To achieve the above objectives, the present invention provides the following technical solution:

[0011] A few-sample multi-label text classification method based on machine reading comprehension includes the following steps:

[0012] Step 1: Data Preprocessing: Represent the standard MLTC dataset as... ,right Data preprocessing was performed on the medium-sized samples to standardize the text format; a set of label names was created. This is used for constructing subsequent auxiliary questions.

[0013] Step 2: Constructing an auxiliary question-text instance joint encoder: Designing three types of auxiliary questions: pseudo-questions, complete questions, and expert questions. And generate corresponding auxiliary questions for each label. The generated auxiliary problem Combined with the original text instances, new input data is formed; RoBERTa is selected as the pre-trained language model as the backbone of the joint encoder to construct the joint encoder, which combines the text instances... and auxiliary issues As input, the backbone network of the pre-trained model aggregates the semantic features of the input sentence pairs, trains the model using the labeled dataset, constructs label-specific text representations by learning the semantic relevance between text instances and auxiliary questions, and introduces an early stopping mechanism to prevent overfitting.

[0014] Step 3: Construct a label graph representing the relationships between labels: A label graph representing the relationships between labels is built based on a Multi-Head GAT neural network. Generate semantic features related to the output of the joint encoder. Fusion of relevance label features To establish label associations among the label predictions of the same text instance;

[0015] Step 4: Feature Fusion: Extract representation features of text instances and label relationships from the joint encoder, and fuse the semantic features related to text instances output by the joint encoder. Relationship with relevance labels represents features To merge;

[0016] Step 5: Train the model: Train the model using the training set data, adjust the model parameters to minimize the model training loss, validate the model using the validation set data, and perform hyperparameter tuning based on performance metrics.

[0017] Step Six: Experimental Analysis: Evaluate the model's performance on the test set using metrics such as Micro-F1, Macro-F1, and Hamming Loss. Compare the few-sample multi-label text classification method based on machine reading comprehension with traditional machine learning methods and state-of-the-art deep learning methods for the current task. Analyze the performance differences and advantages, and analyze the contribution of each component to the model's performance through ablation experiments.

[0018] Preferably, in step one, the standard MLTC dataset Represented as ,in , These are the input text instances and their corresponding tag sets. The index is the target text instance, where M represents the total number of text instances in the target dataset, and l represents the average loss of the entire dataset.

[0019] As a preferred option, in step one... S The steps for sample data preprocessing are as follows:

[0020] S1.1, Remove text Remove all HTML tags, URLs, and non-English alphabetic characters from the text and replace them with spaces;

[0021] S1.2, transfer the text Convert uppercase letters in Chinese and English to lowercase;

[0022] S1.3, Create a collection of label names ,in The target tag name, Indexing for target tags, This indicates the number of labels in the target dataset.

[0023] As a preferred option, in step two, the auxiliary problem is... After augmentation, the original input of the dataset Represented as: ;

[0024] in, Represents a given text instance In the target-aid problem The tags on the page, each tag type There are designated auxiliary issues. Associated with it, and each text instance Will be with the same set of auxiliary problems Matching.

[0025] Preferably, in step two, the auxiliary problem-text instance joint encoder module includes the following:

[0026] A length of augmented text sequence From length of Auxiliary problem sequence With length Text instance sequence Together they form, namely: ;in , These represent the number of document instances and the number of tags in the target dataset, respectively.

[0027] In the RoBERTa model, the augmented text sequence is represented as:

[0028] ;

[0029] Given auxiliary problem With text instances Sentence pairs are concatenated with special tokens and used as input to the RoBERTa pre-trained model;

[0030] ;

[0031] RoBERTa uses and As special tokens, different BERT-like pre-trained models have different special tokens for sentence pair segmentation;

[0032] ;

[0033] Start token The corresponding output vector aggregates the relevant semantic features of the input sentence pairs. , which is used as a hidden state for feature fusion.

[0034] Preferably, in step three, the tag map is based on the Multi-Head GAT. The build phase includes the following:

[0035] Multi-Head GAT employs a two-layer cascaded GAT layer, where each GAT layer adaptively and dynamically assigns differentiated weights to each node and its neighboring nodes through attention.

[0036] The input to Multi-Head GAT is a tag name embedding matrix. As a node feature, the adjacency matrix The adjacency matrix is ​​randomly initialized and serves as a learnable parameter for the model. By learning the adjacency matrix, graph relationships are determined, and a label-related hidden state representation for feature fusion is generated. Indicates the number of tags. The embedding dimension is represented by a node update mechanism that is a weighted sum of neighboring nodes and attention coefficients, calculated as follows:

[0037] For the first layer, the original node features are the label name embedding matrix. The updated node features of the first layer are used as the original node features of the second layer Multi-Head GAT.

[0038] ; ;

[0039] in, For weight parameters, Represents the original node features, and the new node features represent... Based on the original node features Obtained through linear transformation, Index the target node. The index of the neighboring nodes of the target node;

[0040] ;

[0041] node Its neighboring nodes The similarity coefficient between nodes is calculated by feature concatenation. ,in For weight parameters,

[0042] ;

[0043] Randomly initialized adjacency matrices are used as trainable parameters for the model, and attention coefficients are calculated based on a two-layer cascaded Multi-HeadGAT algorithm. ,in Used to characterize the in Layer The first image draws attention to the head. The neighbor node updates the first... The importance of each node;

[0044] ;

[0045] On each layer, use Different graph attention heads capture the correlation between labels and aggregate the output based on features. The node update mechanism is a weighted sum of neighboring nodes and attention coefficients. The output of the last layer is the correlation label features. Relevant semantic features used to output to the joint encoder Incorporate tag relevance.

[0046] Preferably, in step four, the relevant semantic features output by the joint encoder are... Related label features The features are fused to obtain the final feature representation for classification. The calculation method is as follows: ;

[0047] Final text features A classifier with a fully connected layer captures the final representation of the document for classification. Fine-grained characteristics of different regions ,in , , Represents a text instance In the label The confidence level is used to make the final prediction of the label with a probability threshold of 0.5. The prediction results of different target labels for the same text instance are combined into the final result of multi-label text classification prediction.

[0048] Preferably, in step five, the preprocessed training data is input into the multi-label text classification model to obtain the label prediction probability, using the binary cross-entropy loss function, calculated as follows:

[0049] ;

[0050] Among them, the true labels of the data points are The AdamW optimizer is used for training, with the goal of learning a prediction function. It is used to predict whether unseen text belongs to a specified category. During training, a warm-up strategy is used, with an initial learning rate of 0, a maximum learning rate of 1e-5, and a warm-up step count of 10% of the total training steps. The auxiliary question type is a complete question, the dropout parameter is 0.5, and the number of graph attention heads is 4 or 8. An early stopping mechanism is adopted, and training is stopped when the recent loss is not lower than the average loss of the most recent 10 epochs.

[0051] As a preferred embodiment, in step six, the calculation formulas for the three evaluation metrics, Micro-F1, Macro-F1, and Hamming Loss, are as follows:

[0052] ;

[0053] ;

[0054] .

[0055] Compared with the prior art, the technical effects and advantages of the present invention are:

[0056] This few-shot multi-label text classification method based on machine reading comprehension first designs three different types of label-aiding questions, which are input along with a given text instance into a pre-trained language model, such as the RoBERTa model. The model learns the semantic relevance between the text instance and the label-aiding questions to construct a set of label-specific text representations. Next, through a label graph relationship construction task, a graph attention mechanism is used to adaptively establish optimal relationships between labels, thereby predicting multiple labels for the text.

[0057] Compared to traditional multi-label text classification methods, a significant advantage of this invention is its performance under small sample conditions. Thanks to the powerful semantic understanding capabilities of the pre-trained language model, this method can still achieve effective classification of multi-label text with only a small number of training samples, which is particularly important in scenarios where data annotation is costly or data acquisition is difficult. Another advantage is its superior performance when dealing with datasets with long-tailed label distributions. Long-tailed label distributions mean that the number of samples for some labels is very small, which usually leads to poor classification performance for these labels. However, this invention, by constructing label-specific text representations and graph relationships between labels, can effectively improve the recognition ability of low- and mid-frequency labels, ensuring the generalization performance of the model.

[0058] Furthermore, this invention improves model transferability. Because the method does not rely on a large number of training samples, the model can more easily adapt to different datasets and scenarios. This robustness allows the model to be quickly deployed in different applications, providing an efficient solution for multi-domain text classification problems. Attached Figure Description

[0059] Figure 1 This is a flowchart of the multi-label text classification process based on machine reading comprehension in this invention.

[0060] Figure 2 This is a schematic diagram illustrating the transformation of the multi-label text classification of the present invention into a machine reading comprehension task;

[0061] Figure 3 This is a structural diagram of the MLC2MRC framework of the present invention;

[0062] Figure 4 The graph shows the performance of the MLC2MRC method of this invention and the baseline method under different percentages of training samples;

[0063] Figure 5 The graph shows the performance of the MLC2MRC and baseline methods of this invention under different frequency tag groups. Detailed Implementation

[0064] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0065] like Figure 1 As shown, the present invention discloses a few-sample multi-label text classification method based on machine reading comprehension, which is implemented through the following specific steps:

[0066] Step 1: Data Preprocessing

[0067] The standard MLTC dataset can be represented as: ,in , These are the input text instances and their corresponding tag sets. For indexing target text instances, M represents the total number of text instances contained in the target dataset. Data preprocessing for medium-sized samples:

[0068] (1) Remove text All HTML tags, URLs, and non-English alphabetic characters in the text are replaced with spaces.

[0069] (2) Text Convert uppercase letters in Chinese and English to lowercase.

[0070] (3) Create a collection of tag names ,in The target tag name, Indexing for target tags, This indicates the number of labels in the target dataset.

[0071] Table 1 shows the statistical information of the dataset, in which , and These represent the total number of training documents, test documents, and tags, respectively. This represents the average document length in the dataset. This represents the average number of tags related to the document. Task Indicates the dataset task type.

[0072] Table 1. Dataset Statistics

[0073]

[0074] This embodiment uses three widely used multi-label text classification datasets, as shown in Table 1, which contain label categories and training documents of different sizes to verify the performance of the present invention in various scenarios.

[0075] Step 2: Constructing an auxiliary problem-text instance joint encoder

[0076] Pre-trained Transformer models with fine-tuned BERT-like features have demonstrated significant performance in many downstream NLP tasks, and numerous studies have fully demonstrated their superior semantic understanding capabilities. [7][8] It can deeply understand the semantic information in the input text. For example Figure 2 As shown, the multi-label text classification task is treated as a machine reading comprehension task. The model answers natural language auxiliary questions about whether a certain category label is contained based on the semantic information in the text instance: yes (1) or no (0). This enhanced semantic understanding capability enables the model to exhibit higher accuracy in multi-label text classification without relying on a large amount of training data.

[0077] Based on the target dataset Chinese target tag name With task type Task Together, they generate augmented text instances for each tag. auxiliary problems and text instances Both are used as input to the joint encoder. The RoBERTa pre-trained model serves as the backbone of the joint encoder, aggregating the semantic features of the input sentence pairs, specifically including the following:

[0078] 2.2.1 Augmentation of Auxiliary Problems

[0079] Three types of auxiliary problems are defined in this invention. : Pseudo-problems, complete problems, and expert problems.

[0080] Table 2. Paradigm for Auxiliary Question Generation

[0081]

[0082] Table 2 shows three types of question generation paradigms and a set of examples, among which Indicates the target tag name. Task Indicates the task type of the target dataset.

[0083] After augmentation by auxiliary problems, the original input of the dataset Represented as:

[0084] ;

[0085] in Represents a given text instance In the target-aid problem The tags on the page, each tag type There are designated auxiliary issues. Associated with it, and each text instance Will be with the same set of auxiliary problems Matching.

[0086] 2.2.2 Auxiliary Issues - Document Instance Joint Coding

[0087] In the machine reading comprehension-based auxiliary question and text instance correlation feature extraction stage, RoBERTa is selected as the joint encoder backbone to capture the semantic correlation features between text instances and label auxiliary questions, constructing text representations specific to each label auxiliary question for subsequent classification, thus realizing the transformation from multi-label text classification task to machine reading comprehension task. Figure 3 As shown, the auxiliary problem-joint encoder module includes the following:

[0088] A length of augmented text sequence From length of Auxiliary problem sequence With length Text instance sequence Together they form, namely:

[0089] ;

[0090] in , These represent the number of document instances and the number of labels in the target dataset, respectively.

[0091] In the RoBERTa model, the augmented text sequence is represented as:

[0092] ;

[0093] Given auxiliary problem With text instances Sentence pairs are concatenated with special tokens and used as input to the RoBERTa pre-trained model.

[0094] ;

[0095] RoBERTa uses and As special tokens, different BERT-like pre-trained models have different special tokens for sentence pair segmentation.

[0096] ;

[0097] Start token The corresponding output vector aggregates the relevant semantic features of the input sentence pairs. , which is used as a hidden state for feature fusion.

[0098] In this process, Figure 3 All the modules shown are common modules across different labels. Instead of training the parameters of the common modules independently for each label, we adopted a weight-sharing strategy.

[0099] Step 3: Construct a label graph representing the relationships between labels.

[0100] This invention establishes a label graph representing label relationships based on a Multi-Head Graph Attention (GAT) neural network. This generates label relevance features for fusion with the output features of the joint encoder, thereby establishing label associations among the label predictions for the same text instance. The label graph based on Multi-Head GAT... During the construction phase, such as Figure 3 The label relevance capture module, as shown, includes the following:

[0101] Multi-Head GAT employs two cascaded GAT layers. Each GAT layer adaptively and dynamically assigns differentiated weights to each node and its neighboring nodes through attention to capture more complex graph structure information.

[0102] The input to Multi-Head GAT is a tag name embedding matrix. As a node feature, the adjacency matrix... The adjacency matrix is ​​randomly initialized and serves as a learnable parameter for the model. By learning the adjacency matrix, graph relationships are determined, generating a label-related hidden state representation for feature fusion. Indicates the number of tags. This represents the embedding dimension. The node update mechanism is a weighted sum of neighboring nodes and attention coefficients, calculated as follows:

[0103] For the first layer, the original node features are the label name embedding matrix. The updated node features of the first layer are used as the original node features of the second layer Multi-Head GAT.

[0104] ;

[0105] ;

[0106] in, For weight parameters, Represents the original node features, and the new node features represent... Based on the original node features It is obtained through linear transformation. Index the target node. The index of the neighboring nodes of the target node.

[0107] ;

[0108] Furthermore, nodes Its neighboring nodes The similarity coefficient between nodes is calculated by feature concatenation. ,in For weight parameters,

[0109] ;

[0110] Randomly initialized adjacency matrices are used as trainable parameters for the model. Attention coefficients are calculated based on a two-layer cascaded Multi-HeadGAT algorithm. ,in Used to characterize the in Layer The first image draws attention to the head. The neighbor node updates the first... The importance of each node.

[0111] ;

[0112] On each layer, use Each different graph attention head captures the correlation between labels and aggregates the output based on features. The node update mechanism is a weighted sum of neighboring nodes and attention coefficients. The output of the final layer is the correlation label features. Relevant semantic features used to output to the joint encoder Incorporate tag relevance.

[0113] Step 4: Feature Fusion Generation

[0114] Figure 3 The relevant semantic features output by the joint encoder Related label features The features are fused to obtain the final feature representation for classification. The calculation method is as follows:

[0115] ;

[0116] Final text features A classifier with a fully connected layer captures the final representation of the document for classification. Fine-grained characteristics of different regions.

[0117] ;

[0118] in , , Represents a text instance In the label The confidence level is calculated, and finally, a probability threshold of 0.5 is used to predict the label. The prediction results for different target labels of the same text instance will be combined into the final result of multi-label text classification prediction.

[0119] Step 5: Train the model

[0120] The preprocessed training data is input into a multi-label text classification model to obtain the label prediction probability. The binary cross-entropy loss function is used, and the calculation method is as follows:

[0121] ;

[0122] Among them, the true labels of the data points are .

[0123] The AdamW optimizer is used for training, with the goal of learning a prediction function. It is used to predict whether unseen text belongs to a specified category. Figure 3 The modules shown are all common modules across different labels. The model employs a weight-sharing strategy across all labels, eliminating the need for independent parameter training for each label. During training, a warm-up strategy is used, with an initial learning rate of 0 and a maximum learning rate of 1e-5. The number of warm-up steps is 10% of the total training steps. The auxiliary problem type is a complete problem, the dropout parameter is 0.5, and the number of graph attention heads is 4 or 8. An early stopping mechanism is employed: training stops when the most recent loss is not lower than the average loss of the most recent 10 epochs.

[0124] Step Six: Experimental Analysis

[0125] This embodiment presents experimental analyses on three widely used multi-label text classification datasets, containing label categories and training documents of varying sizes, to demonstrate that the present invention maintains good performance across various scenarios. For a fairer comparison, we follow the same dataset partitioning and preprocessing techniques as previous researchers, referring to the following reference: Pal A, Selvakumar M, Sankarasubbu M. Multi-label text classification using attention-based graph neural network[J]. arXiv preprint arXiv:2003.11644,2020. The main evaluation metrics are Micro-F1, Macro-F1, and Hamming Loss, calculated as follows:

[0126] ;

[0127] ;

[0128] For the above formula, Micro-F1 weights the F1-Score of each tag based on its frequency, while Macro-F1 treats each tag equally. The F1-Score of each tag is calculated using three basic elements in its confusion matrix: true positives (1 Positive, TP), false positives (0 Positives, FP), and false negatives (0 Negatives, FN). This represents the total number of categories in a multi-label text classification task. Represents the category index. Indicate category The number of samples correctly predicted as positive by the model. and The meaning is similar.

[0129] ;

[0130] Hamming Loss calculates the proportion of incorrectly predicted labels. In the formula above, This represents the total number of tag categories in the current task. Indicates the tag category index; and These represent the true label and the predicted label of the data, respectively. For multi-label text classification tasks, the label is usually a binary value used to indicate whether the current data belongs to the current label. This represents the XOR operation. The result is 1 when the predicted label does not match the actual label, and 0 otherwise.

[0131] The experimental results on the three datasets are shown in Tables 3, 4, and 5. This means that the higher the score for this indicator, the better the performance. Conversely. With The results indicate that the comparative model's results are cited from the original paper, while the remaining results are experimental results obtained by the inventors after reproducing the model. The sources of the original paper for the baseline model are as follows:

[0132] Fan C, Chen W, Tian J, et al. Accurate use of label dependency in multi-label text classification through the lens of causality[J]. Applied Intelligence, 2023, 53(19): 21841-21857.

[0133] Nam J, Loza Mencía E, Kim HJ, et al. Maximizing subset accuracy with recurrent neural networks in multi-label classification[J]. Advances in neural information processing systems, 2017, 30.

[0134] Liu M, Liu L, Cao J, et al. Co-attention network with label embedding for text classification[J]. Neurocomputing, 2022, 471: 61-69.

[0135] Yarullin R, Serdyukov P. Bert for sequence-to-sequence multi-label text classification[C] / / Analysis of Images, Social Networks and Texts: 9th International Conference, AIST 2020, Skolkovo, Moscow, Russia, October 15–16, 2020, Revised Selected Papers 9. Springer International Publishing, 2021:187-198.

[0136] For metrics not calculated in the original paper, use the "-" symbol:

[0137] Table 3. Predictive power of each comparison method on the Reuters-21578 dataset.

[0138]

[0139] Table 4. Predictive power of each comparison method on the AAPD dataset

[0140]

[0141] Table 5. Predictive power of each comparison method on the Toxic Comment dataset.

[0142]

[0143] Among them, Binary Relevance (BR), Classifier Chain (CC), and Support Vector Machine (SVM) are traditional machine learning methods that transform multi-label text classification into multiple binary text classifications. TextCNN, XML-CNN, and BiLSTM are traditional deep learning methods with fewer model parameters. CFTC BiLSTM By utilizing artificial causal graphs and employing counterfactual classifiers, correlation bias caused by the model's misuse of label dependencies is eliminated, thereby improving prediction accuracy. EncDec applies recurrent neural networks to a Seq2Seq architecture and proposes a label ranking strategy suitable for this architecture. MAGNET is a deep learning method based on graph attention mechanisms, independent of specific label sequences, thus eliminating the Seq2Seq architecture's dependence on predefined label ranking. CNLE, based on the Transformer architecture, co-encodes text and label sequences into mutually engaging representations to focus on the semantic information related to both. BERT and RoBERTa are pre-trained language models based on the Transformer architecture, achieving significant performance improvements in multiple downstream NLP tasks. BERT+SGM Mixed is a hybrid model combining a pre-trained BERT model with a Seq2Seq architecture, achieving state-of-the-art (SOTA) results on three well-studied multi-label text classification datasets and two private datasets. The method proposed in this invention is MLC2MRC.

[0144] MLC2MRC is a technical concept mentioned in this invention, which represents a method for transforming the Multi-Label Text Classification (MLC) task into the Machine Reading Comprehension (MRC) task.

[0145] In multi-label text classification tasks, the goal is to predict the multiple category labels that a given text instance may belong to. Traditional MLC methods typically encode the text directly and then use a classifier to predict the labels.

[0146] The innovation of the MLC2MRC method lies in its approach: instead of directly classifying text, it treats the MLC task as an MRC task. Specifically, it guides the model to understand the text content by designing specific natural language auxiliary questions and answering these questions, thereby indirectly predicting the text's label. This method leverages the strengths of MRC in understanding text semantics, particularly when dealing with small-sample and long-tailed labeled datasets, significantly improving classification accuracy and the model's learning ability.

[0147] By comparing experimental results on three MLTC datasets, the MLC2MRC method significantly improves upon existing baseline methods across all metrics, validating the effectiveness of this invention. Notably, on the Reuters-21578 dataset, the proposed method achieves an 8.68% improvement over the optimal baseline model on the Macro-F1 metric, indicating that this invention has superior predictive ability for low-to-medium frequency labels in unseen data.

[0148] Under the same training environment, we compared the performance of MLC2MRC with existing baseline methods on different percentages of training samples on the Toxic Comment dataset. The results are as follows: Figure 4 As shown, there is a significant gap between the baseline methods and MLC2MRC on a smaller training sample size. MLC2MRC still outperforms most baseline methods on the full sample size even with only 10% of the training samples, indicating that MLC2MRC better utilizes the superior semantic understanding capabilities of the pre-trained model on text. Although this consumes more training resources, it demonstrates the effectiveness of this invention in scenarios where training samples are scarce.

[0149] Maintaining the same training environment, we further compared the performance of MLC2MRC with existing baseline methods on the Reuters-21578 dataset under different frequency label groupings. Figure 5 As shown in (a), the Reuters-21578 dataset exhibits a typical long-tail distribution, dividing all labels into four groups according to frequency. Figure 5 The results in (b) show that the performance of all methods decreases as the frequency of label occurrence decreases, but the performance gap between the baseline method and MLC2MRC increases as the label frequency decreases. This indicates that machine reading comprehension-based methods can improve the model's ability to model low-to-medium frequency label data.

[0150] This machine reading comprehension-based few-sample multi-label text classification method constructs text representations with label semantics specific to the natural language auxiliary problem, enabling more full utilization of the semantic understanding capabilities of pre-trained language models in natural language. The model's better semantic understanding will result in higher multi-label text classification accuracy. Unlike existing methods, this invention does not rely on a large number of training samples, greatly improving the model's application value in data-scarce scenarios.

[0151] The label relevance capture module, based on a multi-head graph attention mechanism, does not rely on manually predefined label order. This mechanism can not only effectively capture complex graph relationships between labels but also handle large-scale label data.

[0152] Experiments demonstrate that this invention performs well on datasets with scarce training samples, showing excellent prediction performance for unseen data and significantly improving the model's ability to model small sample data and long-tailed label distribution data. Simultaneously, it also exhibits superior performance on datasets with abundant training samples. Therefore, this invention can easily achieve model transfer between different scenarios and demonstrates excellent robustness.

[0153] This invention provides a small-sample multi-label text classification method based on machine reading comprehension. The above is merely a preferred embodiment of this invention, intended to enable those skilled in the art to understand the research content and implement it accordingly. Any improvements and modifications made without departing from the principles of this invention should also be considered within the scope of protection of this invention.

Claims

1. A few-sample multi-label text classification method based on machine reading comprehension, characterized in that, Includes the following steps: Step 1: Data Preprocessing: Represent the standard MLTC dataset as S, preprocess the samples in S, clean the data to a uniform text format; create a label name set y. name This is used for constructing subsequent auxiliary questions; Step 2: Constructing an auxiliary question-text instance joint encoder: Designing three types of auxiliary questions Q: pseudo-questions, complete questions, and expert questions. j And generate a corresponding auxiliary question Q for each label. j The generated auxiliary problem Q j Combined with the original text instance, it forms new augmented input data; We select a BERT-like pre-trained language model as the backbone of the joint encoder to construct the joint encoder, which processes text instances X. i And auxiliary question Q j As input, a BERT-like pre-trained language model is used to aggregate the semantic features of input sentence pairs. The model is trained using a labeled dataset. Label-specific text representations are constructed by learning the semantic relevance between text instances and auxiliary questions. An early stopping mechanism is introduced to prevent overfitting. Step 3: Construct a label graph representing the relationships between labels: A label graph representing the relationships between labels is built based on a Multi-Head GAT neural network. Generate semantic features H related to the output of the joint encoder. <s>< / s> <s>Fusion of relevance label features To establish label associations between label predictions for the same text instance;< / s> <s> Step 4: Feature Fusion: Extract representation features of text instances and label relationships from the joint encoder, and fuse the semantic features H related to text instances output by the joint encoder. <s>< / s> <s>The relationship between the relevance label and the feature H gat To merge;< / s> <s> Step 5: Train the model: Train the model using the training set data, adjust the model parameters to minimize the model training loss, validate the model using the validation set data, and perform hyperparameter tuning based on performance metrics. Step Six: Experimental Analysis: Evaluate the model's performance on the test set using Micro-F1, Macro-F1, or Hamming Loss metrics. Compare the machine reading comprehension-based few-sample multi-label text classification method with traditional machine learning methods and state-of-the-art deep learning methods for current tasks to analyze performance differences and advantages. Analyze the contribution of each component to the model's performance through ablation experiments. Furthermore, explore the effectiveness of the proposed method in scenarios with few samples and imbalanced labeled data.

2. The few-sample multi-label text classification method based on machine reading comprehension according to claim 1, characterized in that: In step one, the standard MLTC dataset S is represented as S = {(X...} i Y i )|1≤i≤M}, where X i Y i , i, and M represent the input text instance and its corresponding label set, respectively; i is the index of the target text instance; and M represents the total number of text instances in the target dataset.

3. The few-sample multi-label text classification method based on machine reading comprehension according to claim 1, characterized in that: In step one, the preprocessing steps for sample S data are as follows: S1.1, Remove text X i Remove all HTML tags, URLs, and non-English alphabetic characters from the text and replace them with spaces; S1.2, Transfer text X i Convert uppercase letters in Chinese and English to lowercase; S1.3, Create a collection of label names in is the target label name, j is the target label index, and N represents the number of labels in the target dataset.

4. The few-sample multi-label text classification method based on machine reading comprehension according to claim 1, characterized in that: In step two, through the auxiliary question Q j After augmentation, the original input S of the dataset is represented as: S′={(X i ,Q j ,y ij )|1≤i≤M,1≤j≤N}; Among them, y ij Represents a given text instance X i In the goal-aided problem Q j The tags on the page, each tag type y ij ∈Y i There are designated auxiliary questions Q j ∈Q is associated with it, and each text instance X i It will match the same set of auxiliary problems Q.

5. The few-sample multi-label text classification method based on machine reading comprehension according to claim 1, characterized in that: In step two, the auxiliary problem-text instance co-encoder module includes the following: An augmented text sequence D of length l ij Given an auxiliary problem sequence Q of length n j A sequence of text instances X of length m i Together they form: D ij ={(Q j ||X i )|1≤i≤M,1≤j≤N};where M and N are the number of document instances and the number of tags in the target dataset, respectively; In the RoBERTa model, the augmented text sequence is represented as: < / s> < / s> <s> <s> D ij ＝ <s>Q j < / s> < / s> X i < / s> in RoBERTa; Given the auxiliary problem Q j With text instance X i Sentence pairs are concatenated with special tokens and used as input to the RoBERTa pre-trained model; <s>< / s> in RoBERTa; RoBERTa uses <s> and< / s> As special tokens, different BERT-like pre-trained models have different special tokens for sentence pair segmentation; H <s>< / s> <s>=RoBERTa(D ij ), starting token <s>The corresponding output vector aggregates the relevant semantic features H of the input sentence pairs. <s>< / s> <s> , which is used as a hidden state for feature fusion.< / s> < / s> < / s> <s> <s> <s> 6. The few-sample multi-label text classification method based on machine reading comprehension according to claim 1, characterized in that: In step three, the label graph is based on Multi-Head GAT. The build phase includes the following: Multi-Head GAT employs a two-layer cascaded GAT layer, where each GAT layer adaptively and dynamically assigns differentiated weights to each node and its neighboring nodes through attention. The input to Multi-Head GAT is the tag name embedding matrix. As a node feature, the adjacency matrix The adjacency matrix is ​​randomly initialized and serves as a learnable parameter for the model. By learning the adjacency matrix, graph relationships are determined, and a label-related hidden state representation for feature fusion is generated. Here, N represents the number of labels, d represents the embedding dimension, and the node update mechanism is a weighted sum of neighboring nodes and attention coefficients, calculated as follows: For the first layer, the original node features are the label name embedding matrix. The updated node features from the first layer are used as the original node features for the second layer Multi-Head GAT. in, For weight parameters, Represents the original node features, and the new node features represent... Based on the original node features After linear transformation, p is the index of the target node, and q is the index of the neighboring node of the target node; The similarity coefficient between node p and its neighboring node q is calculated by concatenating features. Where W1 is the weight parameter. Randomly initialized adjacency matrices are used as trainable parameters for the model, and attention coefficients are calculated based on a two-layer cascaded Multi-Head GAT. in Used to characterize the in The importance of the q-th neighbor node to the update of the p-th node is considered at the attention head of the k-th graph. At each layer, K different graph attention heads are used to capture the correlation between labels and the output is aggregated according to features. The node update mechanism is a weighted sum of neighboring nodes and attention coefficients. The output of the last layer is the correlation label features. Relevant semantic features H used for output to the joint encoder <s>< / s> <s> Incorporate tag relevance.< / s> <s> 7. The few-sample multi-label text classification method based on machine reading comprehension according to claim 1, characterized in that: In step four, the relevant semantic features H output by the joint encoder are... <s>< / s> <s>With correlation label feature H gat The features are fused to obtain the final feature representation for classification. The calculation method is as follows:< / s> <s> Final text features A classifier with a fully connected layer captures the final representation of the document for classification. Fine-grained characteristics of different regions in This represents the confidence level of text instance i on label j. The label is then predicted with a probability threshold of 0.

5. The prediction results of the same text instance on different target labels are combined into the final result of multi-label text classification prediction.

8. The few-sample multi-label text classification method based on machine reading comprehension according to claim 7, characterized in that: In step five, the preprocessed training data is input into the multi-label text classification model to obtain the label prediction probability. The binary cross-entropy loss function is used, and the calculation method is as follows: The true label of the data point is y. ij = {0, 1}; Training is performed using the AdamW optimizer, with the goal of learning a prediction function f: D → {0, 1}. M×N It is used to predict whether unseen text belongs to a specified category. During training, a warm-up strategy is used, with an initial learning rate of 0, a maximum learning rate of 1e-5, and a warm-up step count of 10% of the total training steps. The auxiliary question type is a complete question, the dropout parameter is 0.5, and the number of graph attention heads is 4 or 8. An early stopping mechanism is adopted, and training is stopped when the recent loss is not lower than the average loss of the most recent 10 epochs.

9. The few-sample multi-label text classification method based on machine reading comprehension according to claim 1, characterized in that: In step six, the calculation formulas for the three evaluation metrics, Micro-F1, Macro-F1, and Hamming Loss, are as follows: < / s> < / s> < / s> < / s> < / s>

Images