Multi-label text classification method based on semantic representation enhancement and dynamic weighted debiasing contrastive learning and application thereof
By employing semantic representation enhancement and dynamic weighted debiasing contrastive learning, the problems of semantic overlap and incomplete labeling in multi-label text classification are solved, thereby improving the model's classification accuracy and automated labeling capabilities.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- YUNNAN NORMAL UNIV
- Filing Date
- 2026-03-17
- Publication Date
- 2026-06-05
Smart Images
Figure CN121858739B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of natural language processing text classification technology, and in particular to a multi-label text classification method based on semantic representation enhancement and dynamic weighted debiased contrastive learning, and its application. Background Technology
[0002] With the rapid development of natural language processing technology, multi-label text classification has become a core problem in applications such as information extraction, recommendation systems, and intelligent question answering. Compared with single-label text classification, multi-label text classification is characterized by high dimensionality of the label space, complex correlations between labels, and rich and ambiguous text semantics. Traditional multi-label classification methods based on independent binary classifiers or simple cross-entropy loss tend to ignore the structural information of the text representation space when dealing with semantic overlap and incomplete label annotation. They fail to fully utilize the potential semantic relationships between labels, generating false negative samples and leading to a decrease in model classification accuracy.
[0003] In related technical solutions, deep pre-trained language models have been widely used for text feature extraction. By pre-training on large-scale corpora, rich semantic representations can be obtained. However, there are still limitations to directly applying pre-trained models to multi-label text classification: on the one hand, simply using the binary cross-entropy loss function to independently supervise each label fails to fully utilize the semantic correlation between labels; on the other hand, conventional contrastive learning or negative sample weighting methods lack bias removal strategies in multi-label scenarios, do not adequately handle semantic overlap of labels, and are prone to amplifying the interference of false negative samples on the training process.
[0004] Therefore, how to enhance the model's representation ability in scenarios with overlapping and incomplete labels has become an urgent problem to be solved in the field of multi-label text classification. Summary of the Invention
[0005] The main purpose of this application is to provide a multi-label text classification method based on semantic representation enhancement and dynamic weighted debiasing contrastive learning, which aims to solve the problem of how to enhance the model's representation ability in scenarios with multi-label semantic overlap and incomplete labels.
[0006] To achieve the above objectives, this application provides a multi-label text classification method based on semantic representation enhancement and dynamic weighted debiasing contrastive learning, the method comprising:
[0007] S10, Obtain the text sequence and label sequence obtained after performing text segmentation and label encoding on the training dataset, and construct the input tensor representation based on the text sequence and the label sequence;
[0008] S20, the input tensor representation is input into the encoder of the preset model to obtain the outputs of multiple hidden layers of the encoder and then weighted and fused to obtain an enhanced text semantic representation;
[0009] S30, obtain a preset label co-occurrence matrix, transform the preset label co-occurrence matrix into a label adjacency matrix and input it into a graph neural network to obtain a label semantic graph, and process the label semantic graph through residual and layer normalization to obtain label embeddings, map the label embeddings into an adjacency matrix, perform label-aware attention pooling on the enhanced text semantic representation based on the adjacency matrix to obtain a text-label fine-grained alignment vector, and process the text-label fine-grained alignment vector into a multi-label text classification result;
[0010] S40, obtain a loss function based on the debiased weighted contrastive loss and the binary cross-entropy loss function, and constrain the multi-label text classification result with the goal of minimizing the function value of the loss function. The debiased weighted contrastive loss is constructed based on the label similarity matrix, text similarity, dynamic negative sample weights and debiased denominator.
[0011] Optionally, the expression for the debiased weighted contrastive loss is:
[0012]
[0013] In the formula, This indicates the bias-weighted comparison loss. For the label similarity matrix, For dynamic negative sample weights, For text similarity, To remove the partial denominator;
[0014] in,
[0015] In the formula, For any two sample labels in the training dataset and Cosine similarity between them Let be the sum of the cosine similarities of the sample labels in the i-th row.
[0016] Optionally, in step S30, the expression for the label adjacency matrix includes:
[0017]
[0018] in:
[0019]
[0020]
[0021] In the formula, For the label adjacency matrix, To enhance the adjacency matrix, To enhance the adjacency matrix The degree matrix, This is the normalized label co-occurrence matrix. It is a C-order identity matrix, where C represents the order;
[0022] in:
[0023]
[0024]
[0025] In the formula, The number of times the label p appears. For smoothing terms, Represents the label co-occurrence matrix. For any pair of labels, Indicates an indicator function, This indicates that the i-th sample contains label p. This indicates that the i-th sample contains the label q, and N is the total number of samples. This represents the logical AND operation, used to indicate the condition that two labels appear simultaneously in the same sample, if and only if the i-th sample simultaneously satisfies the condition. and When the value is 1, the indicator function takes the value 1; otherwise, it takes the value 0.
[0026] Optionally, in step S30, the preset label co-occurrence matrix is input into a graph neural network to obtain a label semantic graph, and the label semantic graph is processed by residual and layer normalization to obtain label embeddings, including the following steps;
[0027] S31, embed the label matrix in the input tensor representation. Adjacency matrix with labels Perform single-layer graph convolution processing:
[0028]
[0029] In the formula, For the label adjacency matrix, For trainable weight matrix, Activation function
[0030] S32 achieves hierarchical diffusion of semantic information between tags through two stacked graph convolutions to capture the semantic connections contained in tag co-occurrence relationships:
[0031]
[0032]
[0033] In the formula, For trainable weight matrix, This is a temporary fallback function. This indicates the updated label information for the first layer. This represents the updated label information at the second layer, i.e., the label semantic graph;
[0034] S33 introduces residual and layer normalization mechanisms to construct label embeddings. :
[0035]
[0036] In the formula, is the activation function for residuals and layer normalization mechanisms.
[0037] Optionally, in step S30, embedding the label into an adjacency matrix specifically includes:
[0038] S34, embed the tag Perform a linear mapping:
[0039]
[0040] In the formula, The weight matrix is trainable.
[0041] S35, based on linear mapping Calculate the sample-level dynamic adjacency matrix :
[0042]
[0043] In the formula, For the Sigmoid function;
[0044] S36, the sample-level dynamic adjacency matrix Degree normalization is performed to obtain the adjacency matrix. :
[0045] ;
[0046] .
[0047] Optionally, in step S30, the step of performing label-aware attention pooling on the enhanced text semantic representation based on the adjacency matrix to obtain a text-label fine-grained alignment vector includes:
[0048] S37, adjacency matrix The input is fed into a two-layer graph neural network for propagation, resulting in two-layer propagated label embeddings. :
[0049]
[0050] In the formula, Embedding for tags, and These represent the first and second layer neural networks, respectively.
[0051] S38, according to the double-layer propagation tag embedding and the enhanced text semantic representation Determine the document-to-tag attention score matrix. :
[0052]
[0053] In the formula, The learnable temperature coefficient is used to adjust the sharpness of the attention distribution; dim represents the dimension of the label embedding.
[0054] S39, the attention score matrix After normalization, the attention weight matrix is obtained. The attention weight matrix Dot product enhances text semantic representation This yields a fine-grained text-tag alignment vector. :
[0055]
[0056] in, .
[0057] Optionally, in step S30, processing the text-label fine-grained alignment vector into a multi-label text classification result specifically includes:
[0058] The text-label fine-grained alignment vectors are processed by max pooling and multi-branch Dropout, and then a non-linear activation function is used to obtain the multi-label text classification results.
[0059] Optionally, in step S10, constructing the input tensor representation based on the text sequence and the label sequence includes:
[0060] S11, the text sequence With the tag sequence By splicing, a joint sequence is obtained. ;
[0061] S12, determine the joint sequence Is the total length of the sequence less than a preset length threshold?
[0062] S13, if so, select the target sequence in descending order of length, and delete the last word of the target sequence word by word until the total length of the sequence is less than the preset length threshold.
[0063] S14, if not, in the text sequence and the tag sequence Insert a boundary information delimiter [SEP] between the text sequence and the label sequence to distinguish them, and in the text sequence Insert the global representation symbol [CLS] into the header to obtain the input tensor representation. :
[0064] .
[0065] In addition, to achieve the above objectives, this application also provides a multi-label text classification method based on semantic representation enhancement and dynamic weighted debiasing contrastive learning, as described above, and its application in multi-label text classification.
[0066] In addition, to achieve the above objectives, this application also provides a computer system, the computer system comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, it implements the steps of the multi-label text classification method based on semantic representation enhancement and dynamic weighted debiasing contrastive learning as described in any of the preceding claims.
[0067] This application has at least the following beneficial effects:
[0068] 1. By fusing the semantic information of the last four hidden states of the pre-trained language BERT model, we can obtain enhanced text semantic representation and improve the problem of insufficient semantic representation in existing multi-label text classification.
[0069] 2. A dynamic weighted debiased contrastive learning function is proposed to differentially weight negative samples and reduce the interference of false negative samples through a debiasing strategy, which effectively alleviates the problem of semantic overlap of labels and further improves the accuracy of multi-label text classification.
[0070] 3. By combining multi-layer semantic fusion and dynamic weighted bias-removing contrastive learning mechanism, it can effectively handle situations such as overlapping knowledge points, missing labels, or ambiguous label semantics, and achieve end-to-end automated knowledge point annotation and text classification.
[0071] 4. In practical terms, the multi-label text classification method proposed in this application can be widely applied in the education field. Its key feature is its ability to perform high-precision, multi-label semantic analysis and automatic annotation of teaching texts, test questions, and textbook content. This invention can further automatically annotate knowledge points in textbook content, thereby achieving intelligent classification and semantic labeling of questions or texts, providing teachers with functions such as assisting in question creation, test paper generation, and personalized teaching resource recommendations. Attached Figure Description
[0072] Figure 1 This is a flowchart illustrating the multi-label text classification method based on semantic representation enhancement and dynamic weighted debiasing contrastive learning involved in the embodiments of this application.
[0073] Figure 2 This is a flowchart illustrating the multi-label text classification method based on semantic representation enhancement and dynamic weighted debiasing contrastive learning involved in the embodiments of this application.
[0074] Figure 3 The figures show the experimental comparison results involved in the embodiments of this application;
[0075] Figure 4 This is a schematic diagram of the hardware operating environment of the computer system involved in the embodiments of this application.
[0076] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0077] To better understand the above technical solutions, exemplary embodiments of this disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of this disclosure to those skilled in the art.
[0078] First Embodiment
[0079] Reference Figure 1 The flowchart shown, and as follows Figure 2 The flowchart shown illustrates the method. This embodiment provides a multi-label text classification method based on semantic representation enhancement and dynamic weighted debiasing contrastive learning. The method includes the following steps:
[0080] S10, Obtain the text sequence and label sequence obtained after performing text segmentation and label encoding on the training dataset, and construct the input tensor representation based on the text sequence and the label sequence;
[0081] In this embodiment, the input original sample text is first processed. and its corresponding multi-label set The method performs unified subtotalization and label mapping, characterized by ensuring that text semantic information and label information remain independent and fusionable in the vector space, where A represents the number of original samples.
[0082] In some alternative implementations, to ensure the reliability and generalization ability of the model training effect and evaluation results, the dataset used is strictly divided. All data is divided into training set and test set in a ratio of approximately 98.2% and 1.8%, respectively. Giving the training set a higher proportion aims to provide the model with sufficiently diverse samples to capture the potential distribution patterns in the data, thereby improving its generalization performance.
[0083] In some alternative implementations, text segmentation can use the WordPiece segmenter built into the BERT model to perform sub-word-level segmentation on each text sample, generating a word sequence corresponding to each sample text. .
[0084] In some alternative implementations, the set of labels for each sample Perform integer encoding mapping to obtain a label index dictionary. ,in, This is the original index of the tag in the tag set. This is the offset used to avoid conflicts between the label index and the text lexical index, ensuring that the label space and the text space can be distinguished during subsequent tensor concatenation and encoding.
[0085] To facilitate model analysis, the text sequence and the label sequence are constructed as input tensors that are easy for the model to recognize, ensuring that the BERT model can clearly distinguish between document content and label information, and enabling the model to learn text context features and label embedding features simultaneously in the same vector space.
[0086] Further, and optionally, the process of constructing the input tensor representation includes:
[0087] S11, the text sequence With the tag sequence By splicing, a joint sequence is obtained. ;
[0088] S12, determine the joint sequence Is the total length of the sequence less than a preset length threshold?
[0089] In some optional implementations, the preset length threshold is m+n+3>300, where m is the length of the text sequence, n is the length of the label sequence, and 3 is to take into account the three special characters present in the model, namely [CLS][SEP][MASK].
[0090] S13, if so, select the target sequence in descending order of length, and delete the last word of the target sequence word by word until the total length of the sequence is less than the preset length threshold.
[0091] S14, if not, in the text sequence and the tag sequence Insert a boundary information delimiter [SEP] between the text sequence and the label sequence to distinguish them, and in the text sequence Insert the global representation symbol [CLS] into the header to obtain the input tensor representation. :
[0092]
[0093] It should be noted that the introduction of [CLS] is used to aggregate global semantic information of the text sequence during the encoding stage to form an overall semantic representation. Its insertion at the beginning of the text sequence is a preferred implementation consistent with the structure of the pre-trained model and does not constitute a theoretical or technical limitation on the insertion position; it is mostly set to the starting position.
[0094] Furthermore, and optionally, in this embodiment, the input data required by the model can be generated based on the input tensor representation, including:
[0095] (1) Input ID sequence Each text word sequence or label sequence It is mapped to a unique integer ID, which makes it easier for the model's embedding layer to find the corresponding vector representation.
[0096] (2) Attention mask, used to mark the positions of valid lexical terms: Its length is equal to the actual number of input words. In the BERT model's self-attention calculation, only effective words are weighted to avoid interference with the filling positions.
[0097] (3) Type identifier sequence This is used to distinguish paragraph information between text and labels. Text sequences are labeled as 0, and label sequences are labeled as 1. This ensures that the BERT model can identify the source of different sequences in the self-attention mechanism, and achieve the separation and fusion of context and label features.
[0098] Furthermore, and optionally, in this embodiment, the generation logic of the position index mapping of the input tensor representation adopts a dynamic calculation method, which is superior to the traditional static fixed index method. It can maintain high-precision index mapping when processing diverse input samples, avoid confusion of text or label vectors in subsequent processing stages, and adapt to changes in text sequence length and number of labels.
[0099] Specifically, by input sequence Text sequence in With label sequence The lexical positions are mapped to two independent index vectors to obtain the document spatial location index. With tag spatial location index Its mathematical representation is as follows:
[0100]
[0101]
[0102] The offset +2 comes from the [CLS] marker and [SEP] separator before the input sequence, ensuring that the index accurately corresponds to the actual position of the combined sequence.
[0103] S20, the input tensor representation is input into the encoder of the preset model to obtain the outputs of multiple hidden layers of the encoder and then weighted and fused to obtain an enhanced text semantic representation;
[0104] In this embodiment, in order to improve the problem of insufficient semantic representation in existing multi-label text classification, the input tensor representation is encoded and then input into the encoder to obtain the output of multiple hidden layers of the encoder and then weighted and fused to obtain an enhanced text semantic representation.
[0105] In some alternative implementations, the hidden layer output set is defined as follows:
[0106]
[0107] Where B represents the batch size, which is 32; M represents the sequence length, which is 300; dim represents the hidden layer dimension, which is 768; and Deep represents the number of BERT layers, which is 12. However, to obtain smoother semantic features, the last four hidden states are selected and element-wise averaged across the layer dimension. The fusion representation is defined as follows:
[0108]
[0109] Its key feature is that it suppresses noise layer differences by averaging inter-layer information, while capturing the lexical morphological features of the lower layer and the contextual semantic features of the higher layer, thereby obtaining a more stable representation without increasing parameters and preserving deep semantics.
[0110] Finally, for Dropout regularization (with a dropout rate set to 0.2) is applied to prevent overfitting, thereby obtaining an enhanced text semantic representation. .
[0111] S30, obtain a preset label co-occurrence matrix, transform the preset label co-occurrence matrix into a label adjacency matrix and input it into a graph neural network to obtain a label semantic graph, and process the label semantic graph through residual and layer normalization to obtain label embeddings, map the label embeddings into an adjacency matrix, perform label-aware attention pooling on the enhanced text semantic representation based on the adjacency matrix to obtain a text-label fine-grained alignment vector, and process the text-label fine-grained alignment vector into a multi-label text classification result;
[0112] In this embodiment, as a core improvement, enhanced text semantic representation is obtained. Subsequently, in order to achieve fine-grained representation of the relationship between labels, an improved label graph convolutional network is constructed in this step. Regularization, layer normalization, residual connections and label-aware attention pooling are introduced into the graph neural network to enhance the "label-text" interaction.
[0113] Specifically, firstly, a preset tag co-occurrence matrix is obtained, and then the preset tag co-occurrence matrix is transformed into a tag adjacency matrix.
[0114] It should be noted that the purpose of the modified label adjacency matrix is to enable subsequent models to capture the correlation and co-occurrence patterns at the label level. To this end, this embodiment further proposes the steps for constructing the label adjacency matrix:
[0115] Furthermore, and optionally, we first assume that the training set contains N samples and the label set size is C (C=54), using a binary label matrix. This represents the sample-label annotation relationship, where =1 indicates that the i-th sample contains label j. Specifically, for each label... Count the total number of times it appears:
[0116]
[0117] This count forms the basis for subsequent normalization and smoothing. When When =0, it is preferable to set it to 1 to avoid instability caused by division by zero.
[0118] Construct the original co-occurrence count matrix based on sample-level annotation. It is defined as the sum of the number of times any label pair p and q appears simultaneously in all samples:
[0119]
[0120] in This represents an indicator function, and the matrix is a symmetric matrix, with its diagonal elements equal to the occurrence count of each label. .
[0121] Next, to eliminate bias caused by differences in the frequency of different labels, the label co-occurrence matrix is normalized. First, the... Perform row-wise normalization to obtain the normalized co-occurrence matrix. Its elements are defined as:
[0122]
[0123] Here, the denominator is the number of occurrences of label p, representing the relative frequency of label p given label p. Preferably, to avoid noise amplification caused by extremely low frequencies, a smoothing term can be added to the denominator. >0, the specific formula is as follows:
[0124]
[0125] In some alternative implementations, The optimal value is 1, in order to achieve frequency smoothing.
[0126] Then, a self-loop is added to the diagonal to preserve the node's own information, constructing an enhanced adjacency matrix:
[0127]
[0128] in It is a C-order identity matrix, characterized by its self-loop energy preserving the original features of the label during graph convolution propagation, preventing information from being completely replaced by neighbors.
[0129] To ensure numerical stability in graph convolution and to be compatible with standard spectral graph convolution formats, it is preferable to use... The symmetric spectrum normalization is performed, and the calculation process is as follows:
[0130] First, calculate the degree matrix:
[0131]
[0132] To avoid numerical problems caused by a degree of zero, it is preferable to add a lower limit ϵ>0 to the degree. ), and calculate the symmetric normalized label adjacency matrix. :
[0133]
[0134] in It satisfies symmetry and can be used as a propagation matrix in graph convolution, so that the scale of different nodes is balanced during feature propagation, avoiding high-frequency labels from dominating propagation.
[0135] Then, the resulting label adjacency matrix The input graph neural network obtains the label semantic graph, and the label semantic graph is processed by residual and layer normalization to obtain the label embedding, with the aim of improving the generalization ability and robustness of the label embedding.
[0136] Further and optionally, the following steps are included;
[0137] S31, embed the label matrix in the input tensor representation. Adjacency matrix with labels Perform single-layer graph convolution processing:
[0138]
[0139] In the formula, For the label adjacency matrix, For trainable weight matrix, Activation function
[0140] S32 achieves hierarchical diffusion of semantic information between tags through two stacked graph convolutions to capture the semantic connections contained in tag co-occurrence relationships:
[0141]
[0142]
[0143] In the formula, For trainable weight matrix, This is a temporary fallback function. This indicates the updated label information for the first layer. This represents the updated label information at the second layer, i.e., the label semantic graph;
[0144] To prevent semantic degradation and noise accumulation, residual and layer normalization mechanisms are introduced, namely:
[0145] S33 introduces residual and layer normalization mechanisms to construct label embeddings. :
[0146]
[0147] In the formula, is the activation function for residuals and layer normalization mechanisms.
[0148] Next, after obtaining the tag embedding Next, a linear mapping is performed on the label embeddings after label graph convolutional embedding and residual regularization, and a dynamic adjacency matrix is constructed using the similarity of the label embeddings to further refine the labels, specifically including:
[0149] S34, embed the tag Perform a linear mapping:
[0150]
[0151] In the formula, The weight matrix is trainable.
[0152] S35, based on linear mapping Calculate the sample-level dynamic adjacency matrix :
[0153]
[0154] In the formula, This is the Sigmoid function, which maps similarity to the [0,1] interval.
[0155] S36, the sample-level dynamic adjacency matrix Degree normalization is performed to obtain the adjacency matrix. :
[0156] ;
[0157] .
[0158] After obtaining the adjacency matrix Subsequently, label-aware attention pooling is introduced to model the interaction between text features and label embeddings. Specifically, this includes:
[0159] S37, adjacency matrix The input is passed through a two-layer graph neural network to obtain the two-layer propagated label embedding. :
[0160]
[0161] In the formula, Embedding for tags, and These represent the first and second layer neural networks, respectively.
[0162] S38, according to the double-layer propagation tag embedding and the enhanced text semantic representation Determine the document-to-tag attention score matrix. :
[0163]
[0164] In the formula, The learnable temperature coefficient is used to adjust the sharpness of the attention distribution; dim represents the dimension of the label embedding.
[0165] S39, the attention score matrix After normalization, the attention weight matrix is obtained. The attention weight matrix Dot product enhances text semantic representation This yields a fine-grained text-tag alignment vector. :
[0166]
[0167] in, .
[0168] It should be noted that the attention weight matrix can reflect the weight distribution from the label to the document, enabling the model to extract the local information most relevant to the semantics of the label from the overall text.
[0169] After obtaining the fine-grained alignment vector of the text and tag Then, the text-label fine-grained alignment vector is processed into a multi-label text classification result.
[0170] Optionally, the fine-grained alignment vectors of text and labels are max-pooled to retain the most salient global features, and multiple Dropouts are used to enhance the robustness of the samples. The multilayer perceptron with the non-linear activation LeakyReLU function is then used to output the multi-label text classification results.
[0171] Some alternative implementations specifically include:
[0172] First, the sample representation is obtained through max pooling:
[0173]
[0174] Then, by defining multiple Dropout branches To enhance sample robustness, multi-label text classification results are combined with the output of a non-linearly activated multilayer perceptron. :
[0175]
[0176] Here, U represents 4 Dropout branches.
[0177] S40, obtain a loss function based on the debiased weighted contrastive loss and the binary cross-entropy loss function, and constrain the multi-label text classification result with the goal of minimizing the function value of the loss function. The debiased weighted contrastive loss is constructed based on the label similarity matrix, text similarity, dynamic negative sample weights and debiased denominator.
[0178] In this embodiment, as another core improvement, a dynamically weighted debiased weighted contrast loss is introduced to assign higher weights to negative samples, reduce interference from false negative samples, and improve the problem of semantic overlap of labels.
[0179] Specifically, the expression for the bias-free weighted contrastive loss is:
[0180]
[0181] In the formula, This indicates the bias-weighted comparison loss. For the label similarity matrix, For dynamic negative sample weights, For text similarity, To remove the partial denominator;
[0182] in,
[0183] In the formula, For any two sample labels in the training dataset and Cosine similarity between them Let be the sum of the cosine similarities of the sample labels in the i-th row.
[0184] It should be noted that, for samples within a batch, the model outputs the logistic probability. The label vector (probabilistic form) used in the comparison item is: Additionally, let the text be encoded (for measuring similarity between samples) as a vector. (The hidden vector output of the upstream encoder text, which has been Euclidean norm normalized), then construct the biased weighted contrastive loss according to the following steps. :
[0185] (1) Tag similarity matrix C:
[0186] The formula for calculating the cosine similarity (or dot product followed by normalization) between the label probability vectors of any two samples within a batch is shown below:
[0187]
[0188] Where T represents the transpose matrix.
[0189] Then, the label similarity sum for each row (excluding diagonal terms) is calculated, as shown in the formula below:
[0190]
[0191] Preferably, a row-normalized label similarity weight matrix is constructed. The formula is shown below:
[0192]
[0193] in This represents the weight after normalizing the label similarity, used for soft label guidance in comparison alignment.
[0194] (2) Text similarity (distance to similarity conversion):
[0195] First, the obtained text is hidden encoded. First, perform Euclidean transformation on the vectors. .
[0196] Then calculate the Euclidean distance matrix between samples within the batch. .
[0197] Finally, the distance is converted into a similarity measure:
[0198]
[0199] in This represents the text similarity score, and 𝜏 represents the temperature coefficient, with 𝜏=10.0.
[0200] (3) Calculation of dynamic negative sample weights:
[0201] The similarity sum for each row (for each sample i) excluding its own diagonal items is calculated using the following formula:
[0202]
[0203] Define the negative sample weight matrix This weighting allows "samples that are closer in the text space" to have a greater weight in the loss when they are negative samples, thus focusing on distinguishing difficult / nearest neighbor negative samples.
[0204] (4) Partial correction term (partial denominator):
[0205] To correct the influence of false negative samples caused by overlapping labels or incomplete labeling, a bias reduction parameter λ∈[0,1] (preferably 0.5 in this embodiment) is introduced, and the bias reduction denominator is calculated as follows:
[0206]
[0207]
[0208] in Indicates the batch size.
[0209] The denominator interpolates between the global average similarity and the total intra-row similarity, thereby reducing the problem of excessive negative sample penalty caused by partial label overlap.
[0210] On the other hand, the binary cross-entropy loss function uses BCEWithLogitsLoss, which stably merges the Sigmoid and binary cross-entropy values, as the main supervision term to ensure that the probability of the model output is aligned with the probability of the true label, thereby optimizing the model output. The specific expression for the binary cross-entropy loss function is as follows:
[0211]
[0212] in, This represents the Sigmoid function. This represents the binary cross-entropy loss.
[0213] Based on the two loss functions mentioned above, optionally, the comparative loss can be obtained by averaging the bias-free weighted comparative loss for each batch i, as shown in the following formula:
[0214]
[0215] In the formula, B represents the total number of training iterations;
[0216] Weighted loss function , with the loss function The multi-label text classification results are constrained by minimizing the function value as the objective:
[0217]
[0218] In the technical solution provided in this embodiment, the data in the training dataset is preprocessed to obtain the input tensor representation; the text semantic representation is enhanced by fusing multi-layer hidden semantic representations of deep models; then, an improved label graph convolutional network is constructed, introducing regularization, layer normalization, residual connections, and label-aware attention pooling into the graph neural network to achieve fine-grained representation of the relationship between labels and strengthen the "label-text" interaction; finally, a debiased weighted contrastive loss is introduced to construct a dynamically weighted debiased contrastive learning, which assigns higher weights to negative samples, reduces the interference of false negative samples, and thus improves the label semantic overlap problem.
[0219] Furthermore, as an implementation scheme, this application also provides a multi-label text classification method based on semantic representation enhancement and dynamic weighted debiasing contrastive learning as described above, and its application in multi-label text classification.
[0220] Specifically, a multi-label text classification model (referred to as Our in this embodiment) is constructed based on the method proposed in the first embodiment. An experiment with automatic machine evaluation was conducted to verify its performance from the perspectives of precision and recall. Therefore, in this embodiment, Precision@1 (P@1), Precision@3 (P@3), Precision@5 (P@5), NDCG@3 (N@3), and NDCG@5 (N@5) evaluation metrics are used to verify the performance of the multi-label text classification method, yielding the following results: Figure 3 The results show the automatic evaluation comparison between the model and the baseline.
[0221] exist Figure 3 The table lists the machine evaluation results of our proposed model Our and several classic multi-label text classification methods (all models in the table are reduced forms of multi-label text classification): First, through experimental verification, our proposed model Our demonstrates that on the public dataset AAPD, the multi-label classification performance of the method proposed in this application outperforms existing multi-label text classification methods on all machine evaluation metrics, where * indicates experimental results obtained using the original code of this invention; Second, the multi-label text classification model Our proposed in this embodiment has better experimental performance than the current state-of-the-art methods.
[0222] As one implementation scheme, Figure 4 This is a schematic diagram of the hardware operating environment of the computer system involved in the embodiments of this application.
[0223] like Figure 4 As shown, the computer system may include: a processor 1001, such as a CPU; a memory 1005; a user interface 1003; a network interface 1004; and a communication bus 1002. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.
[0224] Those skilled in the art will understand that Figure 4 The computer system architecture shown does not constitute a limitation on the computer system and may include more or fewer components than shown, or combine certain components, or have different component arrangements.
[0225] like Figure 4 As shown, the memory 1005, as a storage medium, may include an operating system, a network communication module, a user interface module, and computer programs. The operating system is a program that manages and controls the hardware and software resources of the computer system, as well as the operation of the computer programs and other software or programs.
[0226] exist Figure 4 In the computer system shown, the user interface 1003 is mainly used to connect to the terminal and communicate with the terminal; the network interface 1004 is mainly used to communicate with the backend server; and the processor 1001 can be used to call the computer program stored in the memory 1005.
[0227] In this embodiment, the computer system includes: a memory 1005, a processor 1001, and a computer program stored in the memory and executable on the processor, wherein:
[0228] When processor 1001 calls a computer program stored in memory 1005, it performs the following operations:
[0229] S10, Obtain the text sequence and label sequence obtained after performing text segmentation and label encoding on the training dataset, and construct the input tensor representation based on the text sequence and the label sequence;
[0230] S20, the input tensor representation is input into the encoder of the preset model to obtain the outputs of multiple hidden layers of the encoder and then weighted and fused to obtain an enhanced text semantic representation;
[0231] S30, obtain a preset label co-occurrence matrix, transform the preset label co-occurrence matrix into a label adjacency matrix and input it into a graph neural network to obtain a label semantic graph, and process the label semantic graph through residual and layer normalization to obtain label embeddings, map the label embeddings into an adjacency matrix, perform label-aware attention pooling on the enhanced text semantic representation based on the adjacency matrix to obtain a text-label fine-grained alignment vector, and process the text-label fine-grained alignment vector into a multi-label text classification result;
[0232] S40, obtain a loss function based on the debiased weighted contrastive loss and the binary cross-entropy loss function, and constrain the multi-label text classification result with the goal of minimizing the function value of the loss function. The debiased weighted contrastive loss is constructed based on the label similarity matrix, text similarity, dynamic negative sample weights and debiased denominator.
[0233] Furthermore, those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program includes program instructions and can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in a computer system to implement the process steps of the embodiments of the above methods.
[0234] Therefore, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the various steps of the multi-label text classification method based on semantic representation enhancement and dynamic weighted debiasing contrastive learning as described in the above embodiments.
[0235] The computer-readable storage medium can be any computer-readable storage medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.
[0236] It should be noted that, since the storage medium provided in the embodiments of this application is the storage medium used to implement the methods of the embodiments of this application, those skilled in the art can understand the specific structure and variations of the storage medium based on the methods described in the embodiments of this application, and therefore will not be repeated here. All storage media used in the methods of the embodiments of this application fall within the scope of protection of this application.
[0237] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0238] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0239] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0240] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0241] It should be noted that any reference signs placed between parentheses in the claims should not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. This application can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.
[0242] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.
[0243] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.
Claims
1. A multi-label text classification method based on semantic representation enhancement and dynamic weighted debiasing contrastive learning, characterized in that, The method includes the following steps: S10, Obtain the text sequence and label sequence obtained after performing text segmentation and label encoding on the training dataset, and construct the input tensor representation based on the text sequence and the label sequence; S20, the input tensor representation is input into the encoder of the preset model to obtain the outputs of multiple hidden layers of the encoder and then weighted and fused to obtain an enhanced text semantic representation; S30, obtain a preset label co-occurrence matrix, transform the preset label co-occurrence matrix into a label adjacency matrix and input it into a graph neural network to obtain a label semantic graph, and process the label semantic graph through residual and layer normalization to obtain label embeddings, map the label embeddings into an adjacency matrix, perform label-aware attention pooling on the enhanced text semantic representation based on the adjacency matrix to obtain a text-label fine-grained alignment vector, and process the text-label fine-grained alignment vector into a multi-label text classification result; S40, obtain a loss function based on the debiased weighted contrastive loss and the binary cross-entropy loss function, and constrain the multi-label text classification result with the goal of minimizing the function value of the loss function. The debiased weighted contrastive loss is constructed based on the label similarity matrix, text similarity, dynamic negative sample weights and debiased denominator. The expression for the debiased weighted contrast loss is: ； In the formula, This indicates the bias-weighted comparison loss. For the label similarity matrix, For dynamic negative sample weights, For text similarity, To remove the partial denominator; in, ; In the formula, For any two sample labels in the training dataset and Cosine similarity between them Let be the sum of the cosine similarities of the sample labels in the i-th row; In step S30, the step of performing label-aware attention pooling on the enhanced text semantic representation based on the adjacency matrix to obtain a text-label fine-grained alignment vector includes: S37, adjacency matrix The input is fed into a two-layer graph neural network for propagation, resulting in two-layer propagated label embeddings. : ； In the formula, Embedding for tags, and These represent the first and second layer neural networks, respectively. S38, according to the double-layer propagation tag embedding and the enhanced text semantic representation Determine the document-to-tag attention score matrix. : ； In the formula, The learnable temperature coefficient is used to adjust the sharpness of the attention distribution; dim represents the dimension of the label embedding. S39, the attention score matrix After normalization, the attention weight matrix is obtained. The attention weight matrix Dot product enhances text semantic representation This yields a fine-grained text-tag alignment vector. : ； in, .
2. The multi-label text classification method based on semantic representation enhancement and dynamic weighted debiasing contrastive learning as described in claim 1, characterized in that, In step S30, the expression for the label adjacency matrix includes: ； in: ； ； In the formula, For the label adjacency matrix, To enhance the adjacency matrix, To enhance the adjacency matrix The degree matrix, This is the normalized label co-occurrence matrix. It is a C-order identity matrix, where C represents the order. in: ； ； In the formula, The number of times the label p appears. For smoothing terms, Represents the label co-occurrence matrix. For any pair of labels, Indicates an indicator function, This indicates that the i-th sample contains label p. This indicates that the i-th sample contains the label q, and N is the total number of samples. This represents the logical AND operation, used to indicate the condition that two labels appear simultaneously in the same sample, if and only if the i-th sample simultaneously satisfies the condition. and When the value is 1, the indicator function takes the value 1; otherwise, it takes the value 0.
3. The multi-label text classification method based on semantic representation enhancement and dynamic weighted debiasing contrastive learning as described in claim 2, characterized in that, In step S30, the preset label co-occurrence matrix is input into a graph neural network to obtain a label semantic graph, and the label semantic graph is processed by residual and layer normalization to obtain label embedding, including the following steps; S31, embed the label matrix in the input tensor representation. Adjacency matrix with labels Perform single-layer graph convolution processing: ； In the formula, For the label adjacency matrix, For trainable weight matrix, Activation function S32 achieves hierarchical diffusion of semantic information between tags through two stacked graph convolutions to capture the semantic connections contained in tag co-occurrence relationships: ； ； In the formula, For trainable weight matrix, This is a temporary fallback function. This indicates the updated label information for the first layer. This represents the updated label information at the second layer, i.e., the label semantic graph; S33 introduces residual and layer normalization mechanisms to construct label embeddings. : ； In the formula, is the activation function for residuals and layer normalization mechanisms.
4. The multi-label text classification method based on semantic representation enhancement and dynamic weighted debiasing contrastive learning as described in claim 3, characterized in that, In step S30, embedding the label into an adjacency matrix specifically includes: S34, embed the tag Perform a linear mapping: ； In the formula, The weight matrix is trainable. S35, based on linear mapping Calculate the sample-level dynamic adjacency matrix : ； In the formula, For the Sigmoid function; S36, the sample-level dynamic adjacency matrix Degree normalization is performed to obtain the adjacency matrix. : ； 。 5. The multi-label text classification method based on semantic representation enhancement and dynamic weighted debiasing contrastive learning as described in claim 1, characterized in that, In step S30, the text-label fine-grained alignment vector is processed into a multi-label text classification result, specifically including: The text-label fine-grained alignment vectors are processed by max pooling and multi-branch Dropout, and then a non-linear activation function is used to obtain the multi-label text classification results.
6. The multi-label text classification method based on semantic representation enhancement and dynamic weighted debiasing contrastive learning as described in claim 1, characterized in that, In step S10, constructing the input tensor representation based on the text sequence and the label sequence includes: S11, the text sequence With the tag sequence By splicing, a joint sequence is obtained. ; S12, determine the joint sequence Is the total length of the sequence less than a preset length threshold? S13, if so, select the target sequence in descending order of length, and delete the last word of the target sequence word by word until the total length of the sequence is less than the preset length threshold. S14, if not, in the text sequence and the tag sequence Insert a boundary information delimiter [SEP] between the text sequence and the label sequence to distinguish them, and in the text sequence Insert the global representation symbol [CLS] into the header to obtain the input tensor representation. : 。 7. The application of a multi-label text classification method based on semantic representation enhancement and dynamic weighted debiased contrastive learning as described in any one of claims 1 to 6 in multi-label text classification.
8. A computer system, characterized in that, The computer system includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps of the multi-label text classification method based on semantic representation enhancement and dynamic weighted debiasing contrastive learning as described in any one of claims 1 to 6.