Hierarchical text classification method and system for fine-grained label confusion
By generating high-quality negative samples from node and sequence perspectives in hierarchical text classification, the problem of fine-grained label confusion is solved, and the model training efficiency and classification effect are improved.
Patent Information
- Application Number
- CN202411993558.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-31
- Publication Date
- 2025-04-29
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing hierarchical text classification methods are difficult to effectively solve the confusion problem between fine-grained labels, resulting in poor performance of the model when dealing with deep and fine-grained labels.
By generating high-quality negative samples from the perspectives of nodes and sequences, expanding the range of negative samples characteristics, improving the quality of negative samples, and dynamically selecting the most difficult negative samples during model training, using high-quality negative samples to give full play to the role of contrast learning.
The quality and feature range of negative samples are improved, the model training is more efficient, and the effect of hierarchical text classification is improved.
Smart Images
Figure CN119903185A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of hierarchical text classification, and more particularly to a hierarchical text classification method and system for fine-grained label confusion. Background Art
[0002] Hierarchical text classification (HTC) plays an important role in processing complex and massive text data in various fields, such as academic literature, legal documents, and online content. HTC is a complex method for classifying text into a multi-level category structure. Unlike traditional flat classification methods that only focus on single label assignment, HTC fully considers the subordinate relationship between categories and constructs a hierarchical structure from abstract to concrete. This hierarchy can adopt a simple two-level structure or a more complex and deeply nested arrangement, usually modeled as a tree or a directed acyclic graph. However, HTC suffers from poor discrimination between finer-grained labels due to unbalanced data distribution and complex dependencies between multiple levels in the hierarchy - this is common in large datasets with insufficient examples of certain categories, and the confusion between labels increases as the hierarchy goes deeper.
[0003] In recent years, researchers have introduced a number of methods to solve the problem of HTC, such as meta-learning, reinforcement learning, and graph convolutional networks. However, existing methods cannot completely solve the problem of fine-grained labels in HTC. Moreover, distinguishing deep and fine-grained labels is crucial to the performance of HTC. Fortunately, some existing studies have shown that contrastive learning performs well in enhancing fine-grained feature representations. It can effectively learn subtle feature representations between fine-grained labels, thereby improving the discrimination between fine-grained labels. However, due to the complex hierarchical structure, existing methods find it difficult to construct high-quality positive and negative samples for contrastive learning in HTC. They only use data augmentation to construct positive samples and simply construct low-quality negative samples from a text perspective, without involving the processing of similar fine-grained labels.
[0004] Therefore, how to provide a hierarchical text classification method and system for fine-grained label confusion, expand the feature range of negative samples, improve the quality of negative samples, make model training more efficient, and improve the hierarchical text classification effect is an urgent problem that technicians in this field need to solve. Summary of the invention
[0005] In view of this, the present invention provides a hierarchical text classification method and system for fine-grained label confusion, which generates high-quality negative samples from both node and sequence perspectives, expands the feature range of negative samples, and improves the quality of negative samples; dynamically selects the most difficult negative samples during the model training process to make the model training more efficient; utilizes high-quality negative samples to give full play to the role of contrastive learning and improves the effect of hierarchical text classification.
[0006] In order to achieve the above object, the present invention adopts the following technical solution: a hierarchical text classification method for fine-grained label confusion, comprising:
[0007] Get the text to be classified;
[0008] Extracting features of the text to be classified, and obtaining label features of the text after processing;
[0009] According to the label feature, a text classification probability is obtained, a node negative sample and a node positive sample are determined based on the text classification probability, and a comparative learning is performed between the node positive sample and the node negative sample to obtain a first loss function;
[0010] Using the label structure and the depth-first traversal algorithm, multiple sequence negative samples and sequence positive samples are constructed, and the contrast loss of the sequence negative samples and the sequence positive samples is calculated to obtain the second loss function;
[0011] Calculate the third loss function for multi-label classification;
[0012] Summing the first loss function, the second loss function and the third loss function to obtain a total loss function;
[0013] Train the BERT model by optimizing the total loss function;
[0014] The text to be classified is classified according to the trained BERT model to obtain the classification result of the text.
[0015] Preferably, the labels with the top n text classification probabilities in non-Groundtruth are selected as node negative samples, and the label with the lowest text classification probability in Groundtruth is selected as node positive sample.
[0016] Preferably, feature extraction is performed on the text to be classified, and label features of the text are obtained after processing, including:
[0017] The BERT model is used to extract features of the text to be classified to obtain text features; the text features are processed by a structural encoder to obtain label features of the text.
[0018] Preferably, the comparative learning between the node positive sample and the node negative sample includes:
[0019] L node =∑ i∈S max(0,p g -p i +ε);
[0020] Among them, p g Represents the probability of a positive sample of a node, p irepresents the probability of node negative samples, ε represents the difference between the classification probabilities of node positive samples and node negative samples, S represents the entire negative sample set, and L node Represents the first loss function.
[0021] Preferably, a label structure and a depth-first traversal algorithm are used to construct multiple sequence negative samples and sequence positive samples, including:
[0022] Use the depth-first traversal algorithm to traverse the entire tag structure and obtain all tag paths;
[0023] Calculate the F1 scores of all label paths and Groundtruth, and group label paths with the same F1 score into one group;
[0024] According to the text classification probability, the nodes on the label path are summed and averaged to obtain the average probability P of the entire label path;
[0025] The label path with the largest average probability P in each group is selected and constructed as a sequence negative sample, and the Ground truth is used as a sequence positive sample.
[0026] Preferably, the comparison loss of the sequence negative samples and the sequence positive samples is calculated to obtain a second loss function, including:
[0027] Compare the sequence negative samples with the sequence positive samples and calculate the loss to get the first sub-loss function.
[0028] Compare the negative samples in the sequence and calculate the contrast loss to get the second sub-loss function.
[0029] A second loss function is obtained according to the first sub-loss function and the second sub-loss function.
[0030] Preferably, calculating the third loss function for multi-label classification includes:
[0031] The third loss function is a cross entropy loss function, which is expressed as:
[0032]
[0033] Among them, y ij is the true label, L C is the cross entropy loss, is the predicted probability.
[0034] Preferably, a hierarchical text classification system for fine-grained label confusion comprises:
[0035] A text acquisition module is used to acquire the text to be classified;
[0036] A text label feature acquisition module is used to extract features from the text to be classified and obtain label features of the text after processing;
[0037] A first loss function acquisition module is used to obtain a text classification probability according to the label feature, determine a node negative sample and a node positive sample based on the text classification probability, and perform comparative learning between the node positive sample and the node negative sample to obtain a first loss function;
[0038] A second loss function acquisition module is used to construct multiple sequence negative samples and sequence positive samples by using a label structure and a depth-first traversal algorithm, calculate the contrast loss of the sequence negative samples and the sequence positive samples, and obtain a second loss function;
[0039] A third loss function acquisition module, used to calculate the third loss function of multi-label classification;
[0040] A total loss function acquisition module, used to sum the first loss function, the second loss function and the third loss function to obtain a total loss function;
[0041] The optimization module is used to train the BERT model by optimizing the total loss function;
[0042] The classification result output module is used to classify the text to be classified according to the trained BERT model to obtain the classification results of the text.
[0043] It can be seen from the above technical solution that, compared with the prior art, the present invention discloses a hierarchical text classification method and system for fine-grained label confusion, including: obtaining a text to be classified; performing feature extraction on the text to be classified, and obtaining label features of the text after processing; obtaining a text classification probability according to the label features, determining a node negative sample and a node positive sample based on the text classification probability, performing comparative learning between the node positive sample and the node negative sample, and obtaining a first loss function; constructing multiple sequence negative samples and sequence positive samples using a label structure and a depth-first traversal algorithm, calculating the comparative loss of the sequence negative samples and the sequence positive samples, and obtaining a second loss function; calculating a third loss function for multi-label classification; summing the first loss function, the second loss function, and the third loss function to obtain a total loss function; training a BERT model by optimizing the total loss function; classifying the text to be classified according to the trained BERT model to obtain a classification result of the text. The present invention has the following beneficial effects: (1) high-quality negative samples are generated from the two perspectives of nodes and sequences, which expands the feature range of negative samples and improves the quality of negative samples; (2) the present invention dynamically selects the most difficult negative samples during the model training process, making the model training more efficient; (3) the present invention utilizes high-quality negative samples to give full play to the role of contrastive learning and improve the hierarchical text classification effect. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.
[0045] Figure 1 A schematic diagram of a probability-guided contrastive learning algorithm framework provided by an embodiment of the present invention;
[0046] Figure 2 A schematic diagram of calculating the loss of comparative learning of positive and negative samples of a node provided in an embodiment of the present invention;
[0047] Figure 3 A schematic diagram of calculating the comparative learning loss for sequence positive and negative samples provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0048] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0049] The embodiment of the present invention discloses a hierarchical text classification method for fine-grained label confusion, comprising:
[0050] Get the text to be classified;
[0051] Extracting features of the text to be classified, and obtaining label features of the text after processing;
[0052] According to the label feature, a text classification probability is obtained, a node negative sample and a node positive sample are determined based on the text classification probability, and a comparative learning is performed between the node positive sample and the node negative sample to obtain a first loss function;
[0053] Using the label structure and the depth-first traversal algorithm, multiple sequence negative samples and sequence positive samples are constructed, and the contrast loss of the sequence negative samples and the sequence positive samples is calculated to obtain the second loss function;
[0054] Calculate the third loss function for multi-label classification;
[0055] Summing the first loss function, the second loss function and the third loss function to obtain a total loss function;
[0056] Train the BERT model by optimizing the total loss function;
[0057] The text to be classified is classified according to the trained BERT model to obtain the classification result of the text.
[0058] Specifically, the labels with the top n text classification probabilities in non-Ground truth are selected as node negative samples, and the label with the lowest text classification probability in Ground truth is selected as node positive sample.
[0059] Specifically, feature extraction is performed on the text to be classified, and label features of the text are obtained after processing, including:
[0060] The BERT model is used to extract features of the text to be classified to obtain text features; the text features are processed by a structural encoder to obtain label features of the text.
[0061] Specifically, the comparative learning between the node positive sample and the node negative sample includes:
[0062] L node =∑ i∈S max(0,p g -p i +ε);
[0063] Among them, p g Represents the probability of a positive sample of a node, p i represents the probability of node negative samples, ε represents the difference between the classification probabilities of node positive samples and node negative samples, S represents the entire negative sample set, and L node Represents the first loss function.
[0064] Specifically, multiple sequence negative samples and sequence positive samples are constructed using the label structure and depth-first traversal algorithm, including:
[0065] Use the depth-first traversal algorithm to traverse the entire tag structure and obtain all tag paths;
[0066] Calculate the F1 score of all label paths and Groundtruth, and exclude the label paths with an F1 score of 1. Then group the label paths with the same F1 score into one group;
[0067] According to the text classification probability, the nodes on the label path are summed and averaged to obtain the average probability P of the entire label path;
[0068] The label path with the largest average probability P in each group is selected and constructed as a sequence negative sample, and the Ground truth is used as a sequence positive sample.
[0069] Specifically, the contrast loss of the sequence negative samples and the sequence positive samples is calculated to obtain the second loss function, including:
[0070] Compare the sequence negative samples with the sequence positive samples and calculate the loss to get the first sub-loss function.
[0071] L pn =∑ j∈S max(0,P p -P j +∈);
[0072] Among them, P p Represents the probability of a positive sample, P j represents the probability of negative samples, ∈ represents the difference between the classification probabilities of sequence positive samples and sequence negative samples; L pn Represents the first sub-loss function;
[0073] Compare the negative samples in the sequence and calculate the contrast loss to get the second sub-loss function.
[0074] L nn =∑ i ∑ j>i max(0,P i -P j +∈ ij );
[0075] Among them, L nn represents the second sub-loss function;
[0076] A second loss function is obtained according to the first sub-loss function and the second sub-loss function.
[0077] L sequence =L pn +L nn .
[0078] Specifically, the third loss function for multi-label classification is calculated, including:
[0079] The third loss function is a cross entropy loss function, which is expressed as:
[0080]
[0081] Among them, y ij is the true label, L C is the cross entropy loss, is the predicted probability.
[0082] The embodiment of the present invention proposes a probability-guided contrastive learning algorithm framework. The framework is based on a tree-like hierarchy of labels (such as Figure 3As shown in the figure, suitable high-quality negative samples are dynamically constructed from both the node and sequence perspectives, thereby assisting contrastive learning to enhance fine-grained label features and separating similar fine-grained labels in the semantic space. The embodiment of the present invention assumes that the Ground truth and combinations closer to the Ground truth should have higher prediction probabilities. Specifically, the embodiment of the present invention consists of two parts: 1) Sequence sample contrastive learning. The core idea of this strategy is to make full use of the overall hierarchical structure of the label system and the depth-first traversal algorithm (DFS) to construct multiple sequence negative samples, and then dynamically select the negative sample combination with the highest prediction probability during model training. These negative samples are considered to be the most challenging to ensure the quality and diversity of negative samples. 2) Node sample contrastive learning. During the model training process, the nodes with the top n predicted probabilities among non-Ground truth nodes are selected as negative samples. These carefully selected negative samples represent the candidates that are most easily confused with positive samples in the feature space, improving the generalization performance of the model.
[0083] The embodiments of the present invention conduct comparative experiments on three widely used datasets (WOS, NYT, and RCV1-V2). As shown in Table 1, except for the NYT dataset where the Macro-F1 index is 0.33 lower than the optimal result, the other results are all optimal.
[0084] Table 1 Comparative experimental results
[0085]
[0086] In a specific embodiment of the present invention, a hierarchical text classification system for fine-grained label confusion includes:
[0087] A text acquisition module is used to acquire the text to be classified;
[0088] A text label feature acquisition module is used to extract features from the text to be classified and obtain label features of the text after processing;
[0089] A first loss function acquisition module is used to obtain a text classification probability according to the label feature, determine a node negative sample and a node positive sample based on the text classification probability, and perform comparative learning between the node positive sample and the node negative sample to obtain a first loss function;
[0090] A second loss function acquisition module is used to construct multiple sequence negative samples and sequence positive samples by using a label structure and a depth-first traversal algorithm, calculate the contrast loss of the sequence negative samples and the sequence positive samples, and obtain a second loss function;
[0091] A third loss function acquisition module, used to calculate the third loss function of multi-label classification;
[0092] A total loss function acquisition module, used to sum the first loss function, the second loss function and the third loss function to obtain a total loss function;
[0093] The optimization module is used to train the BERT model by optimizing the total loss function;
[0094] The classification result output module is used to classify the text to be classified according to the trained BERT model to obtain the classification results of the text.
[0095] In a specific embodiment of the present invention, the present invention provides a probability-guided contrastive learning algorithm framework for hierarchical text classification, such as Figure 1 As shown, specifically including:
[0096] Step S1: Use the pre-trained language model BERT to extract features from the text to be classified and obtain text features; BERT is an encoder based on a multi-layer Transformer structure, each layer of which consists of a multi-head self-attention mechanism and a feedforward neural network, supporting bidirectional contextual representation of the input sequence.
[0097] The text sequence is represented as: x = {[CLS], t1, t2, …, t n-1 ,[SEP]}; where n represents the sequence length, [CLS] and [SEP] are markers for the beginning and end of the sequence; BERT extracts deep features of the input text through a multi-layer bidirectional Transformer structure. It first processes the input text, adds special markers (such as [CLS] and [SEP]) after word segmentation, and generates an embedding vector as input. Then, these vectors pass through a multi-layer Transformer structure, using a self-attention mechanism and a feedforward neural network to extract semantic information layer by layer and generate context-related feature representations. Finally, BERT's output contains the hidden state vector of each word. The text feature extracted by BERT is represented as: H = BERT(x).
[0098] Step S2: Update the text information through the hierarchical structure-aware structure encoder, process the text features, and obtain the label features of the text. In layman's terms, the node feature (i.e., label feature) V is reshaped by the text feature H through linear changes, and the formula is expressed as: V = M·H, where M represents a trainable weight matrix.
[0099] Step S3: The node feature V obtained in the previous step represents the deductive hierarchical perceptual text feature as the input of sigmoid. The output Logits of sigmoid is used as the preliminary classification probability, that is, the text classification probability.
[0100] Step S4, such as Figure 2As shown in the figure, the Groundtruth labels in Logits are removed, and the labels with the top 3 probabilities are selected as node negative samples. At the same time, the labels with the lowest probability in Groundtruth are selected as node positive samples. Finally, the node positive and negative samples are compared for learning loss calculation. The comparison learning loss formula is: L node =∑ i∈S max(0,p g -p i +ε). Among them, p g Represents the probability of a positive sample of a node, p i represents the probability of the node's negative sample, ε represents the difference between the classification probabilities of the node's positive and negative samples, and S represents the entire set of negative samples.
[0101] Step S5, such as Figure 3 As shown. First, use the depth-first traversal algorithm to traverse the entire label structure to obtain all label paths. Then calculate the F1 score for all label paths and Groundtruth, and exclude the label path with an F1 score of 1. Then group the label paths with the same F1 score. According to the Logits obtained in step S3, sum and average the nodes on the label path to obtain the average probability P of the entire label path. Select the label path with the largest probability P in each group and construct it into a sequence negative sample set. Groundtruth is used as the sequence positive sample. Then the comparison loss is calculated between the sequence positive and negative samples, and the formula is: L pn =∑ j∈S max(0,P p -P j +∈). Among them, P p Represents the probability of a positive sample in the sequence, P j represents the probability of a negative sample in the sequence, and ∈ represents the difference between the classification probabilities of positive and negative samples in the sequence. Finally, since there is a ranking relationship between negative samples in the sequence, a contrast loss calculation is also performed between them, and the formula is: L nn =∑ i ∑ j>i max(0,P i -P j +∈ ij ). The total contrast loss function of sequence negative sample contrast learning is: L sequence =L pn +L nn .
[0102] Step S6, for multi-label classification, the cross entropy loss function is Among them, y ij is the true label, L C is the cross entropy loss, is the predicted probability.
[0103] Step S7, summing the three loss functions and training the model by optimizing the loss function.
[0104] In the inference phase, the Logits obtained in step S3 are the classification probabilities. Negative sample generation and contrastive learning are not performed.
[0105] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.
[0106] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A hierarchical text classification method for fine-grained label confusion, characterized in that: include: Get the text to be classified; Extracting features of the text to be classified, and obtaining label features of the text after processing; According to the label feature, a text classification probability is obtained, a node negative sample and a node positive sample are determined based on the text classification probability, and a comparative learning is performed between the node positive sample and the node negative sample to obtain a first loss function; Using the label structure and the depth-first traversal algorithm, multiple sequence negative samples and sequence positive samples are constructed, and the contrast loss of the sequence negative samples and the sequence positive samples is calculated to obtain the second loss function; Calculate the third loss function for multi-label classification; Summing the first loss function, the second loss function and the third loss function to obtain a total loss function; Train the BERT model by optimizing the total loss function; The text to be classified is classified according to the trained BERT model to obtain the classification result of the text.
2. A hierarchical text classification method for fine-grained label confusion according to claim 1, characterized in that: Select the top n labels with the highest text classification probability in non-Groundtruth as node negative samples, and select the label with the lowest text classification probability in Groundtruth as node positive sample.
3. A hierarchical text classification method for fine-grained label confusion according to claim 1, characterized in that: Feature extraction is performed on the text to be classified, and label features of the text are obtained after processing, including: The BERT model is used to extract features of the text to be classified to obtain text features; the text features are processed by a structural encoder to obtain label features of the text.
4. A hierarchical text classification method for fine-grained label confusion according to claim 1, characterized in that: The node positive sample and the node negative sample are subjected to comparative learning, including: L node =∑ i∈S max(0,p g -p i +e); Among them, p g Represents the probability of a positive sample of a node, p i represents the probability of node negative samples, ε represents the difference between the classification probabilities of node positive samples and node negative samples, S represents the entire negative sample set, and L node Represents the first loss function.
5. The hierarchical text classification method for fine-grained label confusion according to claim 1, characterized in that: Use the label structure and depth-first traversal algorithm to construct multiple sequence negative samples and sequence positive samples, including: Use the depth-first traversal algorithm to traverse the entire tag structure and obtain all tag paths; Calculate the F1 score for all label paths and ground truth, and group label paths with the same F1 score into one group; According to the text classification probability, the nodes on the label path are summed and averaged to obtain the average probability P of the entire label path; The label path with the largest average probability P in each group is selected and constructed as a sequence negative sample, and the Ground truth is used as a sequence positive sample.
6. A hierarchical text classification method for fine-grained label confusion according to claim 5, characterized in that: Calculate the contrast loss of the sequence negative samples and the sequence positive samples to obtain the second loss function, including: Compare the sequence negative samples with the sequence positive samples and calculate the loss to get the first sub-loss function. Compare the negative samples in the sequence and calculate the contrast loss to get the second sub-loss function. A second loss function is obtained according to the first sub-loss function and the second sub-loss function.
7. A hierarchical text classification method for fine-grained label confusion according to claim 1, characterized in that: Calculate the third loss function for multi-label classification, including: The third loss function is a cross entropy loss function, which is expressed as: Among them, y ij is the true label, L C is the cross entropy loss, is the predicted probability.
8. A hierarchical text classification system for fine-grained label confusion, characterized in that include: A text acquisition module is used to acquire the text to be classified; A text label feature acquisition module is used to extract features from the text to be classified and obtain label features of the text after processing; A first loss function acquisition module is used to obtain a text classification probability according to the label feature, determine a node negative sample and a node positive sample based on the text classification probability, and perform comparative learning between the node positive sample and the node negative sample to obtain a first loss function; A second loss function acquisition module is used to construct multiple sequence negative samples and sequence positive samples by using a label structure and a depth-first traversal algorithm, calculate the contrast loss of the sequence negative samples and the sequence positive samples, and obtain a second loss function; A third loss function acquisition module, used to calculate the third loss function of multi-label classification; A total loss function acquisition module, used to sum the first loss function, the second loss function and the third loss function to obtain a total loss function; The optimization module is used to train the BERT model by optimizing the total loss function; The classification result output module is used to classify the text to be classified according to the trained BERT model to obtain the classification results of the text.