Text classification method based on bidirectional hierarchical tree network

Abstract

The application discloses a text classification method based on a bidirectional hierarchical tree network, comprising the following steps: obtaining a text classification dataset with labels; performing graph processing on the obtained dataset, and constructing a corresponding dependency analysis graph for each text; constructing an encoding tree corresponding to the text with labels by using the dependency analysis graph constructed for each text; learning and processing by using the constructed encoding tree through a bidirectional hierarchical tree network to establish a text classification model; and performing text classification processing on unmarked text data to be classified by using the established text classification model; the method encodes semantic information of the text by using an encoding tree with structural entropy minimization, and updates hidden features of all nodes by using a bidirectional hierarchical tree network; and the method has the advantages of improved precision, enhanced practicability, simple model and reduced calculation complexity.

Description

Technical Field

[0001] This invention belongs to the field of data processing technology, specifically relating to a text classification method based on a bidirectional hierarchical tree network. Background Technology

[0002] Text classification is a popular problem in Natural Language Processing (NLP) and is widely used in various scenarios, including spam detection, question answering, user intent classification, and sentiment classification. Given a piece of text, text classification analysis can divide the text into different predefined categories based on the content features of the text, thereby achieving automatic classification and labeling of the text.

[0003] Commonly used text classification methods mainly include traditional machine learning methods and deep learning methods. In recent years, deep learning methods have surpassed traditional methods in text classification, primarily due to their powerful nonlinear modeling capabilities, which better capture complex relationships within the data. The core idea of ​​deep learning methods in text classification is to use multi-layer neural network models to learn abstract representations of text and then map these abstract representations to classification categories. By training on large amounts of labeled text data, deep learning models can learn semantic features in the text, thus enabling them to classify texts of unknown categories. Compared to traditional machine learning models, deep learning models can better handle complex semantics and contextual information in text, thereby improving classification accuracy and generalization ability.

[0004] When performing text classification, words and sentences in the text are usually converted into vectors or sequences. However, this conversion process can lead to a loss of semantic structural information to some extent. To better represent information, a widely used approach is to reconstruct the elements in the input text into nodes and edges, thereby constructing a semantic relationship or dependency graph, which enables more effective pattern analysis and reasoning.

[0005] Currently, most advanced text classification methods are based on deep learning models of graph neural networks. These methods often have some challenges, including a large number of model parameters, high computational complexity, and poor performance in practical applications.

[0006] In summary, most current text classification methods suffer from low accuracy, limited practicality, complex models, and high computational complexity. Summary of the Invention

[0007] The purpose of this invention is to provide a text classification method based on a bidirectional hierarchical tree network that improves accuracy, enhances practicality, simplifies the model, and reduces computational complexity.

[0008] The text classification method based on a bidirectional hierarchical tree network provided by this invention includes the following steps:

[0009] S1. Obtain a labeled text classification dataset;

[0010] S2. Perform graphing on the dataset obtained in step S1, and construct a corresponding dependency analysis graph for each text;

[0011] S3. Using the dependency analysis graph constructed for each text in step S2, construct the coding tree corresponding to the labeled text;

[0012] S4. Using the encoding tree constructed in step S3, a text classification model is established through learning and processing via a bidirectional hierarchical tree network;

[0013] S5. Using the text classification model established in step S4, complete the text classification process for the unlabeled text data to be classified;

[0014] Step S2 describes the graphing process performed on the dataset obtained in step S1, constructing a corresponding dependency parsing graph for each text. Specifically, this includes:

[0015] Using D = {w1, ..., w i ,…,w l Let} represent a document D containing l words, where w i This represents the i-th word in document D;

[0016] Analyze the dependency relationships between words based on the sentence's grammatical structure;

[0017] A dependency analysis graph is constructed using words as vertices and the dependency relationships between core words and dependent words as edges.

[0018] Define G = (V, E) as a dependency parsing graph; where V represents the set of vertices of the graph; and E represents the set of edges of the graph.

[0019] The vertex set V and edge set E of a graph are defined using the following formulas:

[0020] V={w i |i∈[1,l]}

[0021] E={e ij |r ij ∈DP}

[0022] Among them, w i Indicates the i-th word; e ij This represents the edge weight, which is fixed at 1 in an unweighted graph; r ij The word w i And the word w jThe dependency relationships between them; DP represents the result of dependency analysis, which is expressed by the following formula:

[0023] DP={r ij |i≠j;i,j≤l}

[0024] Step S3, which involves constructing a coding tree corresponding to the labeled text using the dependency parsing graph built for each text in step S2, specifically includes:

[0025] A greedy algorithm that minimizes structural entropy is used to construct a k-level high encoding tree for the dependency parsing graph, where k represents the input parameters;

[0026] (3-1) Initialize any encoding tree with only a root node, and assign all vertices in the dependency parsing graph as child nodes of the encoding tree;

[0027] (3-2) The iteration continues until a full-height coding tree is obtained by iteratively merging the sibling nodes that generate the largest reduction in structural entropy. The iteration ends then.

[0028] (3-3) Compress the subtrees of nodes whose height exceeds k, so that the height of the coding tree is less than or equal to k;

[0029] (3-4) Select leaf nodes with a depth not equal to k to fill the parent node, so that all leaf nodes are at the k level;

[0030] Based on the greedy algorithm that minimizes structural entropy, the encoding tree ET = (V) of height k is obtained. T E T ); where V T This represents the set of nodes in the coding tree. Represents the set of leaf nodes. This indicates that there is a one-to-one correspondence between all vertices in the graph and the tree nodes in the encoded leaf; E represents the set of nodes at level i; T Represents the edge set of the coding tree;

[0031] Step S4, which uses the encoding tree constructed in step S3, employs a bidirectional hierarchical tree network for learning and processing to establish a text classification model, specifically includes:

[0032] (4-1) Update node features in the forward hierarchy:

[0033] Updating node features in the forward hierarchy specifically includes aggregating features from the leaf node layer to the root node layer of the coding tree. The features of non-leaf nodes are obtained by aggregating and updating the features of their corresponding child nodes.

[0034] The following formula represents the features of the i-th layer node in the forward hierarchical update:

[0035]

[0036] in, represents the i-th layer node in the encoding tree; MLP(·) represents a trainable multilayer perceptron; Pool(·) represents the pooling operation, which can be selected as an invariant function of any permutation, including summation and mean functions; Let u represent the characteristics of node u at level i-1; u represents the child node of node v; C(v) represents the set of child nodes of node v;

[0037] (4-2) Update node features at later levels:

[0038] The subsequent layer updates the node features, specifically including distributing the features of each node from the root node layer of the encoding tree to the corresponding child nodes;

[0039] The following formula represents the feature of the j-th layer node in the forward hierarchical update:

[0040]

[0041] Where Concat(·) represents the vector concatenation operation; p represents the parent node of node v; Represent the characteristics of node p at level j+1;

[0042] (4-3) Establish a text classification model:

[0043] After the bidirectional hierarchical update processing based on steps (4-1) and (4-2), the vector representation of the encoding tree corresponding to the text is obtained through pooling of all leaf nodes. The specific description is expressed by the following formula:

[0044]

[0045] in, This represents the updated leaf node features;

[0046] The predicted label of the text is calculated using the softmax function, and the formula is as follows:

[0047]

[0048] Where W represents the weight parameter; b represents the bias term;

[0049] Using the true label y and the predicted label The cross-entropy loss is used to optimize the model parameters; the formula for calculating the cross-entropy loss is shown below:

[0050]

[0051] Among them, y iThis represents the true label of the i-th sample; This represents the predicted label of the i-th sample;

[0052] Step S5, using the text classification model established in step S4, performs text classification processing on unlabeled text data to be classified, specifically including:

[0053] Unlabeled text data to be classified is obtained, and steps S2 and S3 are used to process the obtained data;

[0054] Select the initial word vectors of all words in the processed text to be classified, and the corresponding encoding tree. Then, obtain the predicted labels through the text classification model established in step S4 to complete the text classification.

[0055] The text classification method based on a bidirectional hierarchical tree network provided by this invention encodes the semantic information of the text through a coding tree that minimizes structural entropy, and updates the hidden features of all nodes using a bidirectional hierarchical tree network. This invention improves accuracy, enhances practicality, simplifies the model, and reduces computational complexity. Attached Figure Description

[0056] Figure 1 This is a schematic diagram of the method flow of the present invention.

[0057] Figure 2 This is a schematic diagram of the method model of the present invention.

[0058] Figure 3 This diagram illustrates the test accuracy (%) of the method of this invention when initialized using GloVe word vectors across four datasets.

[0059] Figure 4 This diagram illustrates the test accuracy (%) of the method of the present invention when fine-tuning the pre-trained model BERT on four datasets. Detailed Implementation

[0060] like Figure 1 The diagram shown is a flowchart of the method of the present invention: The text classification method based on a bidirectional hierarchical tree network provided by the present invention includes the following steps:

[0061] S1. Obtain a labeled text classification dataset;

[0062] S2. Perform graphing on the dataset obtained in step S1, constructing a corresponding dependency parsing graph for each text; specifically including:

[0063] Using D = {w1, ..., w i ,…,w l Let} represent a document D containing l words, where w iThis represents the i-th word in document D;

[0064] Analyze the dependency relationships between words based on the sentence's grammatical structure;

[0065] A dependency analysis graph is constructed using words as vertices and the dependency relationships between core words and dependent words as edges.

[0066] Define G = (V, E) as a dependency parsing graph; where V represents the set of vertices of the graph; and E represents the set of edges of the graph.

[0067] The vertex set V and edge set E of a graph are defined using the following formulas:

[0068] V={w i |i∈[1,l]}

[0069] E={e ij |r ij ∈DP}

[0070] Among them, w i Indicates the i-th word; e ij This represents the edge weight, which is fixed at 1 in an unweighted graph; r ij The word w i And the word w j The dependency relationships between them; DP represents the result of dependency analysis, which is expressed by the following formula:

[0071] DP={r ij |i≠j;i,j≤l}

[0072] S3. Using the dependency parsing graph constructed for each text in step S2, construct the encoding tree corresponding to the labeled text; specifically including:

[0073] A greedy algorithm that minimizes structural entropy is used to construct a k-level high encoding tree for the dependency parsing graph, where k represents the input parameters;

[0074] (3-1) Initialize any encoding tree with only a root node, and assign all vertices in the dependency parsing graph as child nodes of the encoding tree;

[0075] (3-2) The iteration continues until a full-height coding tree is obtained by iteratively merging the sibling nodes that generate the largest reduction in structural entropy. The iteration ends then.

[0076] (3-3) Compress the subtrees of nodes whose height exceeds k, so that the height of the coding tree is less than or equal to k;

[0077] (3-4) Select leaf nodes with a depth not equal to k to fill the parent node, so that all leaf nodes are at the k level;

[0078] Based on the greedy algorithm that minimizes structural entropy, the encoding tree ET = (V) of height k is obtained. T E T ); where V T This represents the set of nodes in the coding tree. Represents the set of leaf nodes. This indicates that there is a one-to-one correspondence between all vertices in the graph and the tree nodes in the encoded leaf; E represents the set of nodes at level i; T Represents the edge set of the coding tree;

[0079] S4. Using the encoding tree constructed in step S3, a text classification model is established through learning and processing via a bidirectional hierarchical tree network; specifically including:

[0080] (4-1) Update node features in the forward hierarchy:

[0081] Updating node features in the forward hierarchy specifically includes aggregating features from the leaf node layer to the root node layer of the coding tree. The features of non-leaf nodes are obtained by aggregating and updating the features of their corresponding child nodes.

[0082] The following formula represents the features of the i-th layer node in the forward hierarchical update:

[0083]

[0084] in, represents the i-th layer node in the encoding tree; MLP(·) represents a trainable multilayer perceptron; Pool(·) represents the pooling operation, which can be selected as an invariant function of any permutation, including summation and mean functions; Let u represent the characteristics of node u at level i-1; u represents the child node of node v; C(v) represents the set of child nodes of node v;

[0085] (4-2) Update node features at later levels:

[0086] The subsequent layer updates the node features, specifically including distributing the features of each node from the root node layer of the encoding tree to the corresponding child nodes;

[0087] The following formula represents the feature of the j-th layer node in the forward hierarchical update:

[0088]

[0089] Where Concat(·) represents the vector concatenation operation; p represents the parent node of node v; Represent the characteristics of node p at level j+1;

[0090] (4-3) Establish a text classification model:

[0091] like Figure 2 The diagram shows the method model of the present invention: After the bidirectional hierarchical update processing based on steps (4-1) and (4-2), the vector representation of the encoding tree corresponding to the text is obtained through pooling of all leaf nodes. The specific description is expressed by the following formula:

[0092]

[0093] in, This represents the updated leaf node features;

[0094] The predicted label of the text is calculated using the softmax function, and the formula is as follows:

[0095]

[0096] Where W represents the weight parameter; b represents the bias term;

[0097] Using the true label y and the predicted label The cross-entropy loss is used to optimize the model parameters; the formula for calculating the cross-entropy loss is shown below:

[0098]

[0099] Among them, y i This represents the true label of the i-th sample; This represents the predicted label of the i-th sample;

[0100] Using the true label y and the predicted label The cross-entropy loss is used to optimize the model parameters; the formula for calculating the cross-entropy loss is shown below:

[0101]

[0102] Among them, y i This represents the true label of the i-th sample; This represents the predicted label of the i-th sample;

[0103] The computational complexity of the bidirectional hierarchical tree model proposed in this invention is O(n), where n represents the number of nodes;

[0104] S5. Using the text classification model established in step S4, perform text classification processing on the unlabeled text data to be classified; specifically including:

[0105] Unlabeled text data to be classified is obtained, and steps S2 and S3 are used to process the obtained data;

[0106] Select the initial word vectors of all words in the processed text to be classified, and the corresponding encoding tree. Then, obtain the predicted labels through the text classification model established in step S4 to complete the text classification.

[0107] In the method of this invention, four datasets are selected, including R8, R52, Ohsumed, and MR. R8 and R52 are two subsets of the Reuters News 21578 dataset. R8 includes 8 categories, 5485 training texts, and 2189 test texts; R52 includes 52 categories, 6532 training texts, and 2568 test texts. Ohsumed is the dataset obtained after removing multi-label data from the MEDLINE dataset. All texts are medical summaries, including 23 disease types, and are divided into 3357 training texts and 4043 test texts. MR is a movie review dataset, including positive and negative reviews, with 10% randomly selected as the test text set. The coding tree heights k in the R8, R52, Ohsumed, and MR datasets are set to 4, 3, 11, and 2, respectively. Figure 3 The diagram shows the test accuracy (%) of the method of this invention when initialized using GloVe word vectors on four datasets: Figure 3 This demonstrates that the method of the present invention achieves optimal performance on the R8 and R52 datasets, and produces competitive results on the remaining datasets; for example Figure 4 The diagram shows the test accuracy (%) of the method of this invention when fine-tuning the pre-trained model BERT on four datasets: Figure 4 This indicates that the method has a certain degree of versatility and can work in conjunction with large pre-trained models.

Claims

1. A text classification method based on a bidirectional hierarchical tree network, comprising the following steps: S1. Obtain a labeled text classification dataset; S2. Perform graphing on the dataset obtained in step S1, and construct a corresponding dependency analysis graph for each text; S3. Using the dependency analysis graph constructed for each text in step S2, construct the coding tree corresponding to the labeled text; S4. Using the encoding tree constructed in step S3, a text classification model is established through learning and processing via a bidirectional hierarchical tree network; S5. Using the text classification model established in step S4, complete the text classification process for the unlabeled text data to be classified.

2. The text classification method based on a bidirectional hierarchical tree network according to claim 1, characterized in that... Step S2 describes the graphing process performed on the dataset obtained in step S1, constructing a corresponding dependency parsing graph for each text. Specifically, this includes: Using D = {w1, ..., w i ,…,w l Let} represent a document D containing l words, where w i This represents the i-th word in document D; Analyze the dependency relationships between words based on the sentence's grammatical structure; A dependency analysis graph is constructed using words as vertices and the dependency relationships between core words and dependent words as edges. Define G = (V, E) as a dependency parsing graph; where V represents the set of vertices of the graph; and E represents the set of edges of the graph. The vertex set V and edge set E of a graph are defined using the following formulas: V={w i |i∈[1,l]} And={and ij |r ij ∈DP} Among them, w i Indicates the i-th word; e ij This represents the edge weight, which is fixed at 1 in an unweighted graph; r ij The word w i And the word w j The dependency relationships between them; DP represents the result of dependency analysis, which is expressed by the following formula: DP={r ij |i≠j;i,j≤l}。 3. The text classification method based on a bidirectional hierarchical tree network according to claim 2, characterized in that... Step S3, which involves constructing a coding tree corresponding to the labeled text using the dependency parsing graph built for each text in step S2, specifically includes: A greedy algorithm that minimizes structural entropy is used to construct a k-level high encoding tree for the dependency parsing graph, where k represents the input parameters; (3-1) Initialize any encoding tree with only a root node, and assign all vertices in the dependency parsing graph as child nodes of the encoding tree; (3-2) The iteration ends when the sibling node with the largest structural entropy is generated by iterative merging until the full-height coding tree is obtained. (3-3) Compress the subtrees of nodes whose height exceeds k, so that the height of the coding tree is less than or equal to k; (3-4) Select leaf nodes with a depth not equal to k to fill the parent node, so that all leaf nodes are at the k level; Based on the greedy algorithm that minimizes structural entropy, the encoding tree ET = (V) of height k is obtained. T E T ); where V T This represents the set of nodes in the coding tree. Represents the set of leaf nodes. This indicates that there is a one-to-one correspondence between all vertices in the graph and the tree nodes in the encoded leaf; E represents the set of nodes at level i; T This represents the edge set of the coding tree.

4. The text classification method based on a bidirectional hierarchical tree network according to claim 3, characterized in that... Step S4, which uses the encoding tree constructed in step S3, employs a bidirectional hierarchical tree network for learning and processing to establish a text classification model, specifically includes: (4-1) Update node features in the forward hierarchy: Updating node features in the forward hierarchy specifically includes aggregating features from the leaf node layer to the root node layer of the coding tree. The features of non-leaf nodes are obtained by aggregating and updating the features of their corresponding child nodes. (4-2) Update node features at later levels: The subsequent layer updates the node features, specifically including distributing the features of each node from the root node layer of the encoding tree to the corresponding child nodes; (4-3) Establish a text classification model: After the bidirectional hierarchical update processing based on steps (4-1) and (4-2), the vector representation of the encoding tree corresponding to the text is obtained through pooling of all leaf nodes. The specific description is expressed by the following formula: in, This represents the updated leaf node features; The predicted label of the text is calculated using the softmax function, and the formula is as follows: Where W represents the weight parameter; b represents the bias term; Using the true label y and the predicted label The cross-entropy loss is used to optimize the model parameters; the formula for calculating the cross-entropy loss is shown below: Among them, y i This represents the true label of the i-th sample; This represents the predicted label of the i-th sample.

5. The text classification method based on a bidirectional hierarchical tree network according to claim 4, characterized in that... The bidirectional hierarchical tree network learning specifically includes: (4-1) Update node features in the forward hierarchy: The following formula represents the features of the i-th layer node in the forward hierarchical update: in, represents the i-th layer node in the encoding tree; MLP(·) represents a trainable multilayer perceptron; Pool(·) represents the pooling operation, which can be selected as an invariant function of any permutation, including summation and mean functions; Let u represent the characteristics of node u at level i-1; u represents the child node of node v; C(v) represents the set of child nodes of node v; (4-2) Update node features at later levels: The following formula represents the feature of the j-th layer node in the forward hierarchical update: Where Concat(·) represents the vector concatenation operation; p represents the parent node of node v; This represents the characteristics of node p at level j+1.

6. The text classification method based on a bidirectional hierarchical tree network according to claim 5, characterized in that... Step S5, using the text classification model established in step S4, performs text classification processing on unlabeled text data to be classified, specifically including: Unlabeled text data to be classified is obtained, and steps S2 and S3 are used to process the obtained data; Select the initial word vectors of all words in the processed text to be classified, and the corresponding encoding tree. Then, obtain the predicted labels through the text classification model established in step S4 to complete the text classification.

Images