A text classification method based on graph convolution network fusion attention mechanism

Abstract

A text classification method based on graph convolution network fusion attention mechanism comprises the following steps: step 1, collecting sample text data, and preprocessing the collected sample text data; step 2, vectorizing the text data after preprocessing; step 3, constructing a text graph containing word nodes and document nodes; step 4, according to the importance of the characteristics, the vectorized text and the characteristics of the text graph are given weights through the attention mechanism; step 5, the text after multi-head self-attention calculation is input into the graph convolution network model, and the relationship between each text and the label category is obtained; step 6, according to the relationship between the text and the label category, the predicted classification result of the text is obtained. Through the above steps, the present application provides a text classification method with high efficiency, accuracy and good classification effect.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, specifically to a text classification method based on graph convolutional networks with an attention mechanism. Background Technology

[0002] Text classification is a fundamental task in the field of natural language processing. It involves associating a piece of text with one or more classification labels based on the characteristics of the text content within a given classification system.

[0003] In recent years, with the development of mobile internet technology, massive amounts of text data have emerged on various online platforms and software. To improve information retrieval and text mining efficiency by classifying this information-rich text data into different domains, the importance of text classification has increased significantly. Consequently, various deep neural network-based methods have been proposed and applied to solve text classification problems.

[0004] Existing text classification methods can be broadly categorized into rule-based feature matching, traditional machine learning, and deep learning-based methods. These methods classify text through processes such as defining a classification system, data preprocessing, data feature extraction, text representation, and classification.

[0005] Problems with existing text classification methods: Although existing methods have made significant progress in text classification research, they cannot obtain global information and sentence grammatical structure when representing text, which limits the effectiveness of text classification. Summary of the Invention

[0006] To address the shortcomings of existing text classification models, this invention provides a text classification method based on graph convolutional networks with an attention mechanism. This method can comprehensively associate the text data to be classified, forming a large text graph, effectively and accurately extracting features from different documents, thereby achieving text classification. Furthermore, this invention introduces an attention mechanism, improving the quality of feature extraction and the accuracy of classification during the process.

[0007] To address the aforementioned technical problems, this invention provides a text classification method based on a graph convolutional network with fused attention mechanism. This method includes the following steps:

[0008] Step 1: Collect sample text data and preprocess the collected sample text data;

[0009] Step 2 involves vectorizing the preprocessed text data.

[0010] Step 3: Construct a text graph containing word nodes and document nodes;

[0011] Step 4: Based on the importance of the features, assign weights to the vectorized text and text graph features using an attention mechanism;

[0012] Step 5 will use the text input graph convolutional network model after multi-head self-attention calculation to obtain the relationship between each text and the label category;

[0013] Step 6: Based on the relationship between the text and the tag category, obtain the predicted classification result of the text.

[0014] The preprocessing method in step 1 is as follows:

[0015] The frequency of words in the dataset is statistically analyzed. For each sample, words with a frequency greater than 5 that are not in the stop word list are concatenated into a new sample. The words appearing in the new sample are then counted and stored in a word dictionary. The number of times each word appears in the dataset, the document ID containing the word, the number of documents containing the word, and the word's ID in the word list are calculated. Finally, the label categories appearing in the entire dataset are recorded to obtain a label category set. The stop word list refers to characters or words automatically filtered out before or after processing natural language data or text. These characters or words are divided into two categories: function words and lexical words.

[0016] The vectorization representation method in step 2 is as follows:

[0017] First, we count the words and documents in the preprocessed text data, calculate the word and document embeddings, and obtain the feature vectors of the documents, words, and documents. Then, we use one-hot encoding to vectorize all documents, words, and documents corresponding to each label category to the same dimension, and obtain the vectors of the document's corresponding label and the vectors of the document and word's corresponding label.

[0018] The construction method in step 3 is as follows:

[0019] Using words and documents from the preprocessed text data as nodes, a large text graph is constructed. During graph construction, connections are only made between documents and words, and between words themselves; document-to-document connections are not performed. When a document is connected to a word, the Term Frequency-Inverse Document Rate (TF-IDF) is used to calculate the weight between a document node and a word node. When words are connected, the PMI (Programme for Meaning and Meaning) is used to calculate the weight between two word nodes, and edges are added between words with positive PMI values. Based on the above, all related nodes in the text data are connected to form a large undirected graph, and the weights between nodes are calculated. Finally, the adjacency matrix A and degree matrix of the obtained undirected graph are calculated. The Laplacian matrix of the undirected graph is obtained, and the Laplacian matrix of the graph is regularized to obtain matrix Z.

[0020] The specific method in step 4 is as follows:

[0021] Multi-head self-attention computation is performed on vectorized text. Since it is a self-attention mechanism, the input matrices Q, K, and V correspond to the three important components of the attention mechanism, query, key, and value, respectively. Attention computation is performed on the regularized Laplacian matrix Z.

[0022] In a multi-head attention mechanism, the input features are mapped to different information subspaces through different weight matrices. Then, the same attention calculation is performed in each subspace to fully extract the latent semantics and structure of the text. Finally, the heads are merged to obtain the output of the multi-head self-attention mechanism. Therefore, Z after multi-head self-attention calculation is...

[0023] The specific method in step 5 is as follows:

[0024] After multi-head self-attention computation, the vector is fed into a graph convolutional network for further feature extraction, using the matrix after multi-head self-attention computation. The matrix X containing all nodes and their features, and the weight matrix W0 that constitutes the edges in the text graph, are used as the inputs to the first layer of the graph convolutional network. ReLU is used as the activation function in the first layer of the graph convolutional network to turn the negative weight values ​​between nodes into 0, reduce the amount of computation, and obtain the output of the first layer of the graph convolutional network.

[0025] Finally, the output of the first layer of the graph convolutional network is used as the activation matrix and fed into the second layer of the graph convolutional network. Similar to the first layer, it also uses matrices. The weight matrix W1 of the edges is used, and the activation function of the second-layer graph convolutional network is set to softmax. The resulting vectors are normalized to obtain standardized weight vectors and the relationship between each text and label category, thereby realizing text classification prediction.

[0026] The specific method in step 6 is as follows:

[0027] Forward propagation yields the prediction results and the loss function. The cross-entropy function is used as the loss function, which is defined as the cross-entropy error of all labeled documents: Where y D It is a set of document indexes with labels, F is the dimension of the output feature, which is equal to the number of classes, and Y is the label indicator matrix; parameter updates are performed using backpropagation with cross-entropy, and the smaller the value of cross-entropy, the more accurate the prediction result.

[0028] The beneficial effects of this invention are:

[0029] 1. This invention uses graph convolutional networks to classify text, which can capture the grammatical structure information and global information of sentences during text representation, thereby improving the text representation ability.

[0030] 2. This invention introduces a multi-head self-attention mechanism into traditional graph convolutional networks to enhance the network's ability to extract key information, thereby improving the accuracy of text classification. Attached Figure Description

[0031] Figure 1 This is a flowchart of the present invention.

[0032] Figure 2 This is a schematic diagram of the structure of a text classification method.

[0033] Figure 3 This diagram compares the present invention with other methods.

[0034] Figure 4 X is the matrix obtained after vectorizing the document set 20NG.

[0035] Figure 5 This is a large text graph Z containing word nodes and document nodes, constructed from the document set 20NG. Detailed Implementation

[0036] To better understand the above-described objectives, features, and advantages of the present invention, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0037] The following section will detail each step of the above technical solution:

[0038] Step 1: Text Data Preprocessing. For the data in the text dataset, the text data is first cleaned, and then a text dictionary and a label set are constructed.

[0039] This invention was tested on three widely used benchmark corpora: the news document set 20NG, containing 18,846 documents in 20 categories; the medical literature database Ohsumed, containing 7,440 documents in 23 categories; and the movie review dataset MR, containing 10,662 documents in 2 categories.

[0040] Step 1-1: Clean the text data. Count the word frequency of all words in the text dataset. For each text, concatenate words with a frequency greater than 5 that are not in the stop word list into a new sample.

[0041] Steps 1-2 construct the text dictionary. Count the words that appear in the new samples and store them in the word dictionary. Assign an ID to each word and document, and calculate the number of times each word appears in the dataset, the document ID containing the word, the number of documents containing the word, and the word's ID in the word list.

[0042] Steps 1-3 construct the label set. Record the types of labels appearing in the text dataset to obtain the label category set.

[0043] Step 2 involves vectorizing the preprocessed text data. This includes statistically analyzing the words and documents within the preprocessed text data, calculating their embeddings, and obtaining the feature matrix of the document and the feature matrices X of the words and documents. Matrix X is shown below. Figure 4 As shown. Then, one-hot encoding is used to vectorize all documents corresponding to each tag category, as well as all words and documents, to the same dimension, resulting in a matrix of documents corresponding to tags and a matrix of documents and words corresponding to tags.

[0044] Step 3 constructs a text graph containing word nodes and document nodes. Words and documents from the preprocessed text data are used as nodes to construct a large text graph. In this invention, when constructing the text graph, connections are only made between documents and words, and between words themselves; connections are not made between documents. The text graph is as follows: Figure 5 As shown.

[0045] Step 3-1: When a document is associated with a word, the weight between a document node and a word node is calculated using TF-IDF (Term Frequency-Inverse Document Rate), defined as follows:

[0046] TF-IDF = TF × IDF;

[0047] Where TF stands for Term Frequency and IDF stands for Inverse Document Frequency. Term Frequency (TF) represents the frequency of a given word in a document, and its calculation formula is as follows:

[0048]

[0049] Inverse Document Frequency (IDF) is a measure of the general importance of a word. It can be obtained by dividing the total number of documents by the number of documents containing that word. The formula is as follows:

[0050]

[0051] Step 3-2 constructs a text graph containing word nodes and document nodes. Words and documents from the preprocessed text data are used as nodes to construct a large text graph. In this invention, when constructing the text graph, connections are only made between documents and words, and between words themselves; connections are not made between documents. When words are connected, the weight between the two word nodes is calculated using PMI:

[0052]

[0053]

[0054]

[0055] Where #W(i) is the number of sliding windows containing word i in the text data, #W(i,j) is the number of sliding windows containing both words i and j, and #W is the total number of sliding windows in the text data. A positive PMI value indicates that the two word nodes in the text data have a high semantic correlation; a negative PMI value indicates that the two word nodes in the text data have almost no semantic correlation. Therefore, in this invention, edges are only added between two words with positive PMI values. Based on the above, all related nodes in the text data are connected, forming a weighted undirected graph.

[0056] Step 3-3 calculates the Laplacian matrix of the undirected graph. Based on the established undirected graph, calculate the adjacency matrix A and the degree matrix. The Laplacian matrix can be obtained by the difference between the degree matrix and the adjacency matrix. In this invention, the Laplacian matrix of an undirected graph is represented using a regularized method, and its calculation formula is as follows:

[0057]

[0058] Step 4 assigns weights to the vectorized text and text graph features based on their importance using an attention mechanism. Multi-head self-attention is performed on the vectorized text. This mechanism captures semantic and syntactic features and can easily and effectively abstract dependencies within the context. In this invention, the three important matrices in the attention mechanism are Q, K, and V, and the weight matrix for these three matrices is initialized to W. q W k W v Perform dot product with the regularized Laplacian matrix respectively:

[0059] Q = Z * W q ;

[0060] K = Z * W k ;

[0061] V = Z * Wp ;

[0062] Then, self-attention is calculated using the following formula:

[0063]

[0064] In multi-head attention mechanisms, the input features are passed through different weight matrices W. i This is mapped to different information subspaces, and then the same attention calculation is performed in each subspace to fully extract the latent semantics and structure in the text. The calculation formula for the attention of the i-th head is as follows:

[0065]

[0066] Finally, the calculation results of the attention from each head are combined to obtain the output of the multi-head self-attention mechanism.

[0067] Step 5 involves using a graph convolutional network model to process the text input after multi-head self-attention computation, and then obtaining the relationship between each text and label category. To further understand the relationship between text and labels, a graph convolutional network model is used to interact with features and extract the implicit relationships between text and labels.

[0068] In this invention, a matrix calculated using multi-head self-attention is used. If a matrix X containing all nodes and their features, and a weight matrix W0 constituting the edges in the text graph are used as inputs to the first layer of the graph convolutional network, then the first layer input is:

[0069]

[0070] In the first layer of the graph convolutional network, ReLU is used as the activation function to change the negative weights between nodes to 0, reducing computational cost. The output of the first layer of the graph convolutional network is then obtained. Then, the output of the first layer graph convolutional network is used. As the activation matrix, it enters the second layer of the graph convolutional network. Similar to the first layer, it also uses matrices. Given the edge weight matrix W1, the input to the second-layer graph convolutional network is:

[0071]

[0072] The activation function of the second-layer graph convolutional network is set to softmax. Then the output of the second-layer graph convolutional network is:

[0073]

[0074] Finally, the obtained vectors are normalized to obtain standardized weight vectors and the relationship between each text and the label category, thereby achieving text classification prediction.

[0075] The definitions of the classification evaluation indicators in step 5 are as follows:

[0076] Precision is defined as:

[0077]

[0078] Step 6: Based on the relationship between the text and the tag category, obtain the predicted classification result of the text. Forward propagation yields the prediction result and the loss function. In this invention, cross-entropy error is used as the loss function.

[0079]

[0080] Among them, y D It is a set of labeled document indexes, F is the dimension of the output features, equal to the number of classes, and Y is the label indicator matrix. Parameter updates are performed using backpropagation with cross-entropy; the smaller the cross-entropy value, the more accurate the prediction.

[0081] A schematic diagram of the text classification method of the present invention is shown below. Figure 1 As shown in the diagram, the structure of the text classification method is illustrated below. Figure 2 As shown in the figure, this invention provides a text classification method based on graph convolutional networks with an attention mechanism, comprising: collecting sample text data and preprocessing the collected sample text data; vectorizing the preprocessed text data; constructing a text graph containing word nodes and document nodes; assigning weights to features according to their importance through an attention mechanism; inputting the text after attention calculation into a graph convolutional network model to obtain the relationship between each text and the label category; and obtaining the predicted classification result of the text based on the relationship between the text and the label category.

[0082] Figure 3 To demonstrate the performance of this invention on three datasets—20NG, Ohsumed, and MR—compared to two other methods, LSTM and Fast Text, the method of this invention is named AText GCN. Precision is a metric for text classification accuracy; higher accuracy indicates more precise classification. Under this metric, compared to the other two methods, this invention achieves the best performance on the 20NG, Ohsumed, and MR datasets.

[0083] In summary, this invention utilizes PMI and TF-IDF to extract weights between word nodes and between document nodes and word nodes, respectively, taking into account the weight information between different nodes; it employs the concept of multi-head self-attention to capture important words in the text, making the classification effect more efficient and accurate; it uses graph convolutional networks to connect related words with documents and words with each other in the text, forming a large text graph, which can more comprehensively obtain word relationships in the document and global word relationships; the graph convolutional network can pass the label information of a document node to its neighboring word nodes, and then to other word nodes and document nodes adjacent to the word node in the first step, so that word nodes can collect comprehensive document label information and act as bridges in the graph, allowing label information to spread to the entire text graph, thereby improving the text classification effect.

[0084] It should be noted that the above description is only a specific embodiment of the present invention and is not intended to limit the present invention. The dataset used in this embodiment is limited to this embodiment. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A text classification method based on graph convolutional networks with attention fusion, characterized in that, Includes the following steps: Step 1: Collect sample text data and preprocess the collected sample text data; Step 2 involves vectorizing the preprocessed text data. Step 3: Construct a text graph containing word nodes and document nodes; The construction method in step 3 is as follows: Using words and documents from the preprocessed text data as nodes, a large text graph is constructed. When constructing the text graph, connections are only made between documents and words, and between words themselves, but not between documents. When a document is connected to a word, the weight between a document node and a word node is calculated using TF-IDF (Term Frequency-Inverse Document Rate). When words are connected, the PMI (Proportional Migration Index) is used to calculate the weight between the two word nodes, and an edge is added between two words with positive PMI values. All related nodes in the text data are connected to form a large undirected graph, and the weights between the nodes are calculated. Finally, the adjacency matrix A and degree matrix of the resulting undirected graph are calculated. The Laplacian matrix of the undirected graph is obtained, and the Laplacian matrix of the graph is regularized to obtain matrix Z; Step 4: Based on the importance of the features, assign weights to the vectorized text and text graph features using an attention mechanism; Step 5 will use the text input graph convolutional network model after multi-head self-attention calculation to obtain the relationship between each text and the label category; Step 6: Based on the relationship between the text and the tag category, obtain the predicted classification result of the text.

2. The text classification method based on graph convolutional network fusion attention mechanism according to claim 1, characterized in that, The preprocessing method in step 1 is as follows: The frequency of words in the dataset is counted. For each sample, words with a frequency greater than 5 that are not in the stop word list are concatenated into a new sample. The words appearing in the new sample are counted and stored in the word dictionary. The number of times each word appears in the dataset, the document ID containing the word, the number of documents containing the word, and the word ID in the word table are calculated. Finally, the types of tags appearing in the entire dataset are recorded to obtain the tag category set.

3. The text classification method based on graph convolutional network fusion attention mechanism according to claim 1, characterized in that, The vectorization representation method in step 2 is as follows: First, we count the words and documents in the preprocessed text data, calculate the word and document embeddings, and obtain the feature vectors of the documents, words, and documents. Then, we use one-hot encoding to vectorize all documents, words, and documents corresponding to each label category to the same dimension, and obtain the vectors of the document's corresponding label and the vectors of the document and word's corresponding label.

4. The text classification method based on graph convolutional network fusion attention mechanism according to claim 1, characterized in that, The specific method in step 4 is as follows: Multi-head self-attention computation is performed on vectorized text. Since it is a self-attention mechanism, the input matrices Q, K, and V correspond to the three important components of the attention mechanism, query, key, and value, respectively. Attention computation is performed on the regularized Laplacian matrix Z. In a multi-head attention mechanism, the input features are mapped to different information subspaces through different weight matrices. Then, the same attention calculation is performed in each subspace to fully extract the latent semantics and structure of the text. Finally, the heads are merged to obtain the output of the multi-head self-attention mechanism. Therefore, Z after multi-head self-attention calculation is...

5. The text classification method based on graph convolutional network fusion attention mechanism according to claim 1, characterized in that, The specific method in step 5 is as follows: After multi-head self-attention computation, the vector is fed into a graph convolutional network for further feature extraction, using the matrix after multi-head self-attention computation. The matrix X containing all nodes and their features, and the weight matrix W0 that constitutes the edges in the text graph, are used as the inputs to the first layer of the graph convolutional network. ReLU is used as the activation function in the first layer of the graph convolutional network to turn the negative weight values ​​between nodes into 0, reduce the amount of computation, and obtain the output of the first layer of the graph convolutional network. Finally, the output of the first layer of the graph convolutional network is used as the activation matrix and fed into the second layer of the graph convolutional network. Similar to the first layer, it also uses matrices. The weight matrix W1 of the edges is used, and the activation function of the second-layer graph convolutional network is set to softmax. The resulting vectors are normalized to obtain standardized weight vectors and the relationship between each text and label category, thereby realizing text classification prediction.

6. The text classification method based on graph convolutional network fusion attention mechanism according to claim 1, characterized in that, The specific method in step 6 is as follows: Forward propagation yields the prediction results and the loss function. The cross-entropy function is used as the loss function, which is defined as the cross-entropy error of all labeled documents: Where y D It is a set of document indexes with labels, F is the dimension of the output feature, which is equal to the number of classes, and Y is the label indicator matrix; parameter updates are performed using backpropagation with cross-entropy, and the smaller the value of cross-entropy, the more accurate the prediction result.

7. The text classification method based on graph convolutional network fusion attention mechanism according to claim 2, characterized in that, The stop word list refers to characters or words that are automatically filtered out before or after processing natural language data or text. These characters or words are divided into two categories: function words and lexical words.

Images