A Multi-View Text Classification Method Based on Graph Neural Networks

Abstract

A multi-view text classification method based on graph neural networks includes: preprocessing the input text; constructing a text graph based on a fixed-size sliding window; converting the text into a low-dimensional vector representation using a GloVe word embedding model; inputting the initial text vector matrix into a gated graph neural network (GGNN) to model and learn semantic relationships and word dependencies in the text data; inputting the output of the GGNN into BiLSTM and TextCNN respectively to enhance the text feature representation; aggregating word node representations using a node-level attention mechanism; fusing multi-view text feature representations, including graph structure features, BiLSTM sequence features, and N-gram local features; inputting the initial text vector matrix into a long-range feature extractor to expand the receptive field of word nodes; using KL divergence to measure the probability distribution between multi-view fused features and long-range node features; and jointly calculating the model loss using the cross-entropy loss function and the KL divergence regularization term.

Description

Technical Field

[0001] This invention relates to a multi-view short text classification method based on graph neural networks, which can be applied to multiple fields such as text sentiment analysis, spam filtering, and document classification. Background Technology

[0002] In today's information-explosive era, the rapid growth of text data has brought enormous challenges to information processing and analysis. Text classification, as an important task in text data processing, is used to divide text into different predefined categories. Existing text classification methods are mainly based on traditional feature extraction and machine learning algorithms, but these methods have certain limitations when dealing with large-scale and complex text data.

[0003] In recent years, deep learning methods have achieved significant breakthroughs in text classification. By constructing multi-layered neural network models, deep learning can automatically learn and represent complex features in text and extract useful information, thereby improving the performance of text classification. Among them, Convolutional Neural Networks (CNNs), through convolution and pooling operations, can effectively capture local features in text, thus learning more abstract and high-level semantic features. Recurrent Neural Networks (RNNs) can model contextual information in text, capturing dependencies between words, thus better understanding the semantics of the text. However, besides CNNs and RNNs, pre-trained models have also been widely used in text classification tasks, such as BERT (Bidirectional Encoder Representations from Transformers) and GPT (Generative Pre-trained Transformer).

[0004] However, CNNs, with their localized perception, limit their ability to understand global semantics. RNNs are prone to vanishing or exploding gradients when processing long sequences, making it difficult to capture long-distance dependencies. While fine-tuning pre-trained models can achieve good results, these models typically have a large number of parameters, high memory consumption, and are expensive. In recent years, Graph Neural Networks (GNNs) have attracted widespread attention due to their ability to handle complex data structures and their powerful expressive capabilities. These are deep learning models based on graph structures that can effectively process complex text data. Compared to traditional methods, GNNs can better capture relationships and dependencies in text data, thereby improving the accuracy and performance of text classification.

[0005] In summary, graph neural network-based text classification methods are better able to handle large-scale, complex text data and achieve better classification results. They can fully utilize the dependencies within the text data, thus providing more accurate classification results. In recent years, many graph neural network-based text classification methods have been proposed, broadly categorized into two types:

[0006] 1) Corpus-based graph construction method: This method constructs a large-scale graph based on the entire corpus. It can aggregate global word co-occurrence information and effectively extract global structural information of the text. In this method, text samples in the corpus are represented as nodes of the graph, and the relationships between nodes can be represented as edges. This method can utilize the global relationships between texts in the entire corpus, thereby capturing the features of the text data more comprehensively.

[0007] 2) Text-level graph construction method: This method constructs an independent graph structure for each text sample and classifies each graph. In this method, each text sample is represented as a graph, where words or phrases in the text serve as nodes, and the relationships between them are represented by edges. This method focuses more on the features of each text sample itself and can capture the local relationships of the text data more precisely.

[0008] Corpus-based graph construction methods use the text from the entire corpus to build a graph. Therefore, the graphs constructed using this method contain many nodes and edges, which typically requires high memory consumption. Furthermore, the model's structure and parameters depend on the corpus, making online testing impossible.

[0009] Text-level graph construction methods build a graph for each individual text, effectively reducing the number of nodes and edges while eliminating dependencies between the input text and the entire corpus. Therefore, how to construct text-level graphs has become an increasingly popular research topic.

[0010] However, text classification methods using graph neural networks still have the following problems:

[0011] (1) Existing graph-based models only consider the graph structure information of text. Specifically, these methods no longer treat text as a sequence of words, but as a group of words that co-occur. They focus more on the co-occurrence relationship between words and lose the sequential semantic relationship.

[0012] (2) In graph neural networks, information propagation of nodes is usually limited to their direct neighbors. This means that nodes can only obtain information from their local neighborhood and cannot obtain information from nodes further away. This locality limitation may lead to information loss or failure to fully utilize important information in the global graph structure.

[0013] For text classification tasks, word order and long-distance dependencies can help models better understand the contextual semantic information of the text. Therefore, how to address the loss of semantic relationships and the limited receptive field in graph neural networks is a pressing problem that needs to be solved in today's big data era. Summary of the Invention

[0014] To address the problems of existing methods that overemphasize graph structural information while neglecting contextual semantic information and having limited receptive fields, this invention proposes a multi-view text classification method based on graph neural networks.

[0015] This invention designs a multi-view feature extractor that fully utilizes the text's order, structure, and local N-gram information. Furthermore, a remote node feature extractor is designed to bridge the gap between words and remote nodes, where interaction is difficult. Finally, KL divergence is incorporated as a regularization term into the cross-entropy loss to jointly optimize the model. Compared to existing methods, this approach achieves more advanced performance.

[0016] To solve the above problems, the technical solution provided by the present invention is as follows:

[0017] A multi-view text classification method based on graph neural networks includes the following steps:

[0018] Step 1: Preprocess the input text, mainly by tokenization and removal of stop words.

[0019] Step 2: Construct a text graph based on a fixed-size sliding window, where each node represents a word and edges represent the semantic relationships between them.

[0020] Step 3: Use the GloVe word embedding model to convert the text into a low-dimensional vector representation that preserves the semantic and syntactic relationships between words.

[0021] Step 4: Input the initial text vector matrix into the Gated Graph Neural Network (GGNN) to model and learn the semantic relationships and word dependencies in the text data.

[0022] Step 5: To further enhance the representation of text features, the output of GGNN is input into BiLSTM and TextCNN respectively to enhance the sequential context features and N-gram local feature representation of the text.

[0023] Step 6: Considering that different nodes have different importance to the text, a node-level attention mechanism is used to aggregate the word node representations.

[0024] Step 7: Input the graph structure features, BiLSTM sequence features, and N-grams local features into the multi-view feature extractor to fuse the multi-view text feature representation.

[0025] Step 8: Input the initial text vector matrix into the long-range feature extractor, use the self-attention mechanism to calculate and replace the adjacency matrix of the graph, and adaptively adjust the receptive field of the word nodes.

[0026] Step 9: Use KL divergence to measure the probability distribution between multi-view fused features and distant node features.

[0027] Step 10: Calculate the model loss using the cross-entropy loss function and the KL divergence regularization term, and jointly optimize the model.

[0028] This invention proposes a multi-view feature extractor (MPE) designed to fully utilize the order, structure, and local N-gram information of text to effectively enhance the model's feature representation capabilities. Simultaneously, a node-level attention mechanism is employed to aggregate global feature information from the entire graph structure, generating a graph-level representation of the text. To bridge the gap between words and distant nodes, a remote node feature extractor (RFE) is also designed. Finally, KL divergence is incorporated as a regularization term into the cross-entropy loss function to jointly optimize the model. This invention relates to probabilistic models, graph neural networks, deep learning, and specifically applies to deep learning-based natural language processing.

[0029] This invention enhances the model's feature representation capabilities by fusing multi-source feature representations through a multi-view feature extractor. Furthermore, with the help of a long-distance node feature extractor, it bridges the gap between words and distant words that are difficult to interact with. Finally, KL divergence is added as a regularization term to the loss function to optimize the entire model. Compared to previous methods, this invention offers improvements in both accuracy and efficiency.

[0030] The advantages of this invention are:

[0031] 1. This invention proposes a graph-based text classification model based on multi-view feature fusion. It simultaneously utilizes sequence information, structural information, and local N-grams information to enrich the semantic features of the text, effectively enhancing the model's feature representation capability.

[0032] 2. This invention designs a remote node feature extractor that can adaptively extract useful information from non-contiguous nodes, bridging the gap between words and remote nodes where interaction is difficult. Attached Figure Description

[0033] Figure 1 This is a schematic diagram of the overall process of the present invention. Detailed Implementation

[0034] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0035] A multi-view text classification method based on graph neural networks includes the following steps:

[0036] Step 1: Preprocess the movie review text, mainly by tokenization and removal of stop words.

[0037] Step 2: Since corpus-level graph-based methods suffer from high memory consumption and lack of online testing support, a text-level graph construction method was chosen. In a text-level graph, each unique word is described as a vertex, and the co-occurrence relationship between words is represented as an edge. If two words appear simultaneously in a sliding window of size k, an undirected edge is added between them. The weight of an edge is determined by the number of times the two words appear together in the sliding window.

[0038] Formally, the text graph is represented as G = (V, E), where V = {v1, v2, ..., v}. n Let E = {e1, e2, ..., e} represent the set of distinct words. n} represents the set of edges formed between words.

[0039] Step 3: Use the GloVe word embedding model to convert the text into a low-dimensional vector representation that preserves the semantic and syntactic relationships between words.

[0040] Step 4: Input the initial text vector matrix into a Gated Graph Neural Network (GGNN) to model and learn semantic relationships and word dependencies in the text data. Nodes can receive information from neighboring nodes and then merge it with their representations from the previous time step to update the hidden state of each node. Since a single-layer GGNN can only learn feature representations from first-order neighbor nodes, we stack multiple GGNNs to learn higher-order neighbor information. The specific propagation process is as follows:

[0041]

[0042] z t =σ(W z a t +U z h t-1 +b z (2)

[0043] r t =σ(W r a t +U r h t-1 +b r (3)

[0044]

[0045]

[0046] in, Let H represent the normalized symmetric adjacency matrix of the graph, D represent the degree matrix, and ⊙ represent the element-wise dot product. w, U, and b are trainable weights and biases. The feature matrix obtained after convolution of the gated graph is represented as H. g .

[0047] Step 5: To enhance the representation of text features, the output of GGNN is input into BiLSTM and TextCNN respectively to enhance the sequential context features and N-gram local feature representation of the text;

[0048] For a text sequence H that has undergone gated graph convolution... g Each word can be viewed as a time step, therefore the feature matrix H obtained by graph convolution is... g It can be directly input into BiLSTM and TextCNN to enhance the feature representation of words.

[0049] H l =BiLSTM(H g (6)

[0050] H c =TextCNN(H g (7)

[0051] Step 6: Considering the varying importance of different nodes to the text, a node-level attention mechanism is used to aggregate the word node representations. Note that this aggregation is only performed on the feature representation H after graph convolution. g and the feature representation H after sequence enhancement l Perform node-level attention aggregation. This is because the feature matrix H g and H l While TextCNN retains the feature information of each word, it loses the feature representation of each word after performing N-gram convolutions. Therefore, focusing on the node level is meaningless.

[0052] Unlike previous methods, this invention incorporates centrality adjustment into the node attention mechanism. Centrality is a commonly used concept in social networks, representing the importance of a node within the network. This invention evaluates the importance of nodes in a sentence based on the centrality of word nodes. For node v, its degree centrality is denoted as... The higher the centrality of a node, the more important it is to the entire sentence. The specific formula is as follows:

[0053] αv =d v ⊙σ(f1(h v (8)

[0054] h v =α v ⊙tanh(f2(h v (9)

[0055]

[0056] f1 and f2 are two multilayer perceptrons, and Maxpooling represents max pooling.

[0057] Step 7: Input the graph structure features, BiLSTM sequence features, and N-grams local features into the multi-view feature extractor and fuse the multi-view text feature representation;

[0058] To enhance the model's feature representation capabilities, textual semantic features from multiple perspectives are fused, resulting in more comprehensive feature information and improved classification performance. During feature fusion, each perspective considers the importance of other perspectives to itself. Specifically, the semantic features of each perspective are treated as a new node, and the semantic features of the other two perspectives are considered as its neighboring nodes. The semantic feature matrix is ​​defined as follows: Adjacency matrix A m The definition is as follows:

[0059]

[0060]

[0061] W is a learnable weight matrix. This indicates a splicing operation. ij Let σ represent the importance of node j to node i. To ensure that the weights of other view nodes are no greater than the weight of the current node, let σ be the sigmoid activation function. Finally, a simple matrix transformation is needed to obtain the output of the multi-view feature extractor.

[0062]

[0063] Step 8: Input the initial text vector matrix into the long-range feature extractor, use the self-attention mechanism to calculate and replace the adjacency matrix of the graph, and adaptively adjust the receptive field of the word nodes;

[0064] Graph neural network-based methods only consider the direct neighbor information of each node. In this case, obtaining information from distant nodes can only be achieved by increasing the number of layers in the GNN, which often leads to oversmoothing. To bridge the gap between words and distant nodes, a distant node feature extractor (RFE) is proposed to enable interaction with distant nodes.

[0065] First, the initial text vector matrix is ​​input into BiLSTM to obtain the long-range semantic features H. r Then, a dot product self-attention mechanism is used to learn the importance of different words in the text.

[0066]

[0067] For A r Elements less than 0 are considered negatively correlated; therefore, A r All elements less than 0 in H are set to 0. Finally, H... r and A r Inputting text features into a gated neural network (GGNN) and employing node-level attention to aggregate text features yields long-range semantic features.

[0068] Step 9: Use KL divergence to measure the probability distribution between multi-view fused features and distant node features;

[0069] We introduce KL divergence as a regularization term in the loss function to jointly optimize the model. The greater the difference between the two probability distributions, the larger the calculated value of the KL divergence. First, we fuse features from multiple perspectives. and remote node characteristics A nonlinear transformation is performed. The calculation process is as follows:

[0070]

[0071]

[0072] Step 10: Calculate the model loss using the cross-entropy loss function and the KL divergence regularization term, and jointly optimize the model.

[0073]

[0074] This invention primarily addresses the problems in natural language processing where graph neural networks overemphasize graph structure information, neglecting the semantic information of text sequences and exhibiting limited receptive fields for individual words. It proposes a graph-text classification model based on a multi-view feature extractor (MPE) and a long-range feature extractor (RFE). This model enhances its feature representation capabilities by fusing multi-source feature representations through the MPE. Furthermore, the long-range node feature extractor bridges the gap between words and distant words that lack interaction. Finally, KL divergence is introduced as a regularization term into the loss function to optimize the entire model.

[0075] The present invention has been described through the above examples; however, it is clear that these examples are for illustrative purposes only and are not intended to limit the invention to the scope of those examples. Those skilled in the art can make non-substantial changes in form and content based on the above steps without departing from the scope of the invention. Therefore, the present invention is not limited to the specific forms and details described above.

Claims

1. A multi-view text classification method based on graph neural networks, comprising the following steps: Step 1: Preprocess the input text; Step 2: Construct a text graph based on a fixed-size sliding window, where each node represents a word and edges represent the semantic relationships between them; Step 3: Use the GloVe word embedding model to convert the text into a low-dimensional vector representation, which preserves the semantic and syntactic relationships between words; Step 4: Input the initial text vector matrix into the Gated Graph Neural Network (GGNN) to model and learn the semantic relationships and word dependencies in the text data; Step 5: To enhance the representation of text features, the output of GGNN is input into BiLSTM and TextCNN respectively to enhance the sequence context features and N-gram local feature representation of the text; Step 6: Considering that different nodes have different importance to the text, a node-level attention mechanism is used to aggregate the word node representations; Step 7: Input the graph structure features, BiLSTM sequence features, and N-grams local features into the multi-view feature extractor and fuse the multi-view text feature representation; Step 8: Input the initial text vector matrix into the long-range feature extractor, use the self-attention mechanism to calculate and replace the adjacency matrix of the graph, and adaptively adjust the receptive field of the word nodes; Step 9: Use KL divergence to measure the probability distribution between multi-view fused features and distant node features; Step 10: Calculate the model loss using the cross-entropy loss function and the KL divergence regularization term, and jointly optimize the model.

2. The multi-view text classification method based on graph neural networks as described in claim 1, characterized in that: Step 2 specifically includes: if two words appear simultaneously in a sliding window of size k, then add an undirected edge between them; the weight of the undirected edge is determined by the number of times the two words co-occur in the sliding window; Formally, the text graph is represented as G = (V, E), where V = {v1, v2, ..., v}. n Let E = {e1, e2, ..., e} represent the set of distinct words. n } represents the set of edges formed between words.

3. The multi-view text classification method based on graph neural networks as described in claim 1, characterized in that: Step 4 specifically includes: a node can receive information from neighboring nodes and then merge it with their representation from the previous time step to update the hidden state of each node; the specific propagation process is as follows: z t =σ(W z a t +U z h t-1 +v z ) (2) r t =σ(W r a t +U r h t-1 +b r ) (3) in, Let H represent the normalized symmetric adjacency matrix of the graph, D represent the degree matrix, and ⊙ represent element-wise dot product; W, U, and b are trainable weights and biases, and the feature matrix obtained after convolution of the gated graph is represented as H. g .

4. The multi-view text classification method based on graph neural networks as described in claim 1, characterized in that: Step 5 specifically includes: H l =BiLSTM(H g ) (6) H c =TextCNN(H g ) (7) 5. The multi-view text classification method based on graph neural networks as described in claim 1, characterized in that: Step 6 specifically includes: only processing the feature representation H after graph convolution. g and the feature representation H after sequence enhancement l Perform node-level attention aggregation; further extract semantic features of the text by calculating the degree centrality and node attention weights of nodes; for a word node v, its degree centrality is denoted as... The higher the degree centrality of a word node, the more important it is to the entire sentence; the specific formula is as follows: α v =d v ⊙σ(f1(h v )) (8) h v =α v ⊙tanh(f2(h v )) (9) f1 and f2 are two multilayer perceptrons, and Maxpooling represents max pooling.

6. The multi-view text classification method based on graph neural networks as described in claim 1, characterized in that: Step 7 specifically includes: In the feature fusion process, the semantic features of each perspective are treated as a new node, and the semantic features of the other two perspectives are treated as its neighboring nodes; the semantic feature matrix is ​​defined as H. m =H g ⊕H l ⊕H c Adjacency matrix A m The definition is as follows: W is a learnable weight matrix, ⊕ denotes a concatenation operation; e ij Let σ represent the importance of node j to node i. To ensure that the weights of other view nodes are no greater than the weight of the current node, let σ be the sigmoid activation function. Finally, a simple matrix transformation is needed to obtain the output of the multi-view feature extractor.

7. The multi-view text classification method based on graph neural networks as described in claim 1, characterized in that: Step 8 specifically includes: First, the initial text vector matrix is ​​input into BiLSTM to obtain the long-range semantic features H. r Then, the dot product self-attention mechanism is used to learn the importance of different words in the text; For A r Elements less than 0 are considered negatively correlated; therefore, A r All elements less than 0 in H are set to 0; finally, H is... r and A r Inputting text features into a gated neural network (GGNN) and employing node-level attention to aggregate text features yields long-range semantic features.

8. The multi-view text classification method based on graph neural networks as described in claim 1, characterized in that: Step 9 specifically includes: the greater the difference between the two probability distributions, the larger the value of the calculated KL divergence; and the fusion of features from multiple perspectives. and remote node characteristics The nonlinear transformation is performed, and the calculation process is as follows:

9. The multi-view text classification method based on graph neural networks as described in claim 1, characterized in that: The model loss mentioned in step 10 is:

10. The multi-view text classification method based on graph neural networks as described in claim 1, characterized in that: Step 4 describes a GGNN consisting of a stack of multiple GGNN networks, with each layer used to learn information about higher-order neighbor nodes.

Images