A text classification method based on graph neural network topic model

Abstract

The present invention discloses a text classification method based on a graph neural network topic model, including: 1. preprocessing the text; 2. representing each word in the text into two word vector forms; 3. using a neural topic model (NVDM‑GSM) to obtain the topic distribution and topic vector of the text for the word vector represented by a bag of words; 4. constructing a graph structure with words in the text as nodes for the word vector represented by a Glove model, and using a gated graph neural network to obtain the adjacency information of each node of the graph; 5. using an attention mechanism between nodes; 6. using a Softmax classifier to classify the trained document graph. The present invention not only considers the characteristics of the dependence of words in the document on context, but also introduces an attention mechanism to weigh the importance of different words, and uses the topic distribution and topic vector of the document to guide the attention mechanism from a global perspective, so as to better classify the text.

Description

Technical Field

[0001] The present invention belongs to the field of text classification, and specifically is a text classification method based on a graph neural network topic model. Background Art

[0002] Text classification is the most basic task in natural language processing. From the 1960s to the 2010s, text classification models based on shallow learning dominated, such as Naive Bayes (NB), K-nearest neighbor (KNN), and support vector machine (SVM). Although these models have obvious advantages in accuracy and stability, they usually ignore the natural sequential structure or contextual information in text data, which makes it difficult to learn the semantic information of words. Since the 2010s, text classification has gradually changed from shallow learning models to deep learning models. Compared with shallow learning-based methods, deep learning can automatically provide semantically meaningful representations for text mining. Therefore, most text classification research works are based on DNN, which is a data-driven method with high computational complexity. Few studies focus on shallow learning models to address computational and data limitations.

[0003] In the past two years, graph-based deep learning methods have become a hot topic of research. For example, graph neural networks and graph convolutional neural networks are widely used in text classification. They represent words in the text as nodes, and the co-occurrence between words as edges to construct a graph structure, which is then sent to the neural network to learn the features of the nodes and finally classified. However, they only consider the local semantic information of the nodes, and do not consider the global text topic semantic information. The topic model is a statistical model that clusters the implicit semantic structure of a text set in an unsupervised learning manner. Typical representatives include the LDA topic model and various variants based on LDA. Although models such as LDA can be used to mine text topic information from a global perspective, the modeling results are often not ideal, and as the model becomes more complex, its derivation becomes increasingly difficult. Summary of the invention

[0004] The present invention aims to address the deficiencies of the above-mentioned prior art and proposes a text classification method based on a graph neural network topic model, so that when the text data is complex, the neural network topic model and the graph neural network can be used to obtain the global and local semantic information of the text, thereby improving the accuracy of text classification.

[0005] In order to achieve the above-mentioned purpose, the present invention adopts the following technical scheme:

[0006] The text classification method based on the graph neural network topic model of the present invention is characterized by being performed in the following steps:

[0007] Step 1: Get a text set D, remove all punctuation marks in the text set D, perform word segmentation, and then remove stop words to obtain a preprocessed text set D′; get a text d and its label set Y from the text set D′, and the text d contains M words, denoted as {x1,x2,...,x i ,...,x M}; where x i Represents the i-th word in text d;

[0008] Step 2: Use the bag-of-words model and the glove model to vectorize each word in the text d to obtain the text represented by the bag-of-words model. and the text represented by the glove model

[0009] Step 3: Set the number of topics in the text set D′ to K; set the vocabulary size of the vocabulary in the neural topic model NVDM-GSM to S;

[0010] Step 4: Get the text represented by the bag-of-words model The topic distribution and the word distribution of the topic, β;

[0011] Step 4.1: Get the text represented by the bag-of-words model The topic distribution

[0012] Step 4.1.1: Use the neural topic model NVDM-GSM to convert the text represented by the bag-of-words model Mapped to the mean of a Gaussian distribution and variance

[0013] Step 4.1.2: Mean of Gaussian distribution and variance Sampling is performed to obtain hidden variables

[0014] Step 4.1.3: Use formula (1) to transform the latent variable After normalization, we get the text The topic distribution

[0015]

[0016] In formula (1), W1 is a parameter matrix of K×L dimensions, and L is a latent variable The dimension of , b1 is the training parameter; softmax(·) represents the activation function; represents the topic distribution of the kth topic, and And obtained by formula (2):

[0017]

[0018] In formula (2), W 1k represents the kth topic value of the parameter matrix W1;

[0019] Step 4.2, get the word distribution β of the topic:

[0020] Step 4.2.1. Define the topic vector matrix And P=(p1,p2,...,p k ,...,p K ) T , where p k represents the kth topic vector, and T represents the transpose;

[0021] Step 4.2.2: Update the topic vector matrix P through the BP back propagation algorithm to obtain the updated topic vector matrix P′=(p′1,p′2,...,p′ k ,...,p′ K ) T , where p′ k represents the updated kth topic vector;

[0022] Step 4.2.3: Use formula (3) to get the word distribution β of the topic:

[0023] β=softmax(P′F T ) (3)

[0024] In formula (3), is the parameter matrix, T represents the transpose; β=(β1,...β k ,...β K ), where β k represents the word distribution under the kth topic, and is obtained by formula (4):

[0025] β k =softmax(p′ k F T ) (4)

[0026] Step 5: Text represented by the glove model constructing a text graph and then updating the text graph using a gated graph neural network;

[0027] Step 5.1: Text represented by the glove model All words in the text are represented as nodes of the text graph, and the co-occurrences between words are represented as edges of the text graph, so as to construct a text graph, denoted as G = (V, E), where V is the set of nodes and E is the set of edges;

[0028] Step 5.2: For any node v∈V in the text graph, use the gated graph neural network to update the feature information of the node, so that the updated node is recorded as Where t represents the number of hops of node v in the gated graph neural network;

[0029] Using formula (5) Perform nonlinear feature transformation to obtain the transformed node h v :

[0030]

[0031] In formula (5), tanh is the hyperbolic tangent function, MLP(·) is the multi-layer perceptron;

[0032] Step 6: Calculate the node h v Attention weights with the topic:

[0033] Step 6.1: Use equation (6) to get the node h v The kth topic p′ k The attention weight a v,k :

[0034] a v,k =softmax((h v ) T p′ k ) (6)

[0035] In formula (6), T represents transposition;

[0036] Step 6.2: Use equation (7) to get the node h v The weights a of the K topic attentions v :

[0037]

[0038] In formula (7), δ is the set threshold, and 0<δ<1;

[0039] Step 7: Use formula (8) to obtain the feature matrix h of the text graph G :

[0040]

[0041] In formula (8), |V| is the number of nodes in the node set V, and Maxpooling(·) is the maximum pooling function;

[0042] Step 8: Use formula (9) to predict the text represented by the glove model Tags

[0043]

[0044] In formula (9), ⊙ is the dot product, W and b are the set weight matrix and bias;

[0045] Step 9: Use formula (10) to establish the cross entropy loss function Loss:

[0046]

[0047] In formula (10), the label y∈Y, Represents the true probability value of the text graph G in the label y; Indicates the predicted probability value of the text graph G in the label y,

[0048] Step 10: Use the ADAM optimizer to minimize the loss function Loss, thereby optimizing all parameters in the text graph G and obtaining an optimal graph neural network model, and use the optimal graph neural network model to classify the text.

[0049] Compared with the prior art, the present invention has the following beneficial effects:

[0050] 1. The present invention considers the global and local semantic information of the text respectively, and uses the neural network topic model and the graph neural network to obtain the global and local semantic information of the text, thereby improving the classification effect;

[0051] 2. From a global perspective, the present invention uses the neural topic model NVDM-GSM to obtain the topic distribution of the text, and decomposes the distribution of words on the topic during the training process to obtain the topic vector. Different from the traditional probabilistic topic model (such as LDA), the neural topic model NVDM-GSM not only overcomes the complexity of long text data, but also effectively adapts to the sparsity of short text, thereby extracting more consistent topics;

[0052] 3. From a local perspective, the present invention first constructs a graph for a single text. Compared with only constructing a single graph with global relationships between documents and words, the text graph takes into account fine-grained text-word interactions, thereby extracting context-aware word relationships in each document; then, the gated graph neural network is used to update the features of each node, so that the features of the current node can reach the node t jumps away to achieve high-order feature interactions, thereby overcoming the problem that the layer can only operate on first-order neighbors;

[0053] 4. Based on the original attention mechanism, the present invention uses the topic vector and the topic distribution of the text to jointly guide the attention mechanism, so as to obtain the attention value of each node, which overcomes the original attention mechanism that only considers local node information and ignores the topic information of the text; then the features of each node are weighted averaged, and the maximum pooling function is used to make the key nodes make clearer contributions; so that the present invention can accurately classify texts even in more complex text types. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 Generate a topic distribution map of the text for the present invention;

[0055] Figure 2 It is the main model diagram of the present invention. DETAILED DESCRIPTION

[0056] In this embodiment, a text classification method based on a graph neural network topic model is performed in the following steps:

[0057] Step 1: Get a text set D, remove all punctuation marks in the text set D, perform word segmentation, and then remove useless stop words, URLs, "@" symbols, etc., to obtain a preprocessed text set D'; get a text d and its label set Y from the text set D', and the text d contains M words, denoted as {x1,x2,...,x i ,...,x M}; where x i Represents the i-th word in text d;

[0058] Step 2: Use the bag-of-words model and the glove model to vectorize each word in the text d to obtain the text represented by the bag-of-words model. and the text represented by the glove model

[0059] Step 3: Set the number of topics in the text set D′ to K; set the vocabulary size of the vocabulary in the neural topic model NVDM-GSM to S;

[0060] Step 4: Get the text represented by the bag-of-words model The topic distribution and the word distribution of the topic, β;

[0061] Step 4.1 Figure 1 To generate a topic distribution map of the text, the neural topic model NVDM-GSM is used to obtain the text represented by the bag-of-words model. The topic distribution

[0062] Step 4.1.1: Use formula (1) to convert the text represented by the bag-of-words model into Mapped to the mean of a Gaussian distribution and variance

[0063]

[0064] In formula (1), f, g1 and g2 are simplified representations of fully connected neural networks with batch normalization and drop-out;

[0065] Step 4.1.2: Use formula (2) to calculate the mean of the Gaussian distribution and variance Sampling is performed to obtain hidden variables

[0066]

[0067] In formula (2), ~ means obey, N means Gaussian distribution;

[0068] Step 4.1.3: Use formula (3) to transform the latent variable After normalization, we get the text The topic distribution

[0069]

[0070] In formula (3), W1 is a parameter matrix of K×L dimensions, and L is a latent variable The dimension of , b1 is the training parameter; softmax(·) represents the activation function; represents the topic distribution of the kth topic, and And obtained by formula (4):

[0071]

[0072] In formula (4), W 1k represents the kth topic value of the parameter matrix W1;

[0073] Step 4.2, get the word distribution β of the topic:

[0074] Step 4.2.1. Define the topic vector matrix And P=(p1,p2,...,p k ,...,p K ) T , where p k represents the kth topic vector, T represents the transpose, and L represents the dimension of the topic vector;

[0075] Step 4.2.2: Update the topic vector matrix P through the BP back propagation algorithm to obtain the updated topic vector matrix P′=(p′1,p′2,...,p′ k ,...,p′ K ) T , where p′ k represents the updated kth topic vector;

[0076] Step 4.2.3: Use formula (5) to get the word distribution β of the topic:

[0077] β=softmax(P′F T ) (5)

[0078] In formula (3), is the parameter matrix, T represents the transpose; β=(β1,...β k ,...β K ), where β k represents the word distribution under the kth topic, and is obtained by formula (6):

[0079] β k =softmax(p′ k F T ) (6)

[0080] Step 5: Text representation of the glove model Build a text graph and then use a gated graph neural network to update the text graph;

[0081] Step 5.1: Text represented by the glove model All words in the text graph are represented as nodes of the text graph, and the co-occurrences between words are represented as edges of the text graph, so as to construct a text graph, denoted as G = (V, E), where V is the set of nodes and E is the set of edges; co-occurrence is described as the relationship between words that appear in a sliding window of a certain size, and there is no direction in the graph;

[0082] Step 5.2: For any node v∈V in the text graph, use the gated graph neural network to update the feature information of the node, and then use formula (7) to get the updated node, which is recorded as Where t represents the number of hops of node v in the gated graph neural network;

[0083]

[0084] In formula (7), is the adjacency matrix, σ is the activation function, and all W, U, and b are trainable weights and biases. z and r are used as update gates and reset gates, respectively, to determine the contribution of neighbor information to the current node embedding.

[0085] Using formula (8), the node Perform nonlinear feature transformation to obtain the transformed node h v :

[0086]

[0087] In formula (8), tanh is the hyperbolic tangent function, MLP(·) is the multilayer perceptron;

[0088] Step 6: Calculate the node h v Attention weights with the topic:

[0089] Step 6.1: Use equation (9) to get the node h v The kth topic p′ k The attention weight a v,k :

[0090] a v,k =softmax((h v ) T p′ k ) (9)

[0091] In formula (9), T represents transposition;

[0092] Step 6.2: Use equation (10) to get the node h v The K topic attention weights a v :

[0093]

[0094] In formula (10), δ is the set threshold, and 0<δ<1;

[0095] Step 7: Use formula (11) to obtain the feature matrix h of the text graph G :

[0096]

[0097] In formula (11), |V| is the number of nodes in the node set V, and Maxpooling(·) is the maximum pooling function;

[0098] Step 8: Use formula (12) to predict the text represented by the glove model Tags

[0099]

[0100] In formula (12), ⊙ is the dot product, W and b are the set weight matrix and bias;

[0101] Step 9: Use formula (13) to establish the cross entropy loss function Loss:

[0102]

[0103] In formula (13), the label y∈Y, Represents the true probability value of the text graph G in the label y; Indicates the predicted probability value of the text graph G in the label y,

[0104] Step 10: Use the ADAM optimizer to minimize the loss function Loss, thereby optimizing all parameters in the text graph G and obtaining the optimal graph neural network model, such as Figure 2 As shown in the figure, the text classification is realized by using the optimal graph neural network model.

Claims

1. A text classification method based on a graph neural network topic model, characterized by Proceed as follows: Step 1: Get the text collection , and remove the text collection After all the punctuation marks in the text are separated, the stop words are removed to obtain the preprocessed text set. ; From the text collection Get a text from and its tag collection , and the text Included words, denoted as ;in, Represents text Middle words; Step 2: Use the bag-of-words model and glove model to analyze the text Each word in is vectorized to obtain the text represented by the bag-of-words model and the text represented by the glove model ; Step 3. Set up the text collection The number of topics in is K; the vocabulary size of the vocabulary table in the neural topic model NVDM-GSM is set to ; Step 4: Get the text represented by the bag-of-words model The topic distribution and word distribution of topics ; Step 4.1: Get the text represented by the bag-of-words model The topic distribution : Step 4.1.1: Use the neural topic model NVDM-GSM to convert the text represented by the bag-of-words model Mapped to the mean of a Gaussian distribution and variance ; Step 4.1.2: Mean of Gaussian distribution and variance Sampling is performed to obtain hidden variables ; Step 4.1.3: Use formula (1) to transform the latent variable After normalization, we get the text The topic distribution : (1) In formula (1), for dimensional parameter matrix, is a hidden variable The dimension of is the training parameter; represents the activation function; , represents the topic distribution of the kth topic, and , and obtained by formula (2): (2) In formula (2), Represents the parameter matrix No. Theme value; Step 4.2: Get the word distribution of the topic : Step 4.2.

1. Define the topic vector matrix ,and ,in, represents the kth topic vector, represents transpose; Step 4.2.2: Use BP back propagation algorithm to process the topic vector matrix Update to get the updated topic vector matrix ,in, represents the updated kth topic vector; Step 4.2.3: Use formula (3) to get the word distribution of the topic : (3) In formula (3), is the parameter matrix, T represents the transpose; ,in, represents the word distribution under the kth topic, and is obtained by formula (4): (4) Step 5: Text represented by the glove model constructing a text graph and then updating the text graph using a gated graph neural network; Step 5.1: Text represented by the glove model All words in are represented as nodes of the text graph, and the co-occurrences between words are represented as edges of the text graph, thereby constructing a text graph, denoted as ,in, is a set of nodes, is the set of edges; Step 5.2: For any node in the text graph , use the gated graph neural network to update the feature information of the node, and then use formula (5) to get the updated node and record it as ,in, Represents a node Hop count in gated graph neural networks; (5) In formula (5), is the adjacency matrix, is the activation function, all , and are trainable weights and biases, and They are used as update gate and reset gate respectively to determine the contribution of neighbor information to the current node embedding; Using formula (6) Perform nonlinear feature transformation to obtain the transformed nodes : (6) In formula (6), tanh is the hyperbolic tangent function, is a multi-layer perceptron; Step 6: Calculate the nodes Attention weights with the topic: Step 6.1: Use formula (7) to get the node The kth topic The attention weight : (7) In formula (7), T represents transposition; Step 6.2: Use formula (8) to get the node of The weight of topic attention : (8) In formula (8), is the threshold value set, and ; Step 7: Use formula (9) to obtain the feature matrix of the text graph : (9) In formula (9), A set of nodes The number of nodes in is the maximum pooling function; Step 8: Use formula (10) to predict the text represented by the glove model Tags : (10) In formula (10), is the dot product, and is the weight matrix and bias set; Step 9: Use formula (11) to establish the cross entropy loss function : (11) In formula (11), the label , Indicated in the label In the text diagram The true probability value of Indicated in the label In the text diagram The predicted probability value of ; Step 10: Use ADAM optimizer to optimize the loss function Minimize the text graph All parameters in are optimized, and an optimal graph neural network model is obtained, and the text classification is realized by the optimal graph neural network model.

Images