A dual-graph neural network fusing co-occurrence graph and dependency graph and a construction method thereof

Abstract

This invention proposes a dual-graph neural network that integrates co-occurrence graphs and dependency graphs, along with its construction method. To address the issues of wasted memory and poor generalization to new samples when constructing graphs for large corpora, this invention proposes the DGNN-B model to capture features from both word co-occurrence and dependency graphs, thereby optimizing text classification performance. Based on the text, co-occurrence and dependency graphs are constructed to extract local word co-occurrence information and inter-word dependency information. Simultaneously, a bidirectional Long Short-Term Memory (LSTM) network is used to integrate glove word vector representations and part-of-speech information. Then, GCN and GGNN are used to capture features from the dependency and co-occurrence graphs, respectively; finally, the dual-graph representations are fused to improve text classification performance.

Description

Technical Field

[0001] This invention relates to the field of text classification model technology, and in particular to a dual-graph neural network that integrates co-occurrence graphs and dependency graphs, and its construction method. Background Technology

[0002] Text classification is a classic task in natural language processing, using predefined labels to tag text. It is widely used in sentiment analysis, recommender systems, news classification, and more. For example, it can automatically determine the category of a news article, such as sports, politics, or economics. In the information age, the amount of text has exploded. However, due to its unstructured nature, it is difficult to extract valuable features from it, thus reducing the generalization performance of text classification models.

[0003] Current text classification methods include traditional methods and deep learning methods. Traditional methods extract handcrafted features, requiring preprocessing of the text before using models like the bag-of-words model or word2vec to represent it. However, this can lead to the curse of dimensionality, neglect of textual semantics, and data sparsity. Deep learning-based models, such as CNNs, have been widely proposed. [1] RNN [2] LSTM [3] However, the aforementioned models, when classifying text, only focus on sequential and local features, ignoring global word co-occurrence information. In 2018, Google developed a model based on the transformer... [4] Bert was built [5] The model solves the sequence-to-sequence problem. By employing a masked language model and next-sentence prediction methods, the model's generalization performance is improved, significantly enhancing unsupervised pre-training techniques in natural language processing.

[0004] Graph neural network technology refers to using neural networks to learn graph-structured data, extract features and information from the graph-structured data, and then perform downstream tasks such as classification, clustering, and prediction. Bruna [6] Some researchers proposed applying CNNs to graphs, and through a clever transformation of convolution operators, they proposed Graph Convolutional Networks (GCNs). [7] This has led to the development of many variations. Mainstream GNN algorithms include Graph Convolutional Neural Networks, Graph Attention Networks, and Gated Graph Neural Networks. Graph neural network methods for text classification are constantly emerging, providing new ideas for text classification. The word order between texts contains graph structures, such as syntactic and semantic parse trees.

[0005] In 2019, Yao Liang [8]A corpus-level word-word-text graph is constructed, and a graph convolutional neural network is run to capture the relationships between words in the text. Because it's a transductive learning approach, it cannot be easily generalized to new samples. When adding new samples, the graph needs to be reconstructed and retrained from scratch, wasting memory and time. To apply GNNs to short text classification, HGAT was proposed in 2019. [9] When constructing a large graph at the corpus level, topic and entity information are introduced to alleviate the sparsity problem of short texts. In order to solve the problems of direct push and memory consumption, Lianzhe Huang and Yufeng Zhang proposed to construct a graph for the text separately. Lianzhe Huang

[10] first constructs a text hierarchy graph for the text, creates an edge sharing matrix to capture global information, and introduces a message passing mechanism.

[0006] However, it failed to consider that word interactions differ across different texts. In 2020, Yufeng Zhang...

[11] A sliding window is used to build an independent graph in each document, employing a graph gating mechanism (GGNN).

[12] This aggregates neighbor information on the graph, allowing for the summarization of new text during testing. (XienLiu, 2020)

[13] This paper proposes a Tensor Graph Convolutional Network (TGW) for text classification. It constructs a text graph tensor to describe semantic, syntactic, and sequence context information, combining intra-graph and inter-graph propagation, but still relying on transductive learning. Other researchers have explored combining BERT or BiLSTM models with GNNs to complement each other and obtain more comprehensive information. The 2021 TW-TGNN was also mentioned.

[14] Global and local information are obtained by introducing global and local sliding windows. GFN

[15] By converting pre-computed co-occurrence statistics and pre-trained embeddings into structural information, graphs at different text levels are constructed. A multi-head fusion module is designed to integrate the text graph results. This allows for inductive learning of text information and easy generalization to new texts.

[0007] Graph neural network methods have achieved good results in text classification, but there are still some shortcomings. (1) The construction of corpus graphs is a direct inference learning, which trains new samples from scratch, wasting time and memory. (2) Some scholars have proposed to construct text-level graphs to solve the above problems, but they ignore the quality of the graph construction, thus affecting the classification performance. (3) Using pre-trained gloves as the initial features of word vectors cannot focus on the characteristics of the text itself. At the same time, when graph neural networks acquire text features, they ignore the text order. Summary of the Invention

[0008] The purpose of this invention is to propose a dual-graph neural network that integrates co-occurrence graphs and dependency graphs, which has good generalization performance and effectively improves classification accuracy, and its construction method.

[0009] To achieve the above objectives, this invention proposes a dual-graph neural network that integrates co-occurrence graphs and dependency graphs. The dual-graph neural network converts text into a graph structure, transforming the text classification task into a graph classification task; it constructs co-occurrence graphs and dependency graphs for individual texts; and it uses a bidirectional long short-term memory network to capture sequence features and compensate for the thousand-layer features obtained by initializing word vectors with glove.

[0010] This invention also proposes a method for constructing a dual-graph neural network that integrates co-occurrence graphs and dependency graphs, comprising the following steps:

[0011] S1: Prepare the text dataset;

[0012] S2: Preprocess the text to divide it into training and test sets;

[0013] S3: Construct a dual graph

[0014] S3.1: Construction of co-occurrence text graph: Use a sliding window to obtain the co-occurrence relationship between words in the text, and construct an undirected co-occurrence text graph based on the co-occurrence relationship;

[0015] S3.2: Construction of Dependency Graph: Stanford NLP is used to extract the dependency relationships and part-of-speech information between words, and an undirected dependency graph is constructed based on the dependency relationships;

[0016] S4: Graph Feature Extraction

[0017] S4.1: Capturing Features of Co-occurring Text Graphs Using Gated Graph Neural Networks: For co-occurring text graphs, a gated graph neural network is used to extract features from the co-occurring text graphs; the embedded representation of the captured co-occurrence graph is denoted as... d t It is the dimension of the co-occurrence text graph embedding representation;

[0018] S4.2: Capturing Features of Dependency Graphs Using Graph Convolutional Neural Networks: A BiLSTM module is used to extract semantic features of the text. The resulting word embeddings are used as the embedding matrix of the text graph. A graph convolutional neural network is then run to extract features of the dependency graph. The embedded representation of the dependency graph is denoted as... d2 is the dimension of the dependency graph embedding representation;

[0019] S5: Fusion classification compares the predicted labels with the true labels and calculates the loss.

[0020] Furthermore, in step S2, the preprocessing of the text includes converting uppercase to lowercase, cleaning non-text content, removing stop words and low-frequency words, and initializing word embeddings with glove; then, the dataset is shuffled and divided into training and test sets in a 7:3 ratio.

[0021] Furthermore, in step S3, when constructing the text graph, the relationships between words are effectively utilized; using T = {w1,...,w...} i ...w n} Mark a text that has n words, where the i-th word in the text is denoted as w. i Input the text into the embedding layer, and initialize the words as d-dimensional vectors using glove; for T, construct the corresponding text graph, where nodes are composed of words and edges are composed of relationships between words. The text graph is denoted as . V t E t These are the corresponding node set and edge set; n G Setting it to 2 indicates that two types of graphs are constructed for a single text: a co-occurrence graph and a dependency graph; the feature matrix of the text graph is denoted as... The vector representation of the i-th node is x. i ∈R d ,n t n is the number of nodes in the text graph. t =|V t |;The adjacency matrix of a text graph is denoted as

[0022] Furthermore, in step S3.1, a co-occurrence text graph is constructed based on the linguistic features of local co-occurrence between words, denoted as G1=(V1,E1); a fixed-size window is used to slide from left to right on the text, and words in the same window are connected in the graph.

[0023] Furthermore, in step S3.2, a dependency graph is constructed based on the dependencies between words. First, for the text in the corpus, Stanford NLP is used to parse the text and obtain the dependencies between words. The extracted dependencies are directed, but for ease of calculation, the edges are treated as undirected. The text dependencies are denoted as: DP = {r ij |i≠j;i,j<n2}, the word w in the text i and w j The dependency relationship between them is denoted as r. ij Construct a dependency graph based on the dependencies, denoted as G2 = (V2, E2), where E2 is represented as: E2 = {e ij |r ij The dependency graph constructed from ∈DP} contains rich semantic and syntactic features.

[0024] Furthermore, in step S4.1, on the constructed co-occurrence text graph, the GGNN network is run to extract node features, using the idea of ​​gated recurrent units;

[0025] When messages are passed on the graph, a node receives information from its neighboring nodes and then merges it with the previous time-point representation to update its own hidden representation; the propagation process of the model at time t is as follows:

[0026] N t =A1H t-1 W N +b N

[0027] U t =σ(W U N t +M U H t-1 )

[0028] R t =σ(W R N t +M R H t-1 )

[0029]

[0030]

[0031] In the formula, It is the adjacency matrix of the co-occurrence graph, denoted as n, which aggregates neighbor information; U and R are the update gate and reset gate, respectively; R t Decide which old information to use; (1-U t Choose to forget unimportant information, U t Choose which newly generated information to remember; ⊙ is the operator for multiplying corresponding elements; It is newly generated information, H t It is the final node embedding representation at time t; the embedding representation of the capture co-occurrence graph is denoted as d t It is the dimension of the co-occurring text graph embedding representation.

[0032] Furthermore, in step S4.2, the dependency graph is constructed by obtaining node features as the initial embedding representation through BiLSTM, and then further extracting features through the GCN network; in order to capture the sequential features between words, the dependency graph extracts deeper text features through BiLSTM.

[0033] A dependency graph is constructed, and the node representations captured by BiLSTM are used as the feature matrix of the dependency graph, with its adjacency matrix denoted as A2. The GCN model is executed on the graph; the number of convolutional layers is set according to the dataset, and finally, the embedding representation of the dependency graph is denoted as... d2 is the dimension of the dependency graph embedding representation.

[0034] Furthermore, in step S5, the loss calculation steps are as follows:

[0035] The node representations of the dependency graph and co-occurrence graph are first aggregated to generate a graph-level representation H. G-dp H G-co Then the two representations are merged to obtain H. total As the final representation of the text; finally, H is used. total To predict the labels of text;

[0036] First, the neighbor node information is aggregated to obtain the full graph representation H of the bigraph. G-dp H G-co The embedding representation H of the co-occurring text graph is obtained through GGNN. co GCN obtains the embedding representation H of the dependency graph. dp Taking the co-occurrence text graph as an example, the full graph representation H is obtained. G-co The formula is as follows:

[0037] f 1v =σ(o(H) v co ))

[0038]

[0039] H v =f 1v ⊙f 2v

[0040]

[0041] In the formula, o and p are multilayer perceptrons, and f 1v A soft attention mechanism is employed, based on the node's embedding representation H. v co Determine which node is relatively important; f 1v It is the attention weight of word v, f 2v It acts as a nonlinear transformation; yielding the weighted representation H of the words. v ; Apply max and global pooling to obtain the graph representation H of the text G-co Similarly, the graph representation H of the dependency graph can be obtained. G-dp ;

[0042] Then, the two text representations are merged using the concat method, resulting in the following final text representation:

[0043] H total =H G-co ||H G-dp

[0044] Finally, based on the text representation H totalWe use a softmax classifier to predict the labels of the texts and the cross-entropy function as the loss function to calculate the loss between the predicted labels and the true labels of all texts.

[0045]

[0046]

[0047] Compared with existing technologies, the advantages of this invention are: the dual-graph neural network of this invention transforms text into a graph structure, converting the text classification task into a graph classification task. It constructs co-occurrence graphs and dependency graphs for individual texts, solving the problems of difficulty in generalizing to new samples and inability to fully utilize inter-word dependency information. It uses a bidirectional long short-term memory network to capture sequence features, and applies glove...

[16] The method uses thousands of layers of features obtained from initializing word vectors for compensation. This invention outperforms classic models in text classification, exhibiting better generalization performance and higher classification accuracy. Attached Figure Description

[0048] Figure 1 This is a flowchart illustrating the construction method of a dual-graph neural network that integrates a co-occurrence graph and a dependency graph in an embodiment of the present invention.

[0049] Figure 2 This is a diagram of the dual-graph neural network architecture in an embodiment of the present invention;

[0050] Figure 3 This is a co-occurring text graph construction diagram in an embodiment of the present invention;

[0051] Figure 4 This is a dependency parsing graph in an embodiment of the present invention. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions of the present invention will be further described below.

[0053] This invention proposes a Dual Graph Neural Network with BiLSTM (DGNN-B) for text classification. It transforms text into a graph structure, converting the text classification task into a graph classification task. Co-occurrence graphs and dependency graphs are constructed for individual text samples, addressing the problems of poor generalization to new samples and incomplete utilization of inter-word dependency information. A bidirectional long short-term memory network is used to capture sequence features, compensating for the thousands of layers of features obtained by initializing word vectors with glove. This method outperforms classical models in text classification, exhibiting better generalization performance and higher classification accuracy.

[0054] like Figure 1As shown, the method for constructing the dual-graph neural network of the present invention is as follows:

[0055] S1: Prepare the text dataset;

[0056] S2: Preprocess the text; the preprocessing of the text includes converting uppercase to lowercase, cleaning non-text content (such as emoticons, HTML), removing stop words and low-frequency words, and initializing word embeddings with glove; then the dataset is shuffled and divided into training set and test set in a 7:3 ratio.

[0057] To address the issues of wasted memory and poor generalization to new samples when constructing large corpus graphs, the DGNN-B model is proposed to capture features from word co-occurrence graphs and dependency graphs, thereby optimizing text classification performance. The architecture is as follows: Figure 2 As shown, a co-occurrence graph and a dependency graph are constructed based on the text to extract local word co-occurrence information and inter-word dependency information. Simultaneously, a bidirectional Long Short-Term Memory (LSTM) network is used to integrate Glove word vector representations and part-of-speech information. Then, GCN and GGNN are used to capture features of the dependency graph and co-occurrence graph, respectively; finally, the dual graph representations are fused to improve text classification performance, with specific steps described in S3-S5.

[0058] S3: Construct a dual graph

[0059] When constructing a text graph, effectively utilize the relationships between words; use T = {w1,...,w} i ...w n} Mark a text that has n words, where the i-th word in the text is denoted as w. i Input the text into the embedding layer, and initialize the words as d-dimensional vectors using glove; for T, construct the corresponding text graph, where nodes are composed of words and edges are composed of relationships between words. The text graph is denoted as . V t E t These are the corresponding node set and edge set; n G Setting it to 2 indicates that two types of graphs are constructed for a single text: a co-occurrence graph and a dependency graph; the feature matrix of the text graph is denoted as... The vector representation of the i-th node is x. i ∈R d ,n t n is the number of nodes in the text graph. t =|V t |;The adjacency matrix of a text graph is denoted as

[0060] S3.1: Construction of the Co-occurrence Text Graph: A co-occurrence text graph is constructed based on the linguistic features of local co-occurrence between words, denoted as G1 = (V1, E1); a fixed-size window is used to slide across the text from left to right, and words within the same window are connected in the graph; for example... Figure 3 As shown.

[0061] Local sliding windows can effectively capture local co-occurrence features between words. After the co-occurrence text graph is constructed, GGNN is run to propagate and aggregate word feature information.

[0062] S3.2: Construction of Dependency Graph: Co-occurrence graphs can capture local co-occurrence features, but lack semantic information. Dependency analysis

[17]

[18] can capture grammatical dependencies between words, helping to understand text. To solve the above problems, such as Figure 4 As shown, dependency analysis is performed on the text to construct a dependency graph based on the dependencies between words. First, for the text in the corpus, Stanford NLP is used to parse the text and obtain the dependencies between words. The extracted dependencies are directed, but for ease of calculation, the edges are treated as undirected. The dependencies of the text are denoted as: DP = {r ij |i≠j;i,j<n2}, the word w in the text i and w j The dependency relationship between them is denoted as r. ij Construct a dependency graph based on the dependencies, denoted as G2 = (V2, E2), where E2 is represented as: E2 = {e ij |r ij The dependency graph constructed from ∈DP} contains rich semantic and syntactic features.

[0063] Constructing text-level graphs can reduce the number of nodes and edges, and when encountering new samples, training from scratch is unnecessary, representing a form of inductive learning. Compared to a single text graph, constructing two different types of graphs, by fusing dependency and co-occurrence information, can capture more information, facilitating feature extraction from two perspectives and complementing each other during graph propagation.

[0064] S4: Graph Feature Extraction

[0065] S4.1: Capturing features of co-occurring text graphs using gated graph neural networks: On the constructed co-occurring text graph, run a GGNN network to extract node features, using the idea of ​​gated recurrent units;

[0066] When messages are passed on the graph, a node receives information from its neighboring nodes and then merges it with the previous time-point representation to update its own hidden representation; the propagation process of the model at time t is as follows:

[0067] N t =A1Ht-1 W N +b N

[0068] U t =σ(W U N t +M U H t-1 )

[0069] R t =σ(W R N t +M R H t-1 )

[0070]

[0071]

[0072] In the formula, It is the adjacency matrix of the co-occurrence graph, denoted as n, which aggregates neighbor information; U and R are the update gate and reset gate, respectively; R t Decide which old information to use; (1-U t Choose to forget unimportant information, U t Choose which newly generated information to remember; ⊙ is the operator for multiplying corresponding elements; It is newly generated information, H t It is the final node embedding representation at time t; the embedding representation of the capture co-occurrence graph is denoted as d t It is the dimension of the co-occurring text graph embedding representation.

[0073] S4.2: Capturing Features of Dependency Graphs Using Graph Convolutional Neural Networks: For the constructed dependency graph, node features are obtained through BiLSTM as initial embedding representations, and then further features are extracted using the GCN network. To capture the sequential features between words, the dependency graph uses BiLSTM to extract deeper textual features, unlike the co-occurrence graph which only uses shallow features (glove). On the one hand, text data has a non-Euclidean structure. BiLSTM can record the positional information of the text, thereby capturing the sequential features of the text. On the other hand, the bidirectional mechanism of BiLSTM allows each word to obtain more semantic information by fully considering contextual information.

[0074] A dependency graph is constructed, and the node representations captured by BiLSTM are used as the feature matrix of the dependency graph, with its adjacency matrix denoted as A2. The GCN model is executed on the graph; the number of convolutional layers is set according to the dataset, and finally, the embedding representation of the dependency graph is denoted as... d2 is the dimension of the dependency graph embedding representation.

[0075] S5: Fusion classification compares the predicted labels with the true labels and calculates the loss:

[0076] The node representations of the dependency graph and co-occurrence graph are first aggregated to generate a graph-level representation H. G-dp H G-co Then the two representations are merged to obtain H. total As the final representation of the text; finally, H is used. total To predict the labels of text;

[0077] First, the neighbor node information is aggregated to obtain the full graph representation H of the bigraph. G-dp H G-co The embedding representation H of the co-occurring text graph is obtained through GGNN. co GCN obtains the embedding representation H of the dependency graph. dp Taking the co-occurrence text graph as an example, the full graph representation H is obtained. G-co The formula is as follows:

[0078] f 1v =σ(o(H) v co ))

[0079]

[0080] H v =f 1v ⊙f 2v

[0081]

[0082] In the formula, o and p are multilayer perceptrons, and f 1v A soft attention mechanism is employed, based on the node's embedding representation H. v co Determine which node is relatively important; f 1v It is the attention weight of word v, f 2v It acts as a nonlinear transformation; yielding the weighted representation H of the words. v ; Apply max and global pooling to obtain the graph representation H of the text G-co Similarly, the graph representation H of the dependency graph can be obtained. G-dp ;

[0083] Then, the two text representations are merged using the concat method, resulting in the following final text representation:

[0084] H total =H G-co ||H G-dp

[0085] Finally, based on the text representation H totalWe use a softmax classifier to predict the labels of the texts and the cross-entropy function as the loss function to calculate the loss between the predicted labels and the true labels of all texts.

[0086]

[0087]

[0088] References for this application:

[0089] [1]Kim Y.Convolutional Neural Networks for Sentence Classification[C] / / Proc of the 19th Conference on Empirical Methods inNatural LanguageProcessing.2014:1746-1751.

[0090] [2]MIKOLOVT,KARAFIATM,BURGETL,etal.Recurrent neural network basedlanguage model[C] / / Proc of the 11th Annual conference of the InternationalSpeech Communication Association.2010:1045-1048.

[0091] [3]Hochreiter S,Schmidhuber J.Long short-term memory[J].NeuralComputation,1997,9(8):1735-1780.

[0092] [4]Vaswani A,Shazeer N,Parmar N,el al.Attention is all you need[C] / / Proc of the 31st International Conference on Neural Information ProcessingSystem.Red Hook,NY:Curran Associates Inc,2017:6000-6010.

[0093] [5]Devlin J,Chang M W,Lee K,et al.Bert:Pre-training of deepbidirectional transformers for language understanding[C] / / Proc of AnnualConference of the North American Chapter of Association for ComputationalLinguistics:Human Languange Technologies.Stroudsburg,PA:Association forComputational Lingustics,2019:4171-4186.

[0094] [6]Bruna J,Zaremba W,Szlam A,et al.Spectral networks and locallyconnected networks on graphs[EB / OL].(2014-05-21).https: / / arxiv.org / ans / 1312.6203.

[0095] [7]Kipf T N,Welling M.Semi-supervised classification with graphconvolutional networks[C] / / Proc of International Conference on LearningRepresentations.2017:36-50.

[0096] [8]Yao L,Mao C,Luo Y.Graph convolutional networks for textclassification[C] / / Proc of the 33rd AAAI conference on ArtificialIntelligence.Palo Alto,CA:AAAI Press,2019:7370-7377.

[0097] [9]Hu,Linmei,Tianchi Yang,Chuan Shi,Houye Ji and Xiaoli Li.“Heterogeneous Graph Attention Networks for Semi-supervised Short TextClassification.”EMNLP(2019).

[0098]

[10] Huang L,Ma D,Li S,et al.Text level graph neural network for textclassification[C] / / Proc of the 2019 Conference on Empirical Methods inNatural Language Processing and the 9th International Joint Conference onNatural Language Processing.2019:3442-3448.

[0099]

[11] Zhang Y,Yu X,Cui Z,et al.Every document owns its structure:Inductive text classification via graph neural networks[C] / / Proc of the 58thAnnual Meeting of the Association for Computational Linguistics.Stroudsburg,PA:Association for Computational Lingustics,2020:334-339.

[0100]

[12] Li Y,Tarlow D,Brockschmidt M,et al.Gated graph sequence neuralnetworks[C] / / Proc of the 4th International Conference on LearningRepresentations.2015.

[0101]

[13] Liu X,You X,Zhang X,et al.Tensor graph convolutional networks fortext classification[C] / / Proc of the 34th AAAI conference on artificialintelligence.Palo Alto,CA:AAAI Press,2020,34(05):8409-8416.

[0102]

[14] Wu X,Luo Z,Du Z,et al.TW-TGNN:Two Windows Graph-Based Model forText Classification[C] / / 2021International Joint Conference on NeuralNetworks.Piscataway,NJ:IEEE,2021:1-8.

[0103]

[15] Dai Y,ShouL,Gong M,et al.Graph fusion network for textclassification[J].Knowledge-Based Systems,2022,236:107659.

[0104]

[16] Pennington J,Socher R,Manning C D.Glove:Global vectors for wordrepresentation[C] / / Proc ofthe 19th Conference on Empirical Methods in NaturalLanguage Processing.2014:1532-1543.

[0105]

[17] Fan Guofeng, Liu Jing, Yao Shaowen, et al. Text classification model with graph network based on semantic dependency parsing[J]. Application Research of Computers, 2020, 37(12):3594-3598.

[0106]

[18] Shao Dangguo, Zhang Chao, Huang Chusheng, et al. Chinese Comment Classification Model Combining ONLSTM-GCN and Attention Mechanism[J]. Journal of Chinese Computer Systems, 2021, 42(7):1377-1381.

[0107] The above are merely preferred embodiments of the present invention and do not constitute any limitation on the present invention. Any equivalent substitutions or modifications made by those skilled in the art to the technical solutions and content disclosed in the present invention without departing from the scope of the present invention shall be deemed to have remained within the protection scope of the present invention.

Claims

1. A method for constructing a dual-graph neural network that integrates co-occurrence graphs and dependency graphs, wherein the constructed dual-graph neural network converts text into a graph structure, transforming the text classification task into a graph classification task; constructs co-occurrence graphs and dependency graphs for individual texts; uses a bidirectional long short-term memory network to capture sequence features and compensates for the thousand-layer features obtained by initializing word vectors with GloVe; characterized in that, Includes the following steps: S1: Prepare the text dataset; S2: Preprocess the text to divide it into training and test sets; S3: Construct a dual graph S3.1: Construction of co-occurrence text graph: Use a sliding window to obtain the co-occurrence relationship between words in the text, and construct an undirected co-occurrence text graph based on the co-occurrence relationship; S3.2: Construction of Dependency Graph: Stanford NLP is used to extract the dependency relationships and part-of-speech information between words, and an undirected dependency graph is constructed based on the dependency relationships; S4: Graph Feature Extraction S4.1: Capturing features of the co-occurring text graph using a gated graph neural network: For the co-occurring text graph, the features of the co-occurring text graph are extracted using the gated graph neural network; the embedding representation of the captured co-occurrence graph is denoted as... , d t It is the dimension of the co-occurrence text graph embedding representation; S4.2: Capture the features of the dependency graph using a graph convolutional neural network: Use the BiLSTM module to extract the semantic features of the text, and use the resulting word embedding representation as the embedding matrix of the text graph. Run the graph convolutional neural network to extract the features of the dependency graph. The embedding representation of the dependency graph is denoted as , It is the dimension of the dependency graph embedding representation; S5: Fusion classification compares the predicted labels with the true labels and calculates the loss.

2. The method for constructing a dual-graph neural network that integrates a co-occurrence graph and a dependency graph according to claim 1, characterized in that, In step S2, the preprocessing of the text includes converting uppercase to lowercase, cleaning non-text content, removing stop words and low-frequency words, and initializing word embeddings using glove; then, the dataset is shuffled and divided into training and test sets in a 7:3 ratio.

3. The method for constructing a dual-graph neural network that integrates a co-occurrence graph and a dependency graph according to claim 1, characterized in that, In step S3, when constructing the text graph, the relationships between words are effectively utilized; using Mark a text that has n The word in the text, the first word Each word is written as w i ; Input the text into the embedding layer and initialize the words using glove. d dimensional vector; for T Construct a corresponding text graph for it, where nodes are composed of words and edges are composed of relationships between words. The text graph is denoted as . , These are the corresponding node sets and edge sets; n G Setting it to 2 indicates that two types of graphs are constructed for a single text: a co-occurrence graph and a dependency graph; the feature matrix of the text graph is denoted as... , No. i The vector representation of the nodes is as follows , This is the number of nodes in the text graph. The adjacency matrix of a text graph is denoted as... .

4. The method for constructing a dual-graph neural network that integrates a co-occurrence graph and a dependency graph according to claim 3, characterized in that, In step S3.1, a co-occurrence text graph is constructed based on the linguistic features of local co-occurrence between words, denoted as... Using a fixed-size window, slide across the text from left to right; words in the same window are connected in the diagram.

5. The method for constructing a dual-graph neural network that integrates a co-occurrence graph and a dependency graph according to claim 4, characterized in that, In step S3.2, a dependency graph is constructed based on the dependencies between words. First, for the text in the corpus, Stanford NLP is used to parse the text and obtain the dependencies between words. The extracted dependencies are directed, but for ease of calculation, the edges are treated as undirected. The dependencies of the text are denoted as: Words in the text and The dependency relationship between them is denoted as ; Construct a dependency graph based on the dependencies, denoted as ,in E 2 Represented as: The dependency graph constructed in this way contains rich semantic and syntactic features.

6. The method for constructing a dual-graph neural network that integrates a co-occurrence graph and a dependency graph according to claim 1, characterized in that, In step S4.1, on the constructed co-occurrence text graph, the GGNN network is run to extract node features, using the idea of ​​gated recurrent units; When messages are passed on the graph, a node receives information from its neighbors and then merges it with the previous time-point representation to update its own hidden representation; the model in The propagation process of time is as follows: ； ； ； ； ； In the formula, It is the adjacency matrix of the co-occurrence graph, and the aggregated neighbor information is denoted as . n ; U and R These are the update door and the reset door; R t Decide which old information to use; (1-U t ) Choose to forget unimportant information. U t Choose which newly generated information to remember; It is the operator for element-wise multiplication; It is newly generated information. yes t The final node embedding representation at each moment; The embedding representation of the capture co-occurrence graph is denoted as , d t It is the dimension of the co-occurring text graph embedding representation.

7. The method for constructing a dual-graph neural network that integrates a co-occurrence graph and a dependency graph according to claim 1, characterized in that, In step S4.2, the dependency graph is constructed by obtaining node features as the initial embedding representation through BiLSTM, and then further extracting features in the GCN network; To capture the sequential features between words, the dependency graph extracts deeper text features using BiLSTM; The dependency graph is constructed using the node representations captured by BiLSTM as the feature matrix of the dependency graph, and its adjacency matrix is ​​denoted as... The GCN model operates on a graph; the number of convolutional layers is set based on the dataset, and the final embedding representation of the dependency graph is denoted as... , It is the dimension of the dependency graph embedding representation.

8. The method for constructing a dual-graph neural network that integrates a co-occurrence graph and a dependency graph according to claim 1, characterized in that, In step S5, the loss is calculated as follows: The node representations of the dependency graph and co-occurrence graph are first aggregated to generate a graph-level representation. H G-dp , H G-co Then, the two representations are merged to obtain... H total As the final representation of the text; used last. H total To predict the labels of text; First, we aggregate the neighbor node information to obtain the full graph representation of the bi-graph. H G-dp , H G-co ; Obtain the embedding representation of the co-occurrence text graph through GGNN. H co GCN obtains the embedding representation of the dependency graph. H dp Taking co-occurring text graphs as an example, the full graph representation is obtained. H G-co The formula is as follows: ； ； ； ； In the formula, and It is a multilayer perceptron. Employing a soft attention mechanism, based on the node's embedding representation. Determine which node is relatively important; It is a word Attention weights It acts as a nonlinear transformation; it yields the weighted representation of words. ; Apply max and global pooling to obtain a graph representation of the text. Similarly, the graphical representation of the dependency graph can be obtained. ; Then, the two text representations are merged using the concat method, resulting in the following final text representation: ； Finally, based on the text representation H total We use a softmax classifier to predict the labels of the texts and the cross-entropy function as the loss function to calculate the loss between the predicted labels and the true labels of all texts. ； 。

Images