A keyword extraction method for graph convolutional neural network based on augmented storage

Abstract

The present application discloses a keyword extraction method based on a graph convolutional neural network with augmented storage, which improves the node representation capability of a multi-layer graph convolutional neural network by setting an augmented storage graph convolutional neural network keyword extraction model. The text is preprocessed and the text is syntactically analyzed, and the analysis results are respectively input into the word embedding layer to obtain the text graph structure information and the embedding information of the word nodes and input into the graph convolutional neural network layer with augmented storage to obtain the corresponding vector representation, which is spliced ​​and input into the LSTM layer to obtain the output vector, and the vector representation and the output vector are transmitted to the output layer for splicing and output to the target layer to obtain the output category. The present application solves the technical problem that the keyword extraction performance of the existing short text keyword extraction method is poor, and the graph convolutional neural network used reduces the node representation capability as the convolutional layer is improved.

Description

Technical Field

[0001] The present application belongs to the technical field of keyword extraction, and in particular, relates to a keyword extraction method of a graph convolutional neural network based on augmented storage. Background Art

[0002] With the rise of social media, short texts have become the main carrier of information transmission on the Internet. For example, the main content of Weibo, Moments, web page titles, search snippets, e-commerce comments, Facebook and other websites are all presented in the form of short texts. Therefore, short text keyword extraction has become a very important research direction in the field of text information extraction. Just as the role of document summaries in long texts (news, blogs, etc.), keywords can accurately reflect the core content of short texts, which is an important way for people to quickly understand the content of documents and grasp the theme of short texts. In addition, short text keyword extraction plays a positive role in short text clustering and short text classification tasks in the field of natural language processing, and also has important application value in information retrieval fields such as automatic question answering, topic tracking and intelligent customer service.

[0003] Deep learning is a method of using computers to imitate the way human neural networks process information. After the 2010s, it has made major breakthroughs in image processing, natural language processing and other fields. The attention mechanism in deep learning draws on a brain information processing mechanism that is unique to human vision. Generally speaking, when humans observe the external environment, they will quickly scan the entire scene, and then quickly lock on to the target area of ​​focus based on brain signals, and finally form a focus of attention. This mechanism can help humans filter out target areas with important valuable information from a large number of irrelevant background areas under limited resources, helping humans to process visual information more efficiently. In the field of natural language processing, the attention mechanism is used to locate key words to achieve the purpose of efficiently and accurately processing text information. Existing short text keyword extraction methods usually use graph convolutional neural networks in deep learning to extract keywords, but the existing short text keyword extraction methods have the following defects:

[0004] 1. The existing open-domain short text keyword extraction methods contain a large number of unregistered words. Therefore, the deep learning-based keyword extraction methods cannot represent unregistered words well, which restricts the keyword extraction performance.

[0005] 2. Graph convolutional neural networks can fuse the information of adjacent nodes and better represent nodes. However, as the number of convolutional layers increases, the representation ability of nodes decreases.

[0006] 3. Existing short text keyword extraction methods do not make good use of the structural information of the current text to improve the vector representation of words, especially the lack of vector representation for unregistered words. Summary of the invention

[0007] The present application provides a keyword extraction method of a graph convolutional neural network based on augmented storage, which is used to solve the technical problems that the keyword extraction performance of existing short text keyword extraction methods is poor and the graph convolutional neural network used reduces the representation ability of nodes as the convolution layer is improved.

[0008] In view of this, the present application provides a keyword extraction method based on a graph convolutional neural network with augmented storage, comprising:

[0009] An augmented storage graph convolutional neural network keyword extraction model is set up. The augmented storage graph convolutional neural network keyword extraction model includes a word embedding layer, an augmented storage graph convolutional neural network layer, an LSTM layer, an output layer, and a target layer.

[0010] Preprocess the text to obtain the semantic graph structure of the text, perform syntactic analysis on the text, and input the analysis results into the word embedding layer respectively;

[0011] The text graph structure information and word node embedding information obtained after the word embedding layer are input into the graph convolutional neural network layer of the augmented storage to obtain the corresponding vector representation, which is then concatenated and input into the LSTM layer.

[0012] The vector representation obtained by the graph convolutional neural network layer after amplification and storage and the output vector obtained by the LSTM layer are transmitted to the output layer for splicing and output to the target layer to obtain the output category.

[0013] Optionally, preprocess the text to obtain a semantic graph structure of the text, perform syntactic analysis on the text, and input the analysis results into the word embedding layer respectively, including:

[0014] Preprocess the text to obtain the word set and text sequence of the text, perform syntactic analysis on the text to obtain syntactic relations, convert the words in the word set into nodes in the text graph, obtain the syntactic dependency between the words in the text sequence based on the syntactic relations, establish edges between the words, and input the processed word set and text sequence into the word embedding layer respectively.

[0015] Optionally, the augmented storage graph convolutional neural network keyword extraction model also includes a Bert layer.

[0016] Optionally, the text graph structure information and the embedding information of the word nodes obtained after the word embedding layer are input into the graph convolutional neural network layer of the augmented storage to obtain the corresponding vector representation, and the vector representation is spliced ​​and input into the LSTM layer, including:

[0017] The initial vector of the word obtained after the word embedding layer is input into the Bert layer to obtain the word set vector representation, and the text graph structure information and the initial vector of the word are input into the graph convolutional neural network layer for amplified storage to obtain the text sequence vector representation of the word node. The obtained word set vector representation and text sequence vector representation are concatenated and input into the LSTM layer.

[0018] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0019] The present application provides a keyword extraction method based on a graph convolutional neural network with augmented storage, which improves the node representation ability of a multi-layer graph convolutional neural network by setting an augmented storage graph convolutional neural network keyword extraction model. The text is preprocessed to obtain the semantic graph structure of the text, the text is syntactically analyzed, and the analysis results are respectively input into the word embedding layer, the text graph structure information obtained after the word embedding layer and the embedding information of the word node are input into the graph convolutional neural network layer with augmented storage to obtain the corresponding vector representation, the vector representation is spliced ​​and input into the LSTM layer, the vector representation obtained by the graph convolutional neural network layer with augmented storage and the output vector obtained by the LSTM layer are transmitted to the output layer for splicing and output to the target layer to obtain the output category. The existing short text keyword extraction method has poor keyword extraction performance, and the technical problem that the graph convolutional neural network used reduces the node representation ability as the convolutional layer is improved is solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0021] Figure 1 A flowchart of a keyword extraction method based on a graph convolutional neural network with augmented storage provided in an embodiment of the present application;

[0022] Figure 2 A schematic diagram of the structure of the augmented storage graph convolutional neural network keyword extraction model provided in the embodiments of the present application;

[0023] Figure 3 A structural diagram of the text graph structure provided in an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0025] For easier understanding, see Figures 1 to 3 , an embodiment of a keyword extraction method based on a graph convolutional neural network with augmented storage provided by the present application includes:

[0026] Step 101, set an augmented storage graph convolutional neural network keyword extraction model, the augmented storage graph convolutional neural network keyword extraction model includes a word embedding layer, an augmented storage graph convolutional neural network layer, an LSTM layer, an output layer and a target layer.

[0027] Step 102: pre-process the text to obtain the semantic graph structure of the text, perform syntactic analysis on the text, and input the analysis results into the word embedding layer respectively.

[0028] Step 103: Input the text graph structure information and the embedding information of the word nodes obtained after the word embedding layer into the graph convolutional neural network layer of the augmented storage to obtain the corresponding vector representation, and concatenate the vector representation and input it into the LSTM layer.

[0029] Step 104: The vector representation obtained by the amplified and stored graph convolutional neural network layer and the output vector obtained by the LSTM layer are transmitted to the output layer for splicing and output to the target layer to obtain the output category.

[0030] It should be noted that spaCy is a Python natural language processing toolkit and a Python NLP toolkit with industrial strength. Cython is used extensively in spaCy to improve the performance of related modules. The augmented storage graph convolutional neural network keyword extraction model also includes a Bert layer. In order to solve the problem that the node representation ability of the graph convolutional neural network decreases as the number of layers increases, the present application sets an augmented storage graph convolutional neural network keyword extraction model, which includes a word embedding layer, a Bert layer, an augmented storage graph convolutional neural network layer, an LSTM layer, an output layer and a target layer. The augmented storage graph convolutional neural network layer improves the expression ability of the node by adding an augmented storage structure to the graph convolutional neural network, and a storage structure is augmented for each node in the text graph. Each text T corresponds to a text graph G, which can be expressed as G = {V, E}, where V represents the node set in the text graph and E represents the set of edges between nodes. When using the augmented storage graph convolutional neural network keyword extraction model to extract text keywords, the text is first preprocessed by performing sentence segmentation, word segmentation, syntactic analysis, and stop word removal operations to obtain the word set and text sequence of the text. For a given text T, the word set of text T is represented by W s ={w s1 ,w s2 ,…,w sm}, the number of words in the text is m, that is, |W s |=m. The text T is composed by s The text sequence is represented as: W t =(w t1 ,w t2 ,…,w tk ), the word w in the sequence ti Satisfy w ti ∈W s , the length of the text sequence is k, that is, |W t |=k. For example, given the text "Omicron is a virus that spreads faster than Delta". After being processed by spaCy, the word segmentation is obtained. After removing stop words, the result is: ['Omicron', 'virus', 'is', 'than', 'Delta', 'virus', 'spread', 'more', 'faster', 'one', 'kind', 'virus'], then the word set of the text is: W s ={'Omicron', 'virus', 'is', 'than', 'Delta', 'spreads', 'more', 'faster', 'one', 'kind'}, and |W s |=10. The text sequence of the text is represented as: W t=('Omicron', 'virus', 'is', 'than', 'Delta', 'virus', 'spread', 'more', 'faster', 'one', 'kind', 'virus'), | W t |=12. Then perform syntactic analysis on the text to obtain syntactic relations and convert the word set W s The words in are converted into text graph nodes, and the text sequence W is obtained according to the syntactic relationship. t The syntactic dependency relationship between words in the text is established, and the edges between words are established. For each text T, there is a text graph G, which can be expressed as G = {V, E}, where V represents the set of nodes in the text graph, E represents the set of edges between nodes, and the word set W s Each word in corresponds to a node in V, and the dependency relationship between words constitutes a set of edges between words. For example, the text graph structure corresponding to the given text "Omicron virus is a virus that spreads faster than Delta virus" is as follows: Figure 3 As shown. The processed word set W s and the text sequence W t Input word embedding layer, word set W s and the text sequence W t After the word embedding layer, the initial vectors of text graph nodes and words are obtained and Will The text graph structure information is input into the graph convolutional neural network layer of the augmented storage, and an additional storage structure is augmented for each node in the graph. Same, expressed as The update method of the t node of the convolutional network of the lth layer in the graph convolutional neural network is:

[0031]

[0032] Where N t represents the set of adjacent nodes of t, W l1 , U l1 is a parameter matrix and σ is a nonlinear activation function.

[0033]

[0034] W l2 , U l2 is the parameter matrix.

[0035] The last layer of graph convolutional neural network obtains the vector representation of text graph nodes After the word embedding layer, we get Input to the Bert layer to obtain the vector representation of the word set with self-attention by To query the table, obtain the text sequence Wt The corresponding vector representation Will and The concatenation is input into the LSTM layer to obtain the output vector, which is the vector representation of each word in the text sequence. The vector representation of the text graph nodes obtained by the graph convolutional neural network layer after augmentation storage After the output layer, the output representation y is obtained s , which is used to indicate whether the word is a component of the keyword, that is, there are two classification labels {yes, no}. The output vector obtained by the LSTM layer After the output layer, the output representation y is obtained t , used to indicate which part of the keyword it is, that is, there are four classification labels {beginning, middle, end, not}, and y s and t The calculation input is sent to the target layer to calculate the joint loss, and the network model is trained according to the gradient descent method. After the training is completed, the keywords of the text are predicted according to the parameters obtained by the training, which can better represent the unregistered words and improve the performance of text keyword extraction.

[0036] A keyword extraction method based on a graph convolutional neural network with augmented storage provided in an embodiment of the present application, firstly, an augmented storage graph convolutional neural network keyword extraction model is set, then, the text is preprocessed to obtain the semantic graph structure of the text, the text is syntactically analyzed, and the analysis results are respectively input into the word embedding layer, and then, the text graph structure information obtained after the word embedding layer and the embedding information of the word node are input into the graph convolutional neural network layer of augmented storage to obtain the corresponding vector representation, and the vector representation is spliced ​​and input into the LSTM layer. Finally, the vector representation obtained by the graph convolutional neural network layer of augmented storage and the output vector obtained by the LSTM layer are transmitted to the output layer for splicing and output to the target layer to obtain the output category. The method of the present invention aims to improve the representation ability of words, especially the representation ability of unregistered words. In the text keyword extraction task, due to the limitation of the word vector table, a large number of unregistered words appear. In the short text keyword extraction in the open field, due to the non-standard word use, this problem is more prominent. For example, in the given text "Sumya Swamina said that the Delta virus spreads faster.", "Sumya Swamina" and "Delta" are both unregistered words. When extracting keywords, they are all represented by the same vector. Obviously, this cannot distinguish the meanings of different unregistered words. The present application can better capture the syntactic relationship between text words by setting a text graph structure, and the node representation obtained by the text through the graph convolutional neural network can better obtain the information of words with relevant dependencies between nodes, which is equivalent to putting the unregistered words in the context to understand them, so the vector representation of the words in the text can be better obtained. In view of the problem that the representation ability of nodes is reduced as the number of layers of the graph convolutional neural network increases, the present invention proposes a graph convolutional neural network model based on an amplified storage structure, which can better fuse the information of adjacent nodes and better represent nodes. It solves the technical problem that the keyword extraction performance of the existing short text keyword extraction method is poor, and the graph convolutional neural network used reduces the representation ability of nodes as the convolutional layer is improved.

[0037] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A keyword extraction method based on graph convolutional neural network with augmented storage, characterized in that: include: An augmented storage graph convolutional neural network keyword extraction model is set up. The augmented storage graph convolutional neural network keyword extraction model includes a word embedding layer, an augmented storage graph convolutional neural network layer, an LSTM layer, an output layer, a target layer, and a Bert layer. Preprocessing the text to obtain a semantic graph structure of the text, performing syntactic analysis on the text, and inputting the analysis results into the word embedding layer respectively; Inputting the text graph structure information and the embedding information of the word nodes obtained after passing through the word embedding layer into the graph convolutional neural network layer of the augmented storage to obtain the corresponding vector representation, and splicing the vector representation and inputting it into the LSTM layer; The vector representation obtained by the amplified and stored graph convolutional neural network layer and the output vector obtained by the LSTM layer are transmitted to the output layer for splicing and output to the target layer to obtain an output category; The steps of preprocessing the text to obtain a semantic graph structure of the text, performing syntactic analysis on the text, and inputting the analysis results into the word embedding layer include: Preprocessing the text to obtain a word set and a text sequence of the text, performing syntactic analysis on the text to obtain syntactic relations, converting words in the word set into nodes in a text graph, obtaining syntactic dependency relations between words in the text sequence according to the syntactic relations, establishing edges between the words, and inputting the processed word set and text sequence into the word embedding layer respectively; The text graph structure information and the embedding information of the word nodes obtained after the word embedding layer are input into the graph convolutional neural network layer of the augmented storage to obtain the corresponding vector representation, and the vector representation is spliced ​​and input into the LSTM layer, including: After the word set passes through the word embedding layer, the initial vector of the text graph node is obtained, and after the text sequence passes through the word embedding layer, the initial vector of the word is obtained; The initial vector of the text graph node and the text graph structure information are input into the graph convolutional neural network layer of the augmented storage, and an additional storage structure is augmented for each node in the text graph, so that the update method of the t node of the convolutional network of the lth layer in the augmented storage graph convolutional neural network layer is: Where N t represents the set of adjacent nodes of t, W l1 , U l1 is the parameter matrix, σ is the nonlinear activation function; W l2 , U l2 is the parameter matrix; The last graph convolutional neural network layer in the expanded and stored graph convolutional neural network layer obtains the vector representation of the text graph nodes; the initial vector of the word is input into the Bert layer to obtain the vector representation of the word set with self-attention; the vector representation of the text graph node obtained by the last layer of graph convolutional neural network is used as a query table to obtain the vector representation corresponding to the text sequence, and the vector representation corresponding to the text sequence and the vector representation of the word set with self-attention are concatenated and input into the LSTM layer to obtain the output vector.

Images