Named entity extraction method, device and storage medium based on graph neural network

Abstract

The application relates to a named entity extraction method and device based on a graph neural network and a storage medium, and aims to solve the problem that the precision of an existing named entity extraction method cannot meet the requirements. In the device, a heterogeneous graph module converts a sentence into a sentence heterogeneous graph based on a directed graph with words as nodes according to the results of the lexical analysis and the dependency relation analysis of the sentence; an initial vector module obtains a word initial word vector by splicing a word semantic information vector, a character information vector and a part-of-speech information vector in the sentence; and a named entity recognition model performs named entity recognition according to the input word initial word vector and the sentence heterogeneous graph and finally outputs the named entity. The application improves the effect and precision of entity extraction.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing technology of artificial intelligence, and specifically to a named entity extraction method based on a graph neural network, a named entity extraction device based on a graph neural network, and a storage medium. Background Art

[0002] Natural language text is one of the most important data types in the digital world, generated across a wide range of domains, including social networks, chatbots, and digital media. Named entity recognition (NER) is fundamental to numerous text processing tasks, such as intelligent search, intent recognition, and knowledge graphs. Named entity models are trained on annotated text to automatically extract the required entities (e.g., people, locations, and events) from text.

[0003] Existing named entity models are primarily based on supervised learning neural network models. These models employ an encoder-decoder architecture to learn the characteristics and patterns of named entities based on word order information and the semantics of the words themselves. The encoder maps words into a low-dimensional vector space based on this word order and semantic information, while the decoder determines whether a word is an entity based on its vector representation. Furthermore, to further enrich the semantic information of words, some methods incorporate syntactic analysis to capture inter-word dependencies, thereby improving the accuracy of entity recognition.

[0004] Current mainstream entity recognition methods model entity recognition as a sequence (sentence) labeling problem, using deep neural network models to learn text features and automatically extract the required entities in an end-to-end manner. Therefore, most of these models use neural network models that can learn word order, such as recurrent neural networks and attention mechanisms, to extract text features. This process also incorporates semantic information from pre-trained language models to enrich the feature information and achieve better entity extraction accuracy.

[0005] Due to the complex grammar of natural languages, relying solely on the word order of text cannot effectively extract structural information between words. For example, active and passive voices can express the same thing, but the word order can be very different. Therefore, some methods use dependency trees between words obtained through grammatical analysis to express word structure and integrate this information into the model to improve entity recognition accuracy. However, current entity recognition models using dependency trees are still based on sequential neural networks and cannot effectively utilize the structural information in sentences to improve entity recognition accuracy. Summary of the Invention

[0006] In view of this, the purpose of the present invention is to provide a named entity extraction method, device and storage medium based on graph neural network, so as to solve the current problem that sequential neural network-based methods cannot effectively improve the accuracy of entity recognition, and achieve the purpose of improving the accuracy of named entity recognition.

[0007] According to a first aspect of an embodiment of the present invention, a method for extracting named entities based on a graph neural network is provided, the method comprising:

[0008] According to the results of sentence lexical analysis and dependency analysis, the sentence is converted into a sentence heterogeneous graph based on a directed graph with words as nodes;

[0009] The initial word vector of the word is obtained by concatenating the semantic information vector, character information vector, and part-of-speech information vector of the word in the sentence;

[0010] The sentence heterogeneous graph and the initial word vector of each word are input into a named entity recognition model, and the extracted named entities are output.

[0011] Preferably, the named entity recognition model includes a multi-layer graph neural network encoder and a conditional random field decoder connected in sequence; the initial word vector of the word is input into the first layer of the graph neural network of the multi-layer stacked graph neural network encoder; the sentence heterogeneous graph is input into each layer of the graph neural network of the multi-layer stacked graph neural network encoder; the conditional random field-based decoder decodes the final word vector output by the multi-layer stacked graph neural network encoder, and extracts the named entity based on the decoding result.

[0012] Preferably, the sentence isomerism graph includes nodes corresponding to words in the sentence, wherein the node attribute is the word part of speech; and also includes directed edges representing dependency relationships between nodes and / or directed edges representing word order relationships between nodes.

[0013] Preferably, in the step of obtaining the initial word vector of a word by splicing the semantic information vector, character information vector, and part-of-speech information vector of the word in the sentence:

[0014] The semantic information vector is obtained by: obtaining the semantic information vector of the word from the pre-trained language model, and using a random vector as the semantic information vector when the word does not exist in the pre-trained language model;

[0015] The character information vector is obtained by encoding the word through an LSTM recurrent neural network and using the output of the last state of the LSTM as the character information vector;

[0016] The part-of-speech information vector is determined by setting a random initial vector for each part-of-speech.

[0017] Preferably, the sentence heterogeneous graph is input to each layer graph neural network of the multi-layer stacked graph neural network encoder, and each single layer graph neural network aggregates information of node neighbors in the sentence heterogeneous graph into nodes, and encodes word order information and dependency relationship information between nodes in the sentence heterogeneous graph into word vectors corresponding to the nodes.

[0018] Preferably, the graph neural network is a graph attention network, and a single layer graph attention network utilizes the encoded word vector of the sentence heterogeneous graph to represent as:

[0019]

[0020] wherein, represents a word vector of the “l”-th layer, represents a neighbor node vector of the “i”-th node in the “l-1”-th layer index;

[0021] wherein, e ij represents attention of the i-th node to the j-th node;

[0022] wherein, e ij =a(Wh i ,W(h j ||r j ), r j represents a type of an edge connected between two node word vectors h i and h j , and a and W are one-dimensional and high-dimensional linear mappings respectively.

[0023] Preferably, the multi-layer graph neural network encoder is a stacked multi-layer graph neural network, and a specific stacking manner is that an output of each layer graph neural network is directly connected to an output end of the multi-layer graph neural network encoder.

[0024] The final output of the multi-layer graph neural network encoder is represented as a splicing result h of outputs of each layer graph neural network:

[0025] h=w||h 1 ||h 2 …||h n ,

[0026] wherein, the symbol “||” represents a splicing relationship, n is a number of layers of the multi-layer graph neural network, and w is an initial word vector of a word.

[0027] Preferably, the graph neural network is a graph convolutional neural network or a graph attention network.

[0028] According to a second aspect of an embodiment of the present application, a named entity extraction device based on a graph neural network is provided, and the device comprises:

[0029] The heterogeneous graph module is used to convert sentences into a sentence heterogeneous graph based on a directed graph with words as nodes based on the results of sentence lexical analysis and dependency analysis;

[0030] The initial vector module is used to obtain the initial word vector of a word by splicing the semantic information vector, character information vector, and part-of-speech information vector of the word in the sentence;

[0031] Named entity recognition model, which is used to perform named entity recognition based on the input word initial word vector and sentence heterogeneous graph, and finally output the extracted named entity;

[0032] The named entity recognition model includes a sequentially connected multi-layer graph neural network encoder and a conditional random field decoder; the initial word vector of the word based on the graph representation is input into the multi-layer stacked graph neural network encoder; the sentence heterogeneous graph is input into each layer of the graph neural network of the multi-layer stacked graph neural network encoder; the conditional random field-based decoder decodes the final word vector output by the multi-layer stacked graph neural network encoder, and extracts the named entity based on the decoding result.

[0033] According to a third aspect of an embodiment of the present invention, a storage medium is provided, on which a processing program for named entity extraction based on a graph neural network is stored. When the processing program for named entity extraction based on a graph neural network is executed by a processor, the steps of the named entity extraction method based on a graph neural network are implemented.

[0034] The technical solutions provided by the embodiments of the present invention may have the following beneficial effects:

[0035] This method, based on graph analysis, fuses the parts of speech, word order, and dependency relationships of words in a text into a sentence heterogeneous graph, and uses a graph attention network or a graph convolutional neural network to extract entity information. This method utilizes the semantic information in the text more comprehensively and effectively than existing methods, thereby improving the effectiveness and accuracy of entity extraction. On the one hand, the present invention provides a directed graph-based text representation by constructing a sentence heterogeneous graph, and on the other hand, it provides an entity extraction method based on a graph neural network.

[0036] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0038] Figure 1 Schematic diagram of a named entity extraction device based on graph neural network;

[0039] Figure 2 This is a flowchart of a named entity extraction method based on graph neural network;

[0040] Figure 3 A diagram of training a named entity model based on graph neural network. DETAILED DESCRIPTION

[0041] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present invention as detailed in the appended claims.

[0042] The technical solution of this invention uses graph analysis to model a sentence as a heterogeneous sentence graph, where words serve as nodes and word order information and inter-word dependencies serve as edges of different categories. An encoder composed of a graph attention network or a graph neural network then extracts text features from the rich graph structure of the heterogeneous sentence graph, achieving more accurate entity recognition.

[0043] like Figure 1 As shown in FIG, a schematic diagram of a named entity extraction device based on a graph neural network is provided, and the device includes:

[0044] The heterogeneous graph module is used to convert sentences into a sentence heterogeneous graph based on a directed graph with words as nodes based on the results of sentence lexical analysis and dependency analysis;

[0045] The initial vector module is used to obtain the initial word vector of a word by splicing the semantic information vector, character information vector, and part-of-speech information vector of the word in the sentence;

[0046] Named entity recognition model, which is used to extract named entities based on the input word initial word vector and sentence heterogeneous graph output;

[0047] The named entity recognition model includes a sequentially connected multi-layer graph neural network encoder and a conditional random field decoder; the initial word vector of the word represented by the graph is input into the multi-layer stacked graph neural network encoder; the sentence heterogeneous graph is input into each layer of the graph neural network of the multi-layer stacked graph neural network encoder;

[0048] The conditional random field-based decoder decodes the final word vector output by the multi-layer stacked graph neural network encoder, and extracts the named entity based on the decoding result.

[0049] like Figure 2As shown, a named entity extraction method based on graph neural network is provided, and the method includes the following steps:

[0050] Step S1: According to the results of lexical analysis and dependency analysis of the sentence, the sentence is converted into a sentence heterogeneous graph based on a directed graph with words as nodes;

[0051] Step S2: Concatenate the semantic information vector, character information vector, and part-of-speech information vector of the word in the sentence to obtain the initial word vector;

[0052] Step S3: Input the sentence isomorphic graph and the initial word vector of the word into the named entity recognition model, and output the extracted named entity.

[0053] Furthermore, the named entity recognition model includes a sequentially connected multi-layer graph neural network encoder and a conditional random field decoder. In step S3, the graph-represented initial word vector of the word is input into the multi-layer stacked graph neural network encoder; the conditional random field-based decoder decodes the final word vector output by the multi-layer stacked graph neural network encoder and extracts the named entity based on the decoding result. The initial word vector of the word is input into the first layer of the graph neural network of the multi-layer stacked graph neural network encoder; the sentence heterogeneous graph is input into each layer of the graph neural network of the multi-layer stacked graph neural network encoder; the conditional random field-based decoder decodes the final word vector output by the multi-layer stacked graph neural network encoder and extracts the named entity based on the decoding result.

[0054] In step S1, based on the results of lexical analysis and dependency analysis, the sentence is converted into a sentence heterogeneous graph based on a directed graph with words as nodes. Specifically, this includes performing lexical analysis and dependency analysis on the sentence to obtain word parts of speech and mutual dependencies; and constructing a sentence heterogeneous graph based on the order and dependency relationships between words.

[0055] In this step, the natural language text in string form is lexically analyzed using the open-source NLTK tool, which converts it into a word sequence and obtains the part of speech of each word. The sentence after lexical analysis is then subjected to dependency analysis using Stanza's syntax analysis tool, obtaining a dependency tree between words. The tree's nodes represent words, while edges indicate whether dependencies exist between words. The edge types represent different types of dependencies.

[0056] The sequence-structured sentence representation is converted into a directed graph-based representation. In the graph corresponding to the sentence, words are established as nodes, with their parts of speech as node attributes. Edges between nodes represent the relationships between words. Two types of edges are established here: 1) edges representing word order: word A has an edge connected to word B if A is the previous word of B in the sentence; 2) edges representing dependency relationships: word A has an edge connected to word B if A is a child of B in the dependency tree, and the edge type corresponds to the dependency relationship type in the dependency tree.

[0057] In step S2, the initial word vector of the word is obtained by splicing the semantic information vector, character information vector, and part-of-speech information vector of the word in the sentence.

[0058] In this step, the word is initialized into a low-dimensional vector, which represents three parts of information:

[0059] 1) Semantic information of words: obtain the vector w representing the semantics of words from open source pre-trained language models such as Bert and Glove. semantic For words that do not exist in the pre-trained language model, random vectors are used;

[0060] 2) Character information of words: consider words as sequences of characters, use LSTM recurrent neural network to encode words, and use the output of the last state of LSTM as vector w character ;

[0061] 3) Part of speech information of the word, set a random initial vector for each part of speech, and use it as the part of speech vector w of the word pos . Splicing the above three vectors to get the initial word vector w=w semantic ||w character ||w pos , the symbol “||” indicates the splicing relationship.

[0062] Taking the graph attention network as an example, the sentence heterogeneous graph is input into each layer of the multi-layer stacked graph neural network encoder. In each layer, the graph attention network (GAT) encodes the word order and dependency information in the sentence heterogeneous graph into the initial word vector of each word, and the word vector of this layer is obtained after encoding.

[0063] A single-layer graph attention network aggregates information about a node’s first-order neighbors into the node: in represents the word vector of the "l"th layer, Represents the neighbor node vector of node i at the “l-1” layer index, αij The calculation method is: Among them, e ij Represents the attention of node i to node j, calculated as: e ij =a(Wh i ,W(h j ||r j ), where r j Indicates h i and h j The type of connected edges, a and W, are one-dimensional and high-dimensional linear mappings respectively. The encoded word vector is further enhanced through the ReLU nonlinear activation function to improve its representation ability.

[0064] In step S3, the initial word vector of the word is input into the first layer of the graph neural network of the multi-layer stacked graph neural network encoder; the sentence heterogeneous graph is input into each layer of the graph neural network of the multi-layer stacked graph neural network encoder; the conditional random field-based decoder decodes the final word vector output by the multi-layer stacked graph neural network encoder, and extracts the named entity according to the decoding result.

[0065] In this step, the receptive field of the model is improved and the information of multi-hop neighbors of nodes in the directed graph is extracted by stacking multiple layers of graph attention networks. Unlike the direct stacking of existing methods, this invention adopts a method similar to Resnet residual connection to reduce the over-smoothing phenomenon caused by the stacking of multiple layers of the model. Specifically, the output of each layer of the graph attention network is directly connected to the output of the entire multi-layer graph neural network encoder. The final output of the model is: h = w||h 1 ||h 2 …||h n , where the symbol “||” represents the splicing relationship, n is the number of network layers of the multi-layer graph neural network encoder, and w is the initial word vector of the word.

[0066] In this step, a conditional random field-based decoder is used to decode the final word vector h obtained from the multi-layer graph neural network encoder, and the named entity label of the word is predicted based on the decoding result.

[0067] The named entity recognition model is pre-trained based on the annotated corpus, and the sentences are annotated in BIO format. The sentences are sequentially annotated in BIO format to express the position of the entity in the sentence. The sentence is sequentially annotated by manual annotation plus a standardized annotation platform for subsequent model training. The annotation adopts the standard BIO format, "B" indicates that the word is the beginning of the entity, "I" indicates that the word belongs to the entity, and "O" indicates that the word does not belong to any entity. For example, the annotation corresponding to "Tom, Cruise, played, the, protestant, in, Mission, Impossible" is "B-PER, I-PER, O, O, O, O, B-MOV, I-MOV". Among them, "PER" indicates that the category of the entity is "person" and "MOV" means "movie".

[0068] When training the model, Figure 3 As shown in the figure, the unlabeled sentences in actual application are replaced with the labeled sentences used for training. During training, the probability of a labeling method is: Where y is the BIO label predicted by the model, w k Represents the weight of the annotation path, f k (y,h) represents the conditional probability, z(h) is the normalization factor. The decoder optimizes the objective function argmax y p(y|h) is used to obtain the optimal entity labeling result. The parameters that can be optimized and adjusted during training mainly include sentence length and the number of graph neural network layers.

[0069] This paper, based on graph analysis, fuses the parts of speech, word order, and dependency relationships of words in a text into a sentence heterogeneous graph, and uses a graph attention network to extract entity information. This method utilizes the semantic information in the text more comprehensively and effectively than existing methods, thereby improving the effectiveness and accuracy of entity extraction. This paper provides a text representation method based on a directed graph and an entity extraction method based on a graph neural network.

[0070] Taking the graph attention network as an example of a graph neural network, when extracting entities based on the graph attention network, the semantic information and part of speech of the word are combined to construct the initial word vector representation model of the word; a multi-layer graph attention network encoder is used to learn the word order and dependency relationship from the sentence heterogeneous graph; a decoder based on conditional random fields determines the label of the word through the learned word vector.

[0071] In addition, an embodiment of the present invention also proposes a computer-readable storage medium, on which a processing program for named entity extraction based on a graph neural network is stored. When the processing program for named entity extraction based on a graph neural network is executed by a processor, the steps of the named entity extraction method based on a graph neural network are implemented.

[0072] It can be understood that the same or similar parts of the above embodiments can be referenced to each other, and the contents not described in detail in some embodiments can refer to the same or similar contents in other embodiments.

[0073] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention pertain.

[0074] It should be understood that each part of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system.

[0075] Those skilled in the art will understand that all or part of the steps in the method of the above embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0076] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing module, or each unit may exist physically separately, or two or more units may be integrated into a single module. The aforementioned integrated modules may be implemented in the form of hardware or in the form of software functional modules. If the integrated modules are implemented in the form of software functional modules and sold or used as independent products, they may also be stored in a computer-readable storage medium.

[0077] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0078] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0079] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A named entity extraction method based on graph neural network, characterized in that: The method comprises: According to the results of sentence lexical analysis and dependency analysis, the sentence is converted into a sentence heterogeneous graph based on a directed graph with words as nodes; The initial word vector of the word is obtained by concatenating the semantic information vector, character information vector, and part-of-speech information vector of the word in the sentence; Inputting the sentence heterogeneous graph and the initial word vectors into a named entity recognition model, and outputting the extracted named entities; The named entity recognition model includes a sequentially connected multi-layer stacked graph neural network encoder and a conditional random field decoder; The initial word vector of the word is input into the first layer of the graph neural network of the multi-layer stacked graph neural network encoder; the sentence heterogeneous graph is input into each layer of the graph neural network of the multi-layer stacked graph neural network encoder; The decoder based on conditional random fields decodes the final word vector output by the multi-layer stacked graph neural network encoder and extracts the named entity based on the decoding result; The sentence heterogeneous graph is input into each layer of the graph neural network of the multi-layer stacked graph neural network encoder. Each single-layer graph neural network aggregates the information of the node neighbors in the sentence heterogeneous graph into the node, and encodes the word order information and dependency relationship information between the nodes in the sentence heterogeneous graph into the word vector corresponding to the node; The graph neural network is a graph attention network. The single-layer graph attention network uses the word vector after sentence heterogeneous graph encoding to express it as: , in, Representative l " layer's word vectors, represent" l -1" layer index is The node's neighbor node vector; in, , represent Node Pair Node attention; in, , Represents two node word vectors and The type of edge connecting the two, and are one-dimensional and high-dimensional linear mappings respectively; The sentence isomerism graph includes nodes corresponding to words in the sentence, wherein the node attribute is the word part of speech; and also includes directed edges representing dependency relationships between nodes and / or directed edges representing word order relationships between nodes.

2. A named entity extraction method based on graph neural network according to claim 1, characterized in that: In the step of obtaining the initial word vector of a word by splicing the semantic information vector, character information vector, and part-of-speech information vector of the word in the sentence: The semantic information vector is obtained by: obtaining the semantic information vector of the word from the pre-trained language model, and using a random vector as the semantic information vector when the word does not exist in the pre-trained language model; The character information vector is obtained by encoding the word through an LSTM recurrent neural network and using the output of the last state of the LSTM as the character information vector; The part-of-speech information vector is determined by setting a random initial vector for each part-of-speech.

3. The method for named entity extraction based on graph neural network according to claim 1, characterized in that: The specific stacking method of the multi-layer stacked graph neural network encoder is that the output of each layer of the graph neural network is directly connected to the output end of the multi-layer stacked graph neural network encoder; The final output of the multi-layer stacked graph neural network encoder is represented as the concatenation of the outputs of each layer of the graph neural network: ， Among them, the symbol "||" represents the splicing relationship. is the number of layers of the multi-layer stacked graph neural network encoder, and w is the initial word vector of the word.

4. A device using the named entity extraction method based on graph neural network according to claim 1, characterized in that: The device comprises: The heterogeneous graph module is used to convert sentences into a sentence heterogeneous graph based on a directed graph with words as nodes based on the results of sentence lexical analysis and dependency analysis; The initial vector module is used to obtain the initial word vector of a word by splicing the semantic information vector, character information vector, and part-of-speech information vector of the word in the sentence; Named entity recognition model, which is used to extract named entities based on the input word initial word vector and sentence heterogeneous graph output; The named entity recognition model includes a multi-layer graph neural network encoder and a conditional random field decoder connected in sequence; the initial word vector of the word is input into the multi-layer stacked graph neural network encoder; the sentence heterogeneous graph is input into each layer of the graph neural network of the multi-layer stacked graph neural network encoder; The conditional random field-based decoder decodes the final word vector output by the multi-layer stacked graph neural network encoder and extracts the named entity based on the decoding result.

5. A storage medium, characterized in that: The storage medium stores a processing program for named entity extraction based on a graph neural network, and when the processing program for named entity extraction based on a graph neural network is executed by a processor, the steps of the named entity extraction method based on a graph neural network as described in any one of claims 1 to 3 are implemented.

Images