Method and system for jointly extracting entity and relation information in text

Abstract

The invention discloses a method and system for jointly extracting entity and relation information in a text, and the method comprises the steps: inputting a text sentence, segmenting the sentence into lexical elements, and obtaining a lexical element sequence; inputting the lexical element sequence into an ALBERT model to generate a first output vector; inputting the lexical element sequence into a TKGTransE knowledge graph embedding model to generate a second output vector; inputting the first output vector and the second output vector into a GFN gating fusion network to generate a first vector representation; and inputting the first vector representation and the lexical element sequence into a non-autoregression decoder to generate a prediction triple, and evaluating a prediction result by adopting a bipartite matching loss function to generate a prediction vector. According to the method, knowledge fusion is realized, and the processing capability of the model on a complex text structure is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a method and system for jointly extracting entity and relationship information from text. Background Art

[0002] With technological advancements and rising living standards, people are shifting their access to information from traditional brick-and-mortar stores to the internet, where information is diverse and rapidly updated. Numerous scholars are diligently exploring methods for extracting high-quality, structured information from diverse and complex data, with knowledge graphs becoming a hot topic. By structuring large amounts of data, knowledge graphs can be constructed, efficiently and accurately extracting information for applications in personalized recommendations, machine question answering, and decision support, meeting user needs. A common approach is to utilize knowledge extraction techniques to construct knowledge graphs from complex datasets. With the rise of deep learning as a key milestone, the development of knowledge extraction can be divided into two phases: traditional and deep learning.

[0003] Traditionally, the knowledge extraction process primarily involves data preprocessing (cleaning, word segmentation, part-of-speech tagging, etc.); entity recognition: identifying entities with specific meanings from text, such as names of people, places, and organizations, through rule-based or dictionary matching methods; relationship definition: identifying the types of relationships to be extracted, such as "father-child relationship" and "employment relationship," and developing corresponding rules and models; relationship extraction: extracting relationships between entities from text based on the defined relationships using techniques such as pattern matching and syntactic analysis; and post-processing: evaluating, correcting, and optimizing the extracted results, such as removing duplicate or erroneous extraction results, to improve the accuracy and completeness of the extraction. Traditional knowledge extraction processes primarily rely on rule-based and feature engineering methods. Researchers manually define rules and extract features from text to identify knowledge elements such as entities, relationships, and attributes. With the rise of deep learning, researchers have used deep neural networks to automatically learn feature representations for text, rather than relying on manually designed features. These networks are capable of capturing more complex and abstract semantic and grammatical information. Alternatively, we can leverage language models pre-trained on large-scale general-purpose corpora (such as BERT and GPT). By collecting large amounts of data, we can divide the processing process into training and prediction phases, and fine-tune them on the collected datasets for specific knowledge extraction tasks to suit the specific task. Since the introduction of deep learning, some end-to-end knowledge extraction model algorithms have achieved great success.

[0004] Traditional knowledge extraction processes manually define rules and extract text features to identify knowledge elements such as entities, relationships, and attributes. This traditional knowledge extraction process has several drawbacks: First, it relies heavily on manual effort: rule development, feature engineering, and relationship definition require extensive manual intervention, consuming significant time and effort, and are difficult to adapt to new domains and data. Second, it suffers from limited generalization: Manually designed rules and features are often targeted at specific domains and datasets. When applied to new and diverse texts, they generalize poorly and are prone to inaccurate or missed extractions. Following the rise of deep learning, some researchers have separated the knowledge extraction task into two independent parts: named entity recognition and relationship extraction. This approach ignores the inherent connection and mutual influence between the two tasks. Entity recognition results may affect the accuracy of relationship extraction, while relationship extraction results may in turn provide additional clues and constraints for entity recognition. Separate tasks may fail to fully exploit this interaction. This separation into two independent tasks is often referred to as a pipeline model. In this pipeline model, entity recognition is typically performed first, followed by relationship extraction based on the identified entities. If an error occurs in the entity recognition stage, the relationship extraction stage will operate based on the wrong entity, resulting in an error propagation problem. The information obtained in the entity recognition stage cannot be directly fully utilized by the relationship extraction stage, and vice versa, resulting in a problem of information non-sharing. In pipeline mode, the model usually runs two tasks, which is computationally intensive and inefficient. These two serial steps take more time than jointly extracting entities and relationships, especially when the amount of data is huge. This efficiency problem is more obvious. The current mainstream model is to jointly extract entities and relationships from text. A joint extraction algorithm, SPN4RE (joint entity and relationship extraction using a set prediction network), has the advantages of fast decoding speed and no need to predefine entity relationship sets, but has some disadvantages: insufficient modeling of dependency relationships, limited long-distance dependency processing capabilities, high computational complexity, and no incorporation of prior knowledge. There is room for further improvement. Summary of the Invention

[0005] In order to overcome the above-mentioned shortcomings and deficiencies of the prior art, an object of the present invention is to provide a method and system for jointly extracting entity and relationship information from text.

[0006] The purpose of the present invention is achieved through the following technical solutions:

[0007] A method for jointly extracting entity and relationship information from text, comprising the following steps:

[0008] Input a text sentence, split the sentence into tokens, and obtain a token sequence;

[0009] Input the word sequence into the ALBERT model to generate the first output vector;

[0010] Input the word sequence into the TKG_TransE knowledge graph embedding model to generate the second output vector;

[0011] Inputting the first output vector and the second output vector into the GFN gated fusion network to generate a first vector representation;

[0012] The first vector representation and word sequence are input into the non-autoregressive decoder to generate a prediction triplet. The prediction result is evaluated using the bipartite matching loss function to generate a prediction vector.

[0013] Furthermore, the word sequence is input into the ALBERT model to generate a first output vector, specifically by inputting the word sequence into a multi-head self-attention mechanism and a feedforward neural network connected in sequence to generate the first output vector.

[0014] Furthermore, the word sequence is input into the ALBERT model to generate the first output vector, specifically:

[0015] Each word is mapped to the corresponding word index according to the ALBERT model's vocabulary;

[0016] The word index input embedding layer is converted into a low-dimensional embedding vector;

[0017] The embedding vector is input into an encoder layer to obtain a first output vector, where the encoder layer includes multiple identical sub-layers, each of which includes a multi-head self-attention layer and a feedforward neural network layer connected in sequence.

[0018] Furthermore, the first output vector and the second output vector are input into the GFN gated fusion network to generate the first vector representation, specifically:

[0019] The first output vector is the embedding vector sequence of the word in the text sentence; the second output vector is the embedding vector sequence of the entity and relationship in the knowledge graph;

[0020] Performing a linear transformation on the first output vector to obtain a gating vector for controlling the weight of text information in the fusion process;

[0021] Perform a linear transformation on the second output vector to obtain a gating vector used to control the weight of the knowledge graph information in the fusion process;

[0022] The text information and knowledge graph information are weightedly summed according to the gated vector obtained above to generate a first vector representation.

[0023] Furthermore, the non-autoregressive decoder is composed of N identical Transformer modules stacked together, each Transformer module including a multi-head self-attention sublayer, a multi-head cross-attention sublayer, a global attention sublayer and a feedforward neural network.

[0024] Furthermore, the step of inputting the word sequence into the TKG_TransE knowledge graph embedding model to generate a second output vector is to convert the word sequence into triples and map them into a low-dimensional vector space, including the following steps:

[0025] Filter meaningless words in the word segmentation sequence, merge consecutive nouns into entities based on part-of-speech tagging and pattern matching, and use regular expressions to locate entities;

[0026] In relation extraction, verb / preposition location is performed first. A set of verbs and prepositions is predefined, verbs or prepositions are filtered as relation candidates, and the subject-verb-object structure is extracted through SpaCy's dependency parsing.

[0027] Combine the nearest relation words of adjacent entities, allow only one-way relations, generate a triple list, remove low-frequency triples, merge duplicate triples, use the TKG_TransE loss function to dynamically adjust, and output the initial triple list.

[0028] Furthermore, the first vector representation and the word sequence are input into a non-autoregressive decoder to generate a prediction triplet, specifically:

[0029] The word sequence is input into the multi-head self-attention sub-layer to obtain the embedding vector b1;

[0030] Input the embedding vector b1 and the first vector representation into the multi-head cross attention sub-layer to obtain the embedding vector b2;

[0031] The embedding vector b2 is input into the global attention mechanism sublayer to obtain the embedding vector b3;

[0032] The embedding vector b3 is input into the feedforward neural network to obtain the predicted triplet.

[0033] Furthermore, in the query process of the multi-head cross attention sub-layer, the key and value come from the output of the GFN gated fusion network, and the query comes from the initialized query vector. The purpose is to incorporate the contextual information of the text sentence into the triple query representation of the non-autoregressive decoder, and make full use of the semantic and grammatical information of the input sentence.

[0034] A system for implementing the method, comprising:

[0035] A sentence encoder, comprising an ALBERT model, a TKG_TransE knowledge graph embedding model, and a GFN gated fusion network;

[0036] The non-autoregressive decoder consists of N identical Transformer modules stacked together. Each Transformer module includes a multi-head self-attention sublayer, a multi-head cross-attention sublayer, a global attention sublayer, and a feedforward neural network to obtain a prediction vector.

[0037] A storage medium stores a program, characterized in that when the program is executed by a processor, it implements the method of jointly extracting entity and relationship information in text.

[0038] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0039] This method introduces the TransE module, which maps entities and relationships in the knowledge graph into a low-dimensional vector space, thereby representing knowledge in a structured manner. In knowledge extraction tasks, this structured knowledge can provide the model with rich prior information. For example, for a text involving multiple entities and complex relationships, the knowledge graph vector representation provided by TransE can help the model understand the potential relationships between entities. These relationships may not be directly expressed in the text, but there are clear connections in the knowledge graph. This helps the model better capture semantic relationships when processing text and improves the accuracy of entity relationship extraction.

[0040] By incorporating information from the knowledge graph, the model can understand the semantics of text from multiple perspectives. Combining entity attributes, relationship types, and other information in the knowledge graph with the contextual information in the text enables the model to gain a more comprehensive and in-depth understanding of the text's semantics. For example, when dealing with ambiguous entities, the entity's classification or related relationship information in the knowledge graph can help the model determine its precise meaning based on the context, thereby improving the accuracy of semantic understanding and, in turn, the quality of knowledge extraction.

[0041] When processing long texts, the problem of long-distance dependencies is a challenge. The knowledge graph information carried by TransE can provide a global information association across long distances. Even if two entities in the text are far apart in position, if they have a clear relationship in the knowledge graph, the model can capture this relationship with the help of TransE's vector representation, thereby alleviating the impact of long-distance dependencies on entity relationship extraction to a certain extent. For example, in a long article discussing historical events, related people and events mentioned in different paragraphs may be far apart in the text, but through the pre-defined relationships in the knowledge graph, the model can better establish the connection between them and improve its ability to handle complex semantic relationships in long texts.

[0042] This method introduces the ALBERT module, which uses cross-layer parameter sharing technology. Unlike traditional Transformer models (such as BERT), it shares some or all parameters between different layers. This means that the number of parameters that need to be updated and stored during model training and inference is significantly reduced. For example, when processing large-scale text data, the computational complexity is effectively reduced, thereby speeding up the model training speed and improving computational efficiency, allowing the model to maintain good performance even with limited resources.

[0043] Cross-layer parameter sharing helps stabilize the network parameter update process. During model training, the stability of parameter updates is crucial for convergence speed and ultimate performance. ALBERT's parameter sharing mechanism ensures relatively smooth parameter changes during training, reducing training non-convergence and performance fluctuations caused by large or unstable parameter updates. This helps the model reach optimal performance more quickly and demonstrates more stable performance across different datasets and tasks.

[0044] This method introduces the GFN module to fuse information from different sources. In this model, it mainly fuses the text information output by the text encoder (such as ALBERT) and the knowledge graph information output by the knowledge graph embedding model (such as TransE). Text information contains rich details such as semantics, grammar, and contextual relationships in the sentence, while the knowledge graph information provides structured entity and relationship knowledge. Through GFN, the model can comprehensively utilize these two information sources to make predictions more accurate and comprehensive. For example, when processing a sentence containing an entity mention, the text information can provide information about the specific context and role of the entity in the sentence, while the knowledge graph information can supplement the relationship and attributes of the entity in a broader knowledge system. The fusion of the two can provide a richer feature representation for entity relationship extraction.

[0045] The introduction of the gating mechanism makes the fusion process dynamic. The gating vector (such as the text gating vector g t and the knowledge graph gate vector g k ) uses learning to control the importance of different information sources in different situations. In some scenarios, if the text describes an entity or relationship very clearly and in detail, the text gating vector may assign a higher weight to the textual information, causing the model to rely more heavily on it for predictions. In other cases, when the knowledge graph contains prior knowledge that is highly relevant to the task at hand, the knowledge graph gating vector is adjusted accordingly, highlighting the role of the knowledge graph information. This dynamic fusion mechanism adaptively adjusts the information fusion strategy based on the characteristics of the input data, improving the model's adaptability and processing capabilities for different types of data.

[0046] This method introduces a Global Attention module. When processing long texts, entities and relations in the text may be far apart, and traditional local attention mechanisms may not be able to effectively capture the relationship between them. Global Attention can perform a weighted summation of all positions in the encoder output sequence, thereby providing the decoder with a representation that contains global information. This allows the model to consider information from the entire input sequence when generating entity and relationship predictions, and establish semantic associations between entities and relations across long distances. For example, in an article discussing complex scientific theories, an entity is mentioned at the beginning, while the related relationship is explained at the end. Global Attention can help the model comprehensively consider information from the beginning and end of the article when predicting the entity relationship, avoiding missing important related information due to the long distance. This improves the ability to handle long-distance dependencies and reduces omissions or errors in entity relationship extraction from long texts.

[0047] By incorporating global information into the decoder's input representation, Global Attention helps the model understand the semantics of the text from a more macroscopic perspective. This allows the model to go beyond local contextual information and comprehensively consider the semantic context of the entire text, thereby improving its ability to understand complex semantic relationships within the text. When dealing with polysemy or semantic ambiguity, incorporating global information can help the model more accurately determine the meaning of entities and relationships, thereby improving the quality and accuracy of knowledge extraction and providing the model with more comprehensive and in-depth semantic understanding support in knowledge extraction tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a schematic diagram of the information flow of the module of the present invention;

[0049] Figure 2 It is a schematic diagram of the module structure of the present invention;

[0050] Figure 3 It is a workflow diagram of the present invention. DETAILED DESCRIPTION

[0051] The present invention will be further described in detail below with reference to the examples, but the embodiments of the present invention are not limited thereto.

[0052] Example

[0053] like Figure 1-Figure 3 As shown, a system for jointly extracting entity and relationship information from text includes a sentence encoder, which includes an ALBERT model, a TKG_TransE knowledge graph embedding model, and a GFN gated fusion network;

[0054] The non-autoregressive decoder consists of N identical Transformer modules stacked together. Each Transformer module includes a multi-head self-attention sublayer, a multi-head cross-attention sublayer, a global attention sublayer, and a feedforward neural network to obtain a prediction vector.

[0055] The present invention improves the long-distance dependency processing capability, reduces the model calculation amount by parameter sharing, realizes knowledge fusion between TKG_TransE and GFN, and improves the model's processing capability for complex text structures.

[0056] The method implemented by the system for jointly extracting entity and relationship information from text includes the following:

[0057] The input text sentence is segmented through Byte Pair Encoding, and the sentence is divided into tokens to obtain a token sequence.

[0058] Specific explanation: Byte pair encoding is a data compression algorithm that is also commonly used for text segmentation in natural language processing. Its basic idea is to iteratively merge the most frequently occurring byte pairs to build a finite vocabulary. During the word segmentation process, the frequency of byte pairs (character pairs) in the text is counted, and then the byte pairs with the highest frequency are selected and merged to form new "characters", and the statistical information of byte pairs in the text is updated. This process is repeated until the preset vocabulary size is reached or other stopping conditions are met. In this way, BPE can segment the text into a series of tokens, which can be single characters, common character combinations, or new symbols formed by merging, thereby effectively handling the problem of out-of-vocabulary words and generating appropriate token representations based on the characteristics of the text data.

[0059] First, initialize the vocabulary, treat each character in the text as a separate token, and count the occurrences of all byte pairs (character pairs). For example, for the input sentence "John visited the Eiffel Tower in Paris.", the initial vocabulary is {"J","o","h","n","","","v","i","s","t","e","d","t","h","e","E","i","f","f","e","l","T","o","w","e","r","i","n","P","a","r","i","s","."}, and count the occurrences of byte pairs such as ("J","o"), ("o","h"), etc.

[0060] Then, the byte pairs with the highest frequency are selected for merging. Assuming that in this example, the byte pair consisting of "e" and "l" appears more frequently, they are merged into a new token "el" and the statistics of byte pairs in the vocabulary and text are updated. At this time, the vocabulary becomes {"J", "o", "h", "n", "", "v", "i", "s", "t", "ed", "th", "e", "E", "i", "f", "fe", "l", "T", "o", "w", "er", "i", "n", "P", "a", "r", "is", "."}, and the text becomes "John visited the Eiffel Tower in Paris."

[0061] Repeat the above merging steps, continuously updating the vocabulary and text, until a preset stopping condition is reached, resulting in a word sequence. For example, the vocabulary size reaches a certain number or there are no more byte pairs that can be merged (the frequency is below a certain threshold). The final word segmentation result may be ["John","visited","the","Eiffel","Tower","in","Paris","."] (the word segmentation results here are only examples; the actual word segmentation may vary depending on the specific BPE implementation and parameter settings).

[0062] Input the word sequence into the ALBERT model to generate the first output vector.

[0063] The specific generation process is as follows:

[0064] First, each word in the word sequence is mapped to a corresponding word index (tokenindices) according to the ALBERT model's vocabulary. During the pre-training process, the ALBERT model builds a vocabulary containing a large number of common words and symbols. By looking up the vocabulary, the input word is converted into a corresponding integer index. For example, if the index of "John" in the ALBERT vocabulary is 100, the index of "visited" is 200, and so on, then the sentence after token segmentation ["John","visited","the","Eiffel","Tower","in","Paris","."] will be converted into the corresponding index sequence [100,200,300,400,500,600,700,800] (the index values here are only examples). At the same time, we need to add some special markers, such as [CLS] (indicating the beginning of a sentence) and [SEP] (indicating the end of a sentence), and change the index sequence to [101, 100, 200, 300, 400, 500, 600, 700, 800, 102] (assuming that the index of [CLS] is 101 and the index of [SEP] is 102).

[0065] The embedding layer of the ALBERT model converts these token indexes into low-dimensional embedding vectors. The embedding layer contains token embeddings, position embeddings, and segment embeddings. Token embeddings maps each token index to a vector of a fixed dimension (such as 768 dimensions), which represents the semantic and grammatical features of the token; position embeddings are used to encode the position information of the token in the sentence, so that the model can distinguish tokens in different positions; segment embeddings are used to distinguish different sentence fragments (for example, when processing text composed of multiple sentences). In the case of single sentence input, segment embeddings can represent the unique identifier of the sentence. These three embeddings are combined by element-wise addition to form the final input embedding vector. For example, for the token "John", its token embedding, position embedding (assuming the position in the sentence is 1) and segment embedding (assuming it is a single sentence and the segment identifier is 0) are added together to obtain a 768-dimensional vector a 1,1 (Here a 1,1 represents the embedding vector corresponding to the first word "John" in the input sentence). Similarly, other words in the sentence will also get their corresponding embedding vectors a. 1,2 ,a 1,3 ,...,a 1,n (n is the number of words in the sentence), the embedding vector of the entire sentence is represented by a1=[a 1,1 ,a 1,2 ,a 1,3 ,…,a 1,n ].

[0066] The low-dimensional embedding vector is input into the encoder layer of the ALBERT model, producing the first output vector. The encoder layer consists of multiple identical sublayers, each of which includes a multi-head self-attention layer and a feed-forward neural network layer. Each sublayer is preceded and followed by residual connections and layer normalization. In the multi-head self-attention layer, each head calculates attention scores between tokens, focusing on information from different parts of the sentence. New token representations are then generated through weighted summation and other operations. This process enables the model to capture semantic relationships and contextual information between tokens. The token representations processed by the multi-head self-attention layer are then further transformed through the feed-forward neural network layer to enhance the model's semantic understanding. After processing through multiple encoder layers, each token is ultimately encoded by the entire model, resulting in a representation vector. These vectors contain rich semantic information about the sentence context and are used for further computation and processing in subsequent model components (such as the non-autoregressive decoder) to achieve tasks such as knowledge extraction. For example, after being processed by all encoder layers, the vector corresponding to the word "John" may incorporate information from other words in the entire sentence, such as its semantic associations with words such as "visited" and "Eiffel Tower," thereby more comprehensively representing its semantic role and contextual relationship in the sentence.

[0067] Input the word sequence into the TKG_TransE knowledge graph embedding model to generate the second output vector, which is mainly used to convert the word sequence into triples and map them to a low-dimensional vector space;

[0068] The specific generation process is as follows:

[0069] Input the tokenized sequence of word tokens (e.g., ["Beijing", "is", "the", "capital", "of", "China"]) into the preprocessing stage. First, predefine a set of meaningless stop words (e.g., "of", "is"), and retain entity and relationship candidate words (e.g., "Beijing", "capital"). Then, perform chunk merging. Based on part-of-speech tagging (POS) and pattern matching (such as noun phrase merging rules), merge consecutive nouns into entities (e.g., ["China", "capital"] -> ["China capital"], where POS filters and marks nouns (NOUN), proper nouns (PROPN), etc. as entity candidates, and use regular expressions to locate entities (such as the noun after the "of" structure as the tail entity). In relation extraction, first perform verb / preposition location. Predefine a set of verbs and prepositions, filter verbs (e.g., "is located in") or prepositions (e.g., "of") as relationship candidates, and extract the subject-verb-object structure through the dependency parsing of SpaCy (e.g., "Beijing - capital - China"). Combine adjacent entity pairs with the nearest relationship word (e.g., pair the relationship "is" between "Beijing" and "China capital"), and only allow one-way relationships (e.g., capital only applies to "Beijing -> China", the reverse is invalid), and output a preliminary list of triples.

[0070] In this vector space, for a correct triple (h, r, t) (head entity h, relationship r, tail entity t), it satisfies the relationship h + r ≈ t. By minimizing a margin-based loss function (such as ∑ (h,r,t) ∈S(h′,r′,t′)∈S′[γ + f(h, r, t) - f(h′, r′, t′)] + ,

[0071] where S is the set of positive sample triples, S′ is the set of negative sample triples, γ is the margin parameter, [x] + = max(0, x)), the model learns the vector representations of entities and relationships, making the scores of positive sample triples lower (satisfying the above approximate relationship), while the scores of negative sample triples are higher (negative samples are generated by randomly replacing the head entity, tail entity, or relationship).

[0072] At the start of training, randomly initialize a low-dimensional vector for each entity and relationship (e.g., in a d-dimensional vector space, d is usually a small integer, such as 50 or 100). Assume the initial vector of the entity "John" is e1 = (e 1,1 , e 1,2 , …, e 1,d ), the initial vector of "Eiffel Tower" is e2 = (e 2,1 , e 2,2 , …, e 2,d ), and the initial vector of the relationship "visited" is r3 = (r3,1 ,r 3,2 ,…,r 3,d )

[0073] For the input triple (1,3,2) (corresponding to "John", "visited", "Eiffel Tower"), calculate the value of h+rt, that is, e1+r3-e2=(e 1,1 +r 3,1 -e 2,1 ,e 1,2 +r 3,2 -e 2,2 ,…,e 1,d +r 3,d -e 2,d ). Then calculate the norm of this vector (such as L2 norm) This norm is the score of the triple in the TKG_TransE model.

[0074] Based on the aforementioned loss function, the loss of the positive triple is calculated, and the entity and relationship vectors are updated using the backpropagation algorithm. For example, if the current triple is a positive sample, the model calculates its loss value according to the loss function. The vector values of e1, r3, and e2 are then adjusted using the gradient descent algorithm to lower the score of the positive sample (closer to the ideal h + r ≈ t relationship). During training, the model iterates over a large number of positive and negative sample triplets, continuously adjusting the vector representation until convergence.

[0075] After sufficient training, the vector representation of entities and relations is the second output vector we need. Assume that the embedding vector of "John" after training is a 2,1 =(a 2,1,1 ,a 2,1,2 ,…,a 2,1,d ), the embedding vector of “Eiffel Tower” is a 2,2 =(a 2,2,1 ,a 2,2,2 ,…,a 2,2,d ), the embedding vector of “visited” is a2,3=(a 2,3,1 ,a 2,3,2 ,…,a 2,3,d ), then the embedding vector of entities and relations in the sentence a2=[a 2,1 ,a 2,3 ,a 2,2] (arranged in the order of entity-relationship-entity). These embedding vectors contain the semantic representation of entities and relationships in a low-dimensional vector space, which can capture the structure and semantic relationship between them. When subsequently fused with other information, they provide the model with structured knowledge in the knowledge graph, enhancing the model's ability to understand and extract entities and relationships in the text.

[0076] The first output vector and the second output vector are input into the GFN gated fusion network to generate a first vector representation.

[0077] The specific process is:

[0078] The first output vector is the embedding vector sequence of the word in the text sentence; the second output vector is the embedding vector sequence of the entity and relationship in the knowledge graph;

[0079] The GFN gated fusion network dynamically determines the importance of each information source by learning gating vectors. For the fusion of text and knowledge graph information, two gating vectors are typically used: a text gating vector and a knowledge graph gating vector. Calculating the gating vectors typically involves linear transformation and nonlinear activation of the corresponding information. The fusion process then performs a weighted summation of the text and knowledge graph information based on the gating vectors.

[0080] The GFN gated fusion network receives the first output vector from the ALBERT model, i.e., the text information representation a1, and the second output vector from the knowledge graph embedding model TKG_TransE, i.e., the knowledge graph information representation a2. Assume that a1 is the embedding vector sequence [a 1,1 ,a 1,2 ,…,a 1,n ] (n is the number of words in the text), a2 is the embedding vector sequence of related entities and relations in the knowledge graph (e.g. for a triple containing two entities and one relation, a2 = [a 2,1 ,a 2,3 ,a 2,2 ], arranged in the order of entity relationship entities).

[0081] Perform linear transformation on text information a1: h t =a1W t , where W t is a trainable weight matrix. For example, if the shape of a1 is n×d1 (n is the number of tokens, d1 is the text embedding dimension), W t The shape of h is d1×d (d is the desired transformed dimension, usually consistent with other vector dimensions in the model), then h t The shape is n×d. The gate vector is obtained by a nonlinear activation function (such as the sigmoid function): t =σ(ht ),in g t The form of h t The same as n × d. This gating vector will be used to control the importance of text information in the fusion process.

[0082] Perform linear transformation on the knowledge graph information a2: h k =a2W k , where W k Is a trainable weight matrix. Assume that the shape of a2 is m×d2 (m is the number of related entities and relations in the knowledge graph, d2 is the embedding degree of the knowledge graph), W k The shape is d2×d ,则 h k The shape of is m×d. Then the gate direction g is obtained through nonlinear activation k =σ(h k ),g k The shape and h k The same as m × d. This gating vector determines the weight of the knowledge graph information in the fusion.

[0083] In order to perform element-wise multiplication, the gating vector g may need to be further adjusted. t 、g k and the dimensions of information a1, a2. For example, if g t The shape of a1 is n×d, and the shape of g is n×d1. t Expand to n × d1 (by copying or other means) so that it matches a1 in dimension, and for a2 and g k Similar dimension resizing operations are also performed to ensure that they can be multiplied element-wise.

[0084] The text information and knowledge graph information are weighted and summed according to the calculated gate vector. The fused information F is calculated as follows: F = g t ⊙a1+g k ⊙a2, where Indicates element-wise multiplication. g t The shape of a1 is the same as that of the adjusted a1, g k The shape of ⊙a2 is the same as the adjusted a2, and then the two results are added together to obtain the fused information F. The shape of F depends on the specific calculation and adjustment method, but it combines the information of text and knowledge graph, providing a richer and more comprehensive input representation for subsequent model components.

[0085] The first vector representation and word sequence are input into the non-autoregressive decoder to generate a prediction triplet. The prediction result is evaluated using the bipartite matching loss function to generate a prediction vector.

[0086] The non-autoregressive decoder includes N identical Transformer modules stacked together, each Transformer module includes a multi-head self-attention sublayer, a multi-head cross-attention sublayer, a global attention sublayer and a feedforward neural network.

[0087] The multi-head self-attention mechanism is one of the core components of the Transformer architecture. It allows the model to focus on different parts of the input sequence simultaneously when processing it, thereby capturing richer semantic and grammatical relationships. The core idea is to calculate the relevance score (attention score) between the token at each position and all other tokens in the sequence, and then perform a weighted sum of the information of all tokens based on these scores to update the token representation at each position.

[0088] These word sequences are input into the multi-head self-attention mechanism sub-layer to obtain the embedding vector b1.

[0089] Input the embedding vector b1 and the initialized triplet query vector (first vector representation) into the Multi-Head Cross-Attention sublayer to obtain the embedding vector b2, and input the embedding vector b2 into the Global Attention sublayer to output the embedding vector b3

[0090] Further explanation:

[0091] The multi-head cross-attention sublayer is similar to the multi-head self-attention sublayer. Its query comes from the output of the previous multi-head cross-attention sublayer, while the key and value come from the output of the sentence encoder. Its purpose is to incorporate the contextual information provided by the sentence encoder into the triple query representation of the decoder, allowing the decoder to fully utilize the semantic and grammatical information of the input sentence when generating triples.

[0092] Assume that the output of the previous multi-head cross attention sub-layer is Q prev , whose shape is m×d (m is the number of triplets, d is the number of hidden units). p rev performs linear transformation to obtain the query vector Q″=Q prev W′ Q (W′ Q is a trainable weight matrix with a shape of d×d k ), the shape is m×d k At the same time, the key vector K″=HW is obtained from the output H of the sentence encoder (ALBERT) (assuming the shape is n×d, n is the sentence length) K′和值向量V″ =HW V′ (W K′ 、W v′is a trainable weight matrix with a shape of d×d k ), the shape is n×d k .

[0093] For each head i, calculate the attention score Then perform a softmax operation to get the attention distribution The computation here is similar to the multi-head self-attention sublayer, but the key and value vectors come from the sentence encoder output, so the relevance of the triple query to each token in the sentence is computed.

[0094] According to the attention distribution α′ ij Perform weighted summation on the value vector V″ to obtain the output of each head The shape is m×d k Finally, the outputs of all heads are concatenated and passed through a linear transformation (weight matrix W O′ , the shape is h×d k ×d) to obtain the final multi-head cross attention sub-layer output O′=Concat(O′1,O′2,…,O′ h )W o′ , the shape is m×d.

[0095] The output of the Global Attention mechanism is a representation of the weighted sum of all positions in the input sequence, which provides the decoder with a representation containing global information to help the decoder better consider the information of the entire input sentence when generating triples.

[0096] Global Attention receives the output H from Multl-Head Cross-Attention and initializes a global attention weight vector a (shape n) to calculate the attention weight for each position of the encoder output sequence.

[0097] The embedding vector b3 is passed through the Feed Forward Network (FFN). FFN converts the information processed by the attention mechanism into specific prediction results, which is an important component of the decoder to generate accurate triples.

[0098] The embedding vector b3 is passed through the Feed Forward Network to generate a predicted triplet. The bipartite matching loss function is used to measure the difference between the predicted triplet and the true triplet and backpropagate the parameters.

[0099] FFN receives the output from the global attention sublayer as input. Assume that the input vector is x, and its shape is m×d (m is the number of triplets or other sequence length related to the model, and d is the number of hidden units).

[0100] First, the input vector x is linearly transformed by a trainable weight matrix W1 (shape d×d fj d ff is the dimension of the middle layer of the feedforward neural network, usually d ff Greater than d), the intermediate result h=xW1 is obtained, the shape is m×d ff This linear transformation can map the input vector from the original hidden unit space to a higher-dimensional space, thereby increasing the expressive power of the model and enabling the model to learn more complex feature representations.

[0101] A nonlinear activation function is applied to the intermediate result h. A common choice is the ReLU (Rectified Linear Unit) function, where h′ = ReLU(h). The ReLU function, defined as ReLU(x) = max(0, x), introduces nonlinearity, enabling the model to learn nonlinear relationships in the input data. Through the ReLU function, the model can process complex semantic and grammatical information, enhancing its ability to express input features. For example, for complex semantic relationships such as negation and transitions, the ReLU function can help the model capture these nonlinear features, thereby better understanding the semantics of the text.

[0102] After nonlinear activation, the result h′ is linearly transformed for the second time and then transformed by another trainable weight matrix W2 (shape d ff ×d), resulting in the final output y = h′W2, with a shape of m × d. This step maps the high-dimensional representation of the intermediate layer back to the original hidden unit dimensions, allowing interaction and information transfer with other components of the model. The second linear transformation adjusts the dimensionality of the feature representation to adapt to subsequent computational requirements, while further optimizing the feature representation and extracting key task-relevant information.

[0103] In the knowledge extraction task, the relation triplets in a sentence are essentially unordered sets, but the traditional autoregressive decoder outputs a sequence, which requires sorting the unordered target triplets. The cross entropy loss function is sensitive to the order of the prediction results and is not suitable for measuring the difference between unordered triple sets. Therefore, this paper proposes a bipartite matching function inspired by the operations research allocation problem.

[0104] Further explanation:

[0105] The bipartite matching loss plays a key role in model training, measuring the difference between predicted and true triplets. In knowledge extraction tasks, the traditional cross-entropy loss function is sensitive to the order of the predictions and is not well suited for measuring the difference between unordered sets of triplets. Therefore, a bipartite matching loss function was proposed. Inspired by the assignment problem in operations research, it generates an optimal bipartite match between predicted and true triplets, thus more reasonably evaluating the model's predictions.

[0106] set up represents the set of true triples, where each

[0107] r i Is the target relationship type (may be a special relationship type indicates no triples), are the starting or ending index of subject s and object o respectively. Represents a set of m triplets predicted by the model (m is usually greater than n), where each These predictions are calculated by the decoder of the model (e.g., a non-autoregressive decoder) (e.g., etc. to calculate the predicted probability of relation type and entity index).

[0108] First, define a pairwise matching cost function Used to measure the true triple Y i and predicted triples It is calculated as here is an indicator function, when 1 when , otherwise 0. This cost function takes into account the prediction accuracy of relation type and entity span. Then, by searching all possible permutations π∈Π(m) (Π(m) is the space of all permutations of length m), we find the total cost The smallest permutation π * ,Right now This optimal matching problem can be solved by the classic Hungarian Algorithm, whose time complexity is O(m 3 ).

[0109] After finding the optimal match, the loss function is calculated. The loss function is defined as This loss function is calculated for all matching pairs. It measures the difference between the predicted triple and the true triple by taking the logarithm of the predicted probability and adding an appropriate penalty term (when the relationship type is not empty, the predicted probability of the entity index is also taken and added). A larger loss value indicates a greater difference between the predicted result and the true situation. During the model training process, optimization algorithms (such as stochastic gradient descent) are used to minimize this loss function, thereby adjusting the model parameters to make the prediction closer to the true triple.

[0110] Unlike the traditional cross-entropy loss function, the bipartite matching loss function is insensitive to the order of predicted triples. In knowledge extraction tasks, the relation triplets in a sentence are inherently unordered. However, traditional autoregressive decoders need to sort the unordered target triplets to adapt to the output sequence. The cross-entropy loss function is sensitive to this ordering and imposes a large penalty for even small changes in the predicted order. The bipartite matching loss function, through the principle of optimal bipartite matching, can directly handle unordered sets of predicted and true triplets, ignoring the order of the triplets and focusing on the accuracy of relation types and entities, thus better meeting the essential characteristics of the task.

[0111] The WEBNLG dataset includes a training set of 5019 items, a validation set of 500 items, and a test set of 703 items. The test results are shown in Table 1.

[0112] Table 1 Experimental results of WEBNLG dataset

[0113]

[0114]

[0115] The NYT dataset includes a training set of 56,195 items, a validation set of 5,000 items, and a test set of 5,000 items. The test results are shown in Table 2.

[0116] Table 2 Experimental results of NYT dataset

[0117]

[0118] This method uses the same dataset as the original model, and the test model scores higher than the original model, indicating that the improvement is effective.

[0119] This embodiment also provides a storage medium, and when the program is executed by a processor, the method of jointly extracting entity and relationship information from text is implemented.

[0120] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A method for jointly extracting entity and relationship information from text, characterized in that: The steps include: Input a text sentence, split the sentence into tokens, and obtain a token sequence; Input the word sequence into the ALBERT model to generate the first output vector; Input the word sequence into the TKG_TransE knowledge graph embedding model to generate the second output vector; Inputting the first output vector and the second output vector into the GFN gated fusion network to generate a first vector representation; The first vector representation and word sequence are input into the non-autoregressive decoder to generate a prediction triplet. The prediction result is evaluated using the bipartite matching loss function to generate a prediction vector.

2. The method according to claim 1, characterized in that The step of inputting the word sequence into the ALBERT model to generate the first output vector is as follows: inputting the word sequence into the multi-head self-attention mechanism and the feedforward neural network connected in sequence to generate the first output vector.

3. The method according to claim 1, characterized in that Input the word sequence into the ALBERT model to generate the first output vector, specifically: Each word is mapped to the corresponding word index according to the ALBERT model's vocabulary; The word index input embedding layer is converted into a low-dimensional embedding vector; The embedding vector is input into an encoder layer to obtain a first output vector, where the encoder layer includes multiple identical sub-layers, each of which includes a multi-head self-attention layer and a feedforward neural network layer connected in sequence.

4. The method according to claim 1, wherein The first output vector and the second output vector are input into the GFN gated fusion network to generate the first vector representation, specifically: The first output vector is the embedding vector sequence of the word in the text sentence; the second output vector is the embedding vector sequence of the entity and relationship in the knowledge graph; Performing a linear transformation on the first output vector to obtain a gating vector for controlling the weight of text information in the fusion process; Perform a linear transformation on the second output vector to obtain a gating vector used to control the weight of the knowledge graph information in the fusion process; The text information and knowledge graph information are weightedly summed according to the gated vector obtained above to generate a first vector representation.

5. The method according to claim 1, wherein The non-autoregressive decoder is composed of N identical Transformer modules stacked together, each of which includes a multi-head self-attention sublayer, a multi-head cross-attention sublayer, a global attention sublayer and a feedforward neural network.

6. The method according to claim 1, characterized in that The step of inputting the word sequence into the TKG_TransE knowledge graph embedding model to generate the second output vector is to convert the word sequence into triples and map them into a low-dimensional vector space, including the following steps: Filter meaningless words in the word segmentation sequence, merge consecutive nouns into entities based on part-of-speech tagging and pattern matching, and use regular expressions to locate entities; In relation extraction, verb / preposition location is performed first. A set of verbs and prepositions is predefined, verbs or prepositions are filtered as relation candidates, and the subject-verb-object structure is extracted through SpaCy's dependency parsing. Combine the nearest relation words of adjacent entities, allow only one-way relations, generate a triple list, remove low-frequency triples, merge duplicate triples, use the TKG_TransE loss function to dynamically adjust, and output the initial triple list.

7. The method according to claim 5, characterized in that The first vector representation and the word sequence are input into the non-autoregressive decoder to generate the prediction triplet, specifically: The word sequence is input into the multi-head self-attention sub-layer to obtain the embedding vector b1; Input the embedding vector b1 and the first vector representation into the multi-head cross attention sub-layer to obtain the embedding vector b2; The embedding vector b2 is input into the global attention mechanism sublayer to obtain the embedding vector b3; The embedding vector b3 is input into the feedforward neural network to obtain the predicted triplet.

8. The method according to claim 5, characterized in that During the query process, the multi-head cross attention sub-layer uses the key and value from the output of the GFN gated fusion network, and the query comes from the initialized query vector. The purpose is to incorporate the contextual information of the text sentence into the triple query representation of the non-autoregressive decoder, and make full use of the semantic and grammatical information of the input sentence.

9. A system for implementing the method according to any one of claims 1 to 8, characterized in that: These include: A sentence encoder, comprising an ALBERT model, a TKG_TransE knowledge graph embedding model, and a GFN gated fusion network; The non-autoregressive decoder consists of N identical Transformer modules stacked together. Each Transformer module includes a multi-head self-attention sublayer, a multi-head cross-attention sublayer, a global attention sublayer, and a feedforward neural network to obtain a prediction vector.

10. A storage medium storing a program, characterized in that: When the program is executed by a processor, the method for jointly extracting entity and relationship information in a text as described in any one of claims 1 to 8 is implemented.

Images