A method and system for jointly extracting entity and relation information in text

Abstract

The application discloses a method and system for jointly extracting entity and relationship information in a text, comprising the following steps: inputting a text sentence, segmenting the sentence into word units, and obtaining a word unit sequence; inputting the word unit sequence into an ALBERT model to generate a first output vector; inputting the word unit sequence into a TKG_TransE knowledge graph embedding model to generate a second output vector; inputting the first output vector and the second output vector into a GFN gate fusion network to generate a first vector representation; inputting the first vector representation and the word unit sequence into a non-autoregressive decoder to generate a predicted triple; and using a dichotomy matching loss function to evaluate the prediction result and generate a prediction vector. The application realizes knowledge fusion and improves the processing capacity of the model for complex text structures.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, and in particular to a method and system for jointly extracting entity and relationship information in text. BACKGROUND

[0002] With the progress of science and technology and the improvement of living standards, people obtain information from traditional physical stores to the network. Network information is diverse and updated quickly. Many scholars strive to find a method to extract high-quality structured information from different expressions and disordered data. Knowledge graph has become a research hotspot. It can establish a complete graph and efficiently and accurately obtain information and apply it to personalized recommendation, machine question answering, decision support and other fields to meet user needs. One common way is to use knowledge extraction technology to build a knowledge graph from complex data sets. With the rise of deep learning as a node, the development of knowledge extraction can be divided into traditional and deep learning stages.

[0003] In the traditional stage, the knowledge extraction process mainly includes data preprocessing (cleaning, segmentation, part-of-speech tagging, etc.), entity recognition: identifying entities with specific meanings from text, such as names, places, organizations, etc., relationship definition: defining the types of relationships to be extracted, such as "parent-child relationship" "employment relationship" etc., and developing corresponding rules and patterns, relationship extraction: based on the defined relationship, using pattern matching, syntax analysis and other techniques to extract the relationship between entities from the text, post-processing: evaluating, correcting and optimizing the extracted results, such as removing duplicate or incorrect extraction results, to improve the accuracy and completeness of the extraction. The traditional process of knowledge extraction mainly relies on rules and feature engineering-based methods. Researchers manually define rules and extract features of text to identify entity, relationship and attribute knowledge elements. After the rise of deep learning, researchers use deep neural networks to automatically learn the feature representation of text instead of relying on manually designed features. These networks can capture more complex and abstract semantic and syntactic information. Language models pre-trained on large general-purpose corpora (such as BERT, GPT, etc.) can also be used. The processing process is divided into training and prediction stages, and fine-tuned on specific knowledge extraction task datasets to adapt to specific tasks. After the introduction of deep learning, some end-to-end knowledge extraction model algorithms have achieved great success.

[0004] Traditional knowledge extraction process identifies knowledge elements such as entities, relationships and attributes by manually defining rules and extracting features of text. The traditional knowledge extraction process has the following shortcomings: first, it is highly dependent on manual work: rule making, feature engineering and relationship definition require a lot of manual intervention, consuming a lot of time and manpower, and it is difficult to adapt to new fields and data. Second, the generalization ability is limited: the rules and features designed by artificial are often for specific fields and data sets, and when applied to new and different texts, the generalization ability is poor, and inaccurate extraction or omission may occur. After the rise of deep learning, some researchers divide the knowledge extraction task into two independent parts: named entity recognition and relationship extraction. This method ignores the internal relationship and mutual influence between the two tasks: the identification result of the entity may affect the accuracy of relationship extraction, and the result of relationship extraction may in turn provide more clues and constraints for entity recognition. Independent division may not fully utilize this interaction. This division into two independent tasks is usually referred to as pipeline mode. In pipeline mode, entity recognition is usually performed first, and then relationship extraction is performed based on the identified entities. If errors occur in the entity recognition stage, the relationship extraction stage will operate based on the wrong entities, resulting in error propagation problems, and the information obtained in the entity recognition stage cannot be directly utilized by the relationship extraction stage, and vice versa, resulting in information sharing problems. In pipeline mode, the model usually runs two tasks, which is computationally intensive and inefficient. The two serial steps will take more time than jointly extracting entities and relationships, especially when the data volume is large, the efficiency problem will be more obvious. The current mainstream mode is to jointly extract entities and relationships from text. A joint extraction algorithm SPN4RE (using set prediction network for joint entity and relationship extraction) has the advantages of fast decoding speed and no need to predefine the entity relationship set, but it has some shortcomings: insufficient modeling of dependent relationships, limited long-distance dependency processing capability, high computational complexity, and no prior knowledge added, which has room for further improvement. SUMMARY

[0005] In order to overcome the above-mentioned shortcomings and deficiencies of the prior art, the purpose of the present application is to provide a method and system for jointly extracting entity and relationship information from text.

[0006] The purpose of the present application is achieved by the following technical solutions:

[0007] A method for jointly extracting entity and relationship information from text, comprising the following steps:

[0008] Inputting a text sentence, dividing the sentence into word units, and obtaining a word unit sequence;

[0009] Inputting the word unit sequence into an ALBERT model to generate a first output vector;

[0010] inputting the token sequence into a TKG_TransE knowledge graph embedding model to generate a second output vector;

[0011] inputting the first output vector and the second output vector into a GFN gate fusion network to generate a first vector representation;

[0012] inputting the first vector representation and the token sequence into a non-autoregressive decoder to generate a predicted triple, using a dichotomy matching loss function to evaluate the prediction result, and generating a prediction vector.

[0013] Further, the first output vector is generated by inputting the token sequence into an ALBERT model, specifically by inputting the token sequence into a multi-head self-attention mechanism and a feedforward neural network connected in sequence to generate the first output vector.

[0014] Further, the first output vector is generated by inputting the token sequence into an ALBERT model, specifically by inputting the token sequence into a multi-head self-attention mechanism and a feedforward neural network connected in sequence to generate the first output vector.

[0015] Each token is mapped to a corresponding token index according to a vocabulary of the ALBERT model;

[0016] The token index is input into an embedding layer to convert it into a low-dimensional embedding vector;

[0017] The embedding vector is input into an encoder layer to obtain the first output vector, and the encoder layer includes a plurality of identical sub-layers, and each sub-layer includes a multi-head self-attention layer and a feedforward neural network layer connected in sequence.

[0018] Further, the first vector representation is generated by inputting the first output vector and the second output vector into a GFN gate fusion network, specifically by:

[0019] The first output vector is an embedding vector sequence of tokens in a text sentence; and the second output vector is an embedding vector sequence of entities and relationships in a knowledge graph;

[0020] The first output vector is linearly transformed to obtain a gate vector for controlling the weight of the text information in the fusion process;

[0021] The second output vector is linearly transformed to obtain a gate vector for controlling the weight of the knowledge graph information in the fusion process;

[0022] The text information and the knowledge graph information are weighted and summed according to the gate vectors obtained above to generate the first vector representation.

[0023] Further, the non-autoregressive decoder is composed of N identical Transformer modules stacked together, and each Transformer module includes a multi-head self-attention sub-layer, a multi-head cross-attention sub-layer, a global attention sub-layer, and a feedforward neural network.

[0024] Further, the word sequence is input into the TKG_TransE knowledge graph embedding model to generate a second output vector, specifically, the word sequence is converted into a triple and mapped to a low-dimensional vector space, including the following steps:

[0025] Filtering meaningless word sequences, merging continuous nouns into entities based on part-of-speech tagging and pattern matching, and positioning entities using regular expressions;

[0026] In relation extraction, first locate the verb / preposition, predefine the verb and preposition set, filter the verb or preposition as the relationship candidate, and extract the subject-predicate-object structure through SpaCy dependency parsing;

[0027] Combine adjacent entities with the nearest relationship word, only allow one-way relationship, generate a triple list, remove low-frequency triples, merge duplicate triples, dynamically adjust using the TKG_TransE loss function, and output the initial triple list.

[0028] Further, the first vector representation and the word sequence are input into the non-autoregressive decoder to generate a predicted triple, specifically:

[0029] The word sequence is input into the multi-head self-attention sublayer to obtain an embedding vector b1;

[0030] The embedding vector b1 and the first vector representation are input into the multi-head cross-attention sublayer to obtain an embedding vector b2;

[0031] The embedding vector b2 is input into the global attention mechanism sublayer to obtain an embedding vector b3;

[0032] The embedding vector b3 is input into the feedforward neural network to obtain the predicted triple.

[0033] Further, in the query process, the multi-head cross-attention sublayer, the key and value come from the output of the GFN gate fusion network, and the query comes from the initialized query vector. The purpose is to integrate the context information of the text sentence into the triple query representation of the non-autoregressive decoder, and fully utilize the semantic and syntactic information of the input sentence.

[0034] A system for implementing the method, comprising the following:

[0035] A sentence encoder, the sentence encoder comprising an ALBERT model, a TKG_TransE knowledge graph embedding model, and a GFN gate fusion network;

[0036] A non-autoregressive decoder includes N identical stacked Transformer modules, each including a multi-head self-attention sublayer, a multi-head cross-attention sublayer, a global attention sublayer, and a feed-forward neural network, for obtaining a prediction vector.

[0037] A storage medium stores a program, and when the program is executed by a processor, the method for jointly extracting entity and relationship information in text is implemented.

[0038] Compared with the prior art, the present application has the following advantages and beneficial effects:

[0039] The method introduces a TransE module to map entities and relationships in the knowledge graph to a low-dimensional vector space, thereby representing knowledge in a structured manner. In the knowledge extraction task, this structured knowledge can provide rich prior information for the model. For example, for a text involving multiple entities and complex relationships, the knowledge graph vector representation provided by TransE can help the model understand the potential relationships between entities, which may not be directly expressed in the text but have explicit connections in the knowledge graph. This helps the model better capture semantic relationships when processing text, improving the accuracy of entity relationship extraction.

[0040] By incorporating information from the knowledge graph, the model can understand the semantics of the text from multiple perspectives. The combination of entity attributes, relationship types, and other information in the knowledge graph with the context information in the text enables the model to have a more comprehensive and in-depth understanding of the semantics of the text. For example, when dealing with ambiguous entities, the classification or related relationship information about the entity in the knowledge graph can help the model determine its accurate meaning based on the context, thereby improving the accuracy of semantic understanding and ultimately improving the quality of knowledge extraction.

[0041] When processing long texts, the long-distance dependency problem is a challenge. The knowledge graph information carried by TransE can provide a global information connection that spans long distances. Even if two entities in the text are far apart in position, if they have a clear relationship in the knowledge graph, the model can capture this relationship through the vector representation of TransE, thereby to some extent alleviating the impact of the long-distance dependency problem on entity relationship extraction. For example, in a long article discussing historical events, related characters and events mentioned in different paragraphs may be far apart in the text, but through the pre-defined relationships in the knowledge graph, the model can better establish the connection between them, improving the ability to handle complex semantic relationships in long texts.

[0042] The method introduces an ALBERT module using cross-layer parameter sharing technology. Unlike traditional Transformer models (such as BERT), it shares some or all parameters between different layers. This means that the number of parameters that need to be updated and stored during model training and inference is significantly reduced. For example, when processing large-scale text data, the computational complexity is effectively reduced, thereby speeding up the training of the model, improving computational efficiency, and enabling the model to maintain good performance even in resource-limited situations.

[0043] Cross-layer parameter sharing helps stabilize the parameter update process. During model training, the stability of parameter updates is crucial for convergence speed and final performance. The parameter sharing mechanism of ALBERT makes the model's parameter changes relatively more stable during training, reducing the problem of training divergence or performance fluctuations caused by excessive or unstable parameter updates, and helping the model reach a better performance state faster and exhibit more stable performance on different datasets and tasks.

[0044] The introduction of the GFN module in the method enables the fusion of information from different sources. In this model, the main fusion is the text information output by the text encoder (such as ALBERT) and the knowledge graph information output by the knowledge graph embedding model (such as TransE). Text information contains rich details such as semantics, syntax, and contextual relationships in sentences, while knowledge graph information provides structured entity and relationship knowledge. Through GFN, the model can comprehensively utilize these two information sources to make more accurate and comprehensive predictions. For example, when processing a sentence containing entity mentions, text information can provide specific context and roles of the entity in the sentence, while knowledge graph information can supplement the entity's relationships and attributes in a broader knowledge system. The fusion of the two can provide more rich feature representations for entity relationship extraction.

[0045] The introduction of the gating mechanism makes the fusion process dynamic. Gating vectors (such as text gating vector g t and knowledge graph gating vector g k ) control the importance of different information sources in different situations through learning. In some scenarios, if the description of an entity or relationship in the text is very clear and detailed, the text gating vector may give the text information a higher weight, making the model rely more on text information for prediction; in other cases, when there is prior knowledge in the knowledge graph that is highly relevant to the current task, the knowledge graph gating vector will adjust accordingly, highlighting the role of knowledge graph information. This dynamic fusion mechanism can adaptively adjust the information fusion strategy according to the characteristics of the input data, improving the model's adaptability and processing capacity for different types of data.

[0046] The method introduces a Global Attention module. When processing long texts, the entities and relationships in the text may be far apart, and traditional local attention mechanisms may not be able to effectively capture the relationship between them. Global Attention can perform weighted summation on all positions of the encoder output sequence, thereby providing the decoder with a representation containing global information. This allows the model to consider the information of the entire input sequence when generating entity and relationship predictions, establishing semantic associations between entities and relationships across long distances. For example, in an article discussing complex scientific theories, a certain entity is mentioned at the beginning, while the related relationship is described at the end. Global Attention can help the model consider the information at the beginning and end of the article when predicting the entity relationship, avoiding missing important associated information due to the long distance, improving the processing ability of long-distance dependency relationships, and reducing the omission or error in long text entity relationship extraction.

[0047] By incorporating global information into the input representation of the decoder, Global Attention helps the model understand the semantics of the text from a more macro perspective. This allows the model to consider not only local context information but also the semantic context of the entire text, thereby improving the understanding of complex semantic relationships in the text. In cases of polysemy or semantic ambiguity, combining global information can help the model more accurately determine the meaning of entities and relationships, thereby improving the quality and accuracy of knowledge extraction and providing more comprehensive and in-depth semantic understanding support for the model in the knowledge extraction task. BRIEF DESCRIPTION OF DRAWINGS

[0048] Fig. 1 is a schematic diagram of the module information flow of the present application;

[0049] Fig. 2 is a schematic diagram of the module structure of the present application;

[0050] Fig. 3 is a workflow diagram of the present application. DETAILED DESCRIPTION

[0051] The present application will be further described in detail below in conjunction with embodiments, but the embodiments of the present application are not limited thereto.

[0052] Embodiments

[0053] As shown in Figs. 1-3 , a system for jointly extracting entity and relationship information in text includes a sentence encoder, which includes an ALBERT model, a TKG_TransE knowledge graph embedding model, and a GFN gated fusion network.

[0054] The non-autoregressive decoder comprises N identical transformer modules stacked, each comprising a multi-head self-attention sublayer, a multi-head cross-attention sublayer, a global attention sublayer and a feedforward neural network, for obtaining a prediction vector.

[0055] The application improves long-distance dependence processing capability, reduces model calculation amount through parameter sharing, realizes knowledge fusion through TKG_TransE and GFN, and improves the processing capability of the model for complex text structure.

[0056] The method for jointly extracting entity and relationship information in text realized by the system comprises the following steps:

[0057] The input text sentence is subjected to a word segmentation operation through byte pair encoding (BPE) to segment the sentence into tokens and obtain a token sequence.

[0058] Specific description: Byte pair encoding is a data compression algorithm and is also commonly used in text segmentation in natural language processing. The basic idea is to iteratively merge the most frequent byte pairs to build a limited vocabulary. In the segmentation process, the frequency of byte pairs (character pairs) in the text is counted, and then the byte pair with the highest frequency is selected for merging to form a new "character", and the statistical information of byte pairs in the text is updated. This process is repeated until the preset vocabulary size is reached or other stopping conditions are met. In this way, BPE can segment the text into a series of tokens, which can be single characters, common character combinations or new symbols formed by merging, thereby effectively handling out-of-vocabulary words and generating appropriate token representations according to the characteristics of the text data.

[0059] First, initialize the vocabulary, treat each character in the text as a separate token, and count the number of occurrences of all byte pairs (character pairs). For example, for the input sentence "John visited the Eiffel Tower in Paris.", the initial vocabulary is {“J”, “o”, “h”, “n”, “”, “v”, “i”, “s”, “t”, “e”, “d”, “t”, “h”, “e”, “E”, “i”, “f”, “f”, “e”, “l”, “T”, “o”, “w”, “e”, “r”, “i”, “n”, “P”, “a”, “r”, “i”, “s”, “.”} and the number of occurrences of byte pairs such as (“J”, “o”), (“o”, “h”) is counted.

[0060] Then, the byte pair with the highest frequency is selected for merging. Assuming that in this example, the byte pair consisting of "e" and "l" has a high frequency, they are merged into a new token "el", and the vocabulary and the statistics of byte pairs in the text are updated. At this time, the vocabulary becomes {"J", "o", "h", "n", " ", "v", "i", "s", "t", "ed", "th", "e", "E", "i", "f", "fe", "l", "T", "o", "w", "er", "i", "n", "P", "a", "r", "is", "."} and the text becomes "John visited the Eiffel Tower in Paris.".

[0061] The above merging step is repeated, and the vocabulary and the text are constantly updated until a preset stopping condition is reached, obtaining a token sequence, such as the size of the vocabulary reaching a certain number or there being no byte pair that can be merged (with a frequency lower than a certain threshold). The final tokenization result may be ["John", "visited", "the", "Eiffel", "Tower", "in", "Paris", "."] (the tokenization result here is only an example, and the actual tokenization may differ due to specific BPE implementation and parameter settings).

[0062] The token sequence is input into the ALBERT model to generate a first output vector.

[0063] The specific generation process is as follows:

[0064] First, each token in the token sequence is mapped to the corresponding token index according to the ALBERT model vocabulary. The ALBERT model constructs a vocabulary containing a large number of common tokens and symbols during pre-training, and by looking up the vocabulary, the input token is converted into the corresponding integer index. For example, if "John" has an index of 100 in the ALBERT vocabulary and "visited" has an index of 200, etc., the tokenized sentence ["John", "visited", "the", "Eiffel", "Tower", "in", "Paris", "."] will be converted into the corresponding index sequence [100, 200, 300, 400, 500, 600, 700, 800] (the index values here are only examples). At the same time, some special markers such as [CLS] (representing the beginning of the sentence) and [SEP] (representing the end of the sentence) need to be added, and the index sequence becomes [101, 100, 200, 300, 400, 500, 600, 700, 800, 102] (assuming that the index of [CLS] is 101 and the index of [SEP] is 102).

[0065] The embedding layer of the ALBERT model converts these token indices into low-dimensional embedding vectors. The embedding layer includes token embeddings, position embeddings, and segment embeddings. Token embeddings map each token index to a fixed-dimensional (e.g., 768-dimensional) vector that represents the semantic and syntactic features of the token; position embeddings are used to encode the positional information of tokens in a sentence, enabling the model to distinguish tokens in different positions; segment embeddings are used to distinguish different sentence segments (e.g., when processing text composed of multiple sentences), and in the case of a single sentence input, the segment embeddings can represent the unique identification of the sentence. The three types of embeddings are combined through element-wise addition to form the final input embedding vector. For example, for the token "John", its token embedding, position embedding (assuming a position of 1 in the sentence), and segment embedding (assuming a single sentence with a segment identifier of 0) are added together to obtain a 768-dimensional vector a 1,1 (representing the embedding vector corresponding to the first token "John" in the input sentence), and similarly, other tokens in the sentence will also obtain their respective embedding vectors a 1,1 . 1,2 . 1,3 . 1,n (n is the number of tokens in the sentence), and the embedding vector of the entire sentence is represented as a1 = [a 1,1 . 1,2 . 1,3 . 1,n .

[0066] The low-dimensional embedding vector input is input into the encoder layer of the ALBERT model to obtain a first output vector. The encoder layer is composed of multiple identical sub-layers, each of which includes a multi-head self-attention layer and a feed forward network layer, and has a residual connection and a layer normalization operation before and after each sub-layer. In the multi-head self-attention layer, each head calculates the attention score between word units, focuses on the information of different parts of the sentence, and then obtains a new word unit representation through weighted summation and other operations. This process enables the model to capture the semantic relationship and context information between word units. After the multi-head self-attention layer processing, the word unit representation is further converted by the feed forward network layer to enhance the model's understanding of semantics. After processing by multiple encoder layers, the final representation vector of each word unit after encoding by the entire model is obtained. These vectors contain rich sentence context semantic information and will be used for further calculation and processing in subsequent model components (such as non-autoregressive decoders) to achieve knowledge extraction and other tasks. For example, after processing by all encoder layers, the vector corresponding to the word unit "John" may have fused the information of other word units in the entire sentence, such as the semantic association with the word units "visited" and "Eiffel Tower", thereby more comprehensively representing its semantic role and context relationship in the sentence.

[0067] The word unit sequence is input into the TKG_TransE knowledge graph embedding model to generate a second output vector, which is mainly used to convert the word unit sequence into a triple and map it to a low-dimensional vector space.

[0068] The specific generation process is as follows:

[0069] Input the tokenized token sequence (such as ["Beijing", "is", "the", "capital", "of", "China"]) into the preprocessing stage. First, predefine a set of meaningless stop words (such as "of", "is"), and retain entity and relationship candidate words (such as "Beijing", "capital"). Then perform chunk merging. Based on part-of-speech tagging (POS) and pattern matching (such as noun phrase merging rules), merge consecutive nouns into entities (such as ["China", "capital"] -> ["China's capital"], where the part-of-speech filtering marks nouns (NOUN), proper nouns (PROPN), etc. as entity candidates, and use regular expressions to locate entities (such as the noun after the "of" structure as the tail entity). In relation extraction, first perform verb / preposition location. Predefine a set of verbs and prepositions, and filter verbs (such as "is located in") or prepositions (such as "of") as relationship candidates. Extract the subject-predicate-object structure through the dependency parsing of SpaCy (such as "Beijing - capital - China"). Combine adjacent entity pairs with the nearest relationship word (such as the relationship between "Beijing" and "China's capital" is "is"), and only allow one-way relationships (such as: capital only applies to "Beijing -> China", and the reverse is invalid), and output a preliminary list of triples.

[0070] Making in this vector space, for a correct triple (h, r, t) (head entity h, relationship r, tail entity t), satisfy the relationship of h + r ≈ t. By minimizing a margin-based loss function (such as ∑ (h,r,t) ∈S(h′,r′,t′)∈S′[γ + f(h, r, t) - f(h′, r′, t′)] + ,

[0071] where S is the set of positive sample triples, S′ is the set of negative sample triples, γ is the margin parameter, [x] + = max(0, x)), the model learns the vector representations of entities and relationships, making the scores of positive sample triples lower (satisfying the above approximate relationship), while the scores of negative sample triples are higher (negative samples are generated by randomly replacing the head entity, tail entity, or relationship).

[0072] At the beginning of training, randomly initialize a low-dimensional vector for each entity and relationship (for example, in a d-dimensional vector space, d is usually a small integer, such as 50 or 100). Assume the initial vector of entity "John" is e1 = (e 1,1 , e 1,2 , …, e 1,d ), the initial vector of "Eiffel Tower" is e2 = (e 2,1 , e 2,2 , …, e 2,d ), and the initial vector of relationship "visited" is r3 = (r3,1 ,r 3,2 ,…,r 3,d )

[0073] For the input triple (1, 3, 2) (corresponding to "John", "visited", "Eiffel Tower"), the value of h + r - t is calculated, i.e. e1+ r3- e2= (e 1,1 +r 3,1 -e 2,1 ,e 1,2 +r 3,2 -e 2,2 ,…,e 1,d +r 3,d -e 2,d ). Then the norm of this vector (e.g. L2 norm) This norm is the score of this triple in the TKG_TransE model.

[0074] According to the above loss function, the loss of the positive sample triple is calculated, and the vectors of entities and relations are updated through the back propagation algorithm. For example, if the current triple is a positive sample, the model will calculate its loss value according to the loss function. And adjust the vector values of e1, r3 and e2 according to the gradient descent algorithm, so that the score of the positive sample is lower (closer to the ideal h + r ≈ t relationship). During the training process, the model will traverse a large number of positive sample and negative sample triples, and constantly adjust the vector representation until convergence.

[0075] After sufficient training, the obtained vector representation of entities and relations is the second output vector we need. Assuming that the embedding vector of "John" after training is a 2,1 =(a 2,1,1 ,a 2,1,2 ,…,a 2,1,d ), the embedding vector of "Eiffel Tower" is a 2,2 =(a 2,2,1 ,a 2,2,2 ,…,a 2,2,d ), and the embedding vector of "visited" is a2,3= (a 2,3,1 ,a 2,3,2 ,…,a 2,3,d ), then the embedding vector of the entity and relation in the sentence is a2= [a 2,1 ,a 2,3 ,a 2,2(Arranged in the order of entity-relation-entity). These embedding vectors contain semantic representations of entities and relations in a low-dimensional vector space, capturing the structural and semantic relationships between them. Furthermore, when fused with other information, they provide the model with structured knowledge from the knowledge graph, enhancing the model's ability to understand and extract entities and relations from the text.

[0076] The first and second output vectors are input into the GFN gated fusion network to generate the first vector representation.

[0077] The specific process is as follows:

[0078] The first output vector is the sequence of embedding vectors of words in the text sentence; the second output vector is the sequence of embedding vectors of entities and relations in the knowledge graph.

[0079] GFN (Gated Fusion Network) dynamically determines the importance of each information source by learning gating vectors. For the fusion of textual and knowledge graph information, there are typically two gating vectors: a text gating vector and a knowledge graph gating vector. The calculation of gating vectors usually involves linear transformations and nonlinear activations of the corresponding information. The fusion process then performs a weighted summation of the textual and knowledge graph information based on the gating vectors.

[0080] The GFN gated fusion network receives a first output vector (text information representation a1) from the ALBERT model and a second output vector (knowledge graph information representation a2) from the knowledge graph embedding model TKG_TransE. Assume a1 is the sequence of embedding vectors in the text encoded by the ALBERT model [a...]. 1,1 ,a 1,2 ,…,a 1,n (n is the number of terms in the text), a2 is the embedding vector sequence of related entities and relations in the knowledge graph (e.g., for a triple containing two entities and one relation, a2 = [a 2,1 ,a 2,3 ,a 2,2 Arranged according to the order of entities in the entity relationship).

[0081] Perform a linear transformation on the text information a1: h t =a1W t , where W t It is a trainable weight matrix. For example, if a1 has an shape of n×d1 (n is the number of tokens, and d1 is the text embedding dimension), W t If the shape is d1×d (d is the desired transformed dimension, usually consistent with the dimensions of other vectors in the model), then h t The shape is n×d. The gate vector g is obtained through a non-linear activation function (such as the sigmoid function). t =σ(h)t ), where g t t is the same as h k , i.e., nxd. This gating vector will be used to control the importance of textual information in the fusion process.

[0082] Performing a linear transformation on the knowledge graph information a2: h k = a2W k , where W k is a trainable weight matrix. Assuming a2 has a shape of mxd2 (m is the number of related entities and relationships in the knowledge graph, and d2 is the knowledge graph embedding degree), W k has a shape of d2xd ,则 , and h k has a shape of mxd. Then, through a nonlinear activation, the gating vector g k = σ(h k ) is obtained, and g k has the same shape as h k , i.e., mxd. This gating vector determines the weight of the knowledge graph information in the fusion.

[0083] To perform element-wise multiplication operations, further adjustments to the dimensions of the gating vectors g t , g k and the information a1, a2 may be needed. For example, if g t has a shape of nxd, a1 has a shape of nxd1, g t may need to be expanded to nxd1 (by copying or other means) to match a1 in dimension, and similar dimension adjustment operations are performed on a2 and g k to ensure that they can perform element-wise multiplication.

[0084] According to the calculated gating vectors, the textual information and the knowledge graph information are weighted and summed. The fused information F is calculated as follows: F = g t ⊙a1 + g k ⊙a2, where denotes element-wise multiplication. g t ⊙a1 has the same shape as the adjusted a1, and g k ⊙a2 has the same shape as the adjusted a2, and then the two results are added to obtain the fused information F. The shape of F depends on the specific calculation and adjustment method, but it integrates the textual and knowledge graph information to provide a richer and more comprehensive input representation for subsequent model components.

[0085] The first vector representation and the token sequence are input into a non-autoregressive decoder to generate predicted triples, and a dichotomous matching loss function is used to evaluate the prediction results to generate a prediction vector.

[0086] The non-autoregressive decoder comprises N identical stacks of Transformer modules, each comprising a multi-head self-attention sublayer, a multi-head cross-attention sublayer, a global attention sublayer, and a feed-forward neural network.

[0087] The multi-head self-attention mechanism is one of the core components in the Transformer architecture, which allows the model to simultaneously focus on different parts of the input sequence when processing it, thereby capturing richer semantic and syntactic relationships. The core idea is to compute the relevance scores (attention scores) between each position's token and all other tokens in the sequence, and then weight-sum the information of all tokens according to these scores to update the representation of each position's token.

[0088] The sequence of tokens is input into the multi-head self-attention mechanism sublayer to obtain the embedding vector b1.

[0089] The embedding vector b1 and the initialized triple query vector (first vector representation) are input into the multi-head cross-attention mechanism sublayer (Multi-Head Cross-Attention) to obtain the embedding vector b2, and the embedding vector b2 is input into the global attention mechanism sublayer (Global Attention) to output the embedding vector b3

[0090] Further explanation:

[0091] The multi-head cross-attention sublayer is similar to the multi-head self-attention sublayer, with the query coming from the output of the previous multi-head cross-attention sublayer, and the key and value coming from the output of the sentence encoder. The purpose is to incorporate the context information provided by the sentence encoder into the triple query representation of the decoder, so that the decoder can fully utilize the semantic and syntactic information of the input sentence when generating triples.

[0092] Suppose the output of the previous multi-head cross-attention sublayer is Q prev , with shape mxd (m is the number of triples, d is the number of hidden units). Perform linear transformation on Q p rev to obtain the query vector Q″ = Q prev W′ Q (W′ Q is a trainable weight matrix with shape dxd k ), with shape mxdd k . At the same time, obtain the key vector K″ = HW K′和值向量V″ = HW V′ from the output H of the sentence encoder (ALBERT) (assuming shape nxd, n is the length of the sentence), and the value vector V″ = HW K′ = HW v′is trainable, shape dxd k ), shape nxd k .

[0093] For each head i, compute attention scores Then apply softmax to get attention distribution Here the computation is similar to multi-head self-attention sublayer, but the key and value vectors come from the sentence encoder output, so the computation is the relevance of the triple query to each wordpiece in the sentence.

[0094] According to the attention distribution a′ ij Weighted sum of value vectors V″ to get the output of each head Shape mxd k Finally, concatenate the outputs of all heads and pass through a linear transformation (weight matrix W O′ , shape hxd k xd) to get the final multi-head cross-attention sublayer output O′ = Concat(O′1, O′2, …, O′ h )W o′ , shape mx d.

[0095] The output of the Global Attention mechanism is the weighted sum of all positions in the input sequence, providing the decoder with a representation that contains global information to help the decoder better consider the information of the entire input sentence when generating triples.

[0096] Global Attention receives the output H from Multl-Head Cross-Attention, and at the same time, initializes a global attention weight vector a (shape n) to calculate the attention weight for each position in the encoder output sequence.

[0097] Embedding vector b3 is passed through a Feed Forward Network (FFN) to convert the information processed by the attention mechanism into specific prediction results, which is an important part of the decoder to generate accurate triples.

[0098] Embedding vector b3 is passed through a Feed Forward Network (FFN) to generate a predicted triple, and the binary matching loss function is used to measure the difference between the predicted triple and the real triple, and the parameters are backpropagated.

[0099] FFN receives the output from the global attention sublayer as input. Let the input vector be x, with shape m x d (m is the number of triplets or other sequence length related to the model, and d is the number of hidden units).

[0100] First, a first linear transformation is performed on the input vector x by a trainable weight matrix W1 (with shape d x d fj d ff The dimension of the intermediate layer of the feedforward neural network, usually d ff is greater than d), to obtain an intermediate result h = xW1, with shape m x d ff This linear transformation can map the input vector from the original hidden unit space to a higher dimensional space, thereby increasing the expressive power of the model and enabling the model to learn more complex feature representations.

[0101] A nonlinear activation function is applied to the intermediate result h, and a common choice is the ReLU (Rectified Linear Unit) function, i.e., h' = ReLU(h). The definition of the ReLU function is ReLU(x) = max(0, x), which can introduce nonlinearity, enabling the model to learn nonlinear relationships in the input data. Through the action of the ReLU function, the model can handle complex semantic and syntactic information, enhancing the expressive power of the input features. For example, for some complex semantic relationships such as negation, transition, etc., the ReLU function can help the model capture these nonlinear features, thereby better understanding the semantics in the text.

[0102] After the nonlinear activation, a second linear transformation is performed on the result h' by another trainable weight matrix W2 (with shape d ff x d), to obtain the final output y = h'W2, with shape m x d. This step maps the high-dimensional representation of the intermediate layer back to the original hidden unit dimension, so as to interact with other components of the model and pass information. The role of the second linear transformation is to adjust the dimension of the feature representation to adapt to the subsequent calculation requirements, while further optimizing the feature representation and extracting key information related to the task.

[0103] In the knowledge extraction task, the relationship triplets in the sentence are essentially an unordered set, but the traditional autoregressive decoder outputs a sequence, which needs to sort the unordered target triplets. The cross-entropy loss function is sensitive to the order of the predicted results, and is not suitable for measuring the difference between unordered triplet sets. Therefore, inspired by the assignment problem in operations research, this paper proposes a bipartite matching function

[0104] Further explanation:

[0105] Bipartite Matching Loss plays a key role in model training, which measures the difference between predicted triples and real triples. In the knowledge extraction task, the traditional cross-entropy loss function is sensitive to the arrangement of the prediction results, and is not suitable for measuring the difference between unordered triple sets. Therefore, Bipartite Matching Loss is proposed, which is inspired by the assignment problem in operations research. It can generate the optimal bipartite matching between predicted triples and real triples, so as to more reasonably evaluate the prediction results of the model.

[0106] Let Y = { (s, r, o)} be the set of real triples, where each (s, r, o) ∈ Y represents a real triple. r i is the target relation type (possibly a special relation type denotes the set of triples without triples, are the start or end indices of the subject s and object o, respectively. Let X = { (s, r, o)} be the set of m triples predicted by the model (m is usually greater than n), where each (s, r, o) ∈ X represents a predicted triple. These prediction values are calculated by the model's decoder (such as non-autoregressive decoder) (for example, and other formulas to calculate the prediction probability of relation type and entity index).

[0107] First, define a pairwise matching cost function to measure the difference between real triples Y i and predicted triples represent a permutation). Its calculation method is Here is an indicator function, which is 1 when , otherwise 0. This cost function considers the prediction accuracy of relation type and entity span. Then by searching all possible permutations π ∈ Π(m) (Π(m) is the space of all m-length permutations), find the permutation π that minimizes the total cost * , that is This optimal matching problem can be solved by the classic Hungarian Algorithm, with a time complexity of O(m 3 ).

[0108] After finding the optimal matching, the loss function is calculated. The loss function is defined as This loss function calculates for all matched pairs, measures the difference between the predicted triplets and the true triplets by taking the logarithm of the predicted probability and adding a proper penalty term (taking the logarithm of the predicted probability of the entity index and adding it when the relation type is not empty). The larger the loss value, the greater the difference between the predicted result and the true situation. In the training process, the model will minimize this loss function through optimization algorithms (such as stochastic gradient descent) to adjust the parameters of the model, so that the predicted result is closer to the true triplets.

[0109] Unlike the traditional cross-entropy loss function, the binary matching loss function is not sensitive to the order of the predicted triplets. In the knowledge extraction task, the relationship triplets in the sentence are essentially an unordered set, but the traditional autoregressive decoder needs to sort the unordered target triplets to adapt to the form of the output sequence, and the cross-entropy loss function is very sensitive to this sorting, and will give a large penalty for a small change in the predicted order. The binary matching loss function can directly handle unordered predicted and true triplet sets by the optimal binary matching principle, ignoring the order of triplets and focusing on the accuracy of relation types and entities, thus more conforming to the essential characteristics of the task.

[0110] The WEBNLG dataset includes 5019 entries in the training set, 500 entries in the validation set, and 703 entries in the test set. The test results are shown in Table 1.

[0111] Table 1 Experimental results of WEBNLG dataset

[0112]

[0113]

[0114] The NYT dataset includes 56195 entries in the training set, 5000 entries in the validation set, and 5000 entries in the test set. The test results are shown in Table 2.

[0115] Table 2 Experimental results of NYT dataset

[0116]

[0117] The method uses the same dataset as the original model, and the test model score is higher than the original model, indicating that the improvement is effective.

[0118] The embodiment also provides a storage medium, wherein the program is executed by a processor to realize the method for jointly extracting entity and relationship information in text.

[0119] The above embodiments are the preferred embodiments of the present application, but the embodiments of the present application are not limited by the above embodiments, and any changes, modifications, substitutions, combinations, simplifications, etc. made without departing from the spirit and principles of the present application shall be equivalent replacement manners and shall be included in the protection scope of the present application.

Claims

1. A method for jointly extracting entity and relation information in text, characterized in that, The method comprises the following steps: inputting a text sentence, segmenting the sentence into wordpieces, and obtaining a wordpiece sequence; inputting the wordpiece sequence into an ALBERT model to generate a first output vector; inputting the wordpiece sequence into a TKG_TransE knowledge graph embedding model to generate a second output vector; inputting the first output vector and the second output vector into a GFN gate fusion network to generate a first vector representation; inputting the first vector representation and the wordpiece sequence into a non-autoregressive decoder to generate a predicted triple, evaluating the prediction result by using a dichotomy matching loss function, and generating a prediction vector; The first output vector is an embedding vector sequence of the wordpieces in the text sentence; and the second output vector is an embedding vector sequence of entities and relationships in the knowledge graph. performing linear transformation on the first output vector to obtain a gate vector for controlling the weight of the text information in the fusion process; performing linear transformation on the second output vector to obtain a gate vector for controlling the weight of the knowledge graph information in the fusion process; performing weighted summation on the text information and the knowledge graph information according to the gate vectors obtained above to generate the first vector representation; The non-autoregressive decoder is composed of N identical Transformer modules stacked together, and each Transformer module comprises a multi-head self-attention sublayer, a multi-head cross-attention sublayer, a global attention sublayer and a feedforward neural network. The first vector representation and the wordpiece sequence are inputted into the non-autoregressive decoder to generate a predicted triple, and the specific process is as follows: the wordpiece sequence is inputted into the multi-head self-attention sublayer to obtain an embedding vector b1; the embedding vector b1 and the first vector representation are inputted into the multi-head cross-attention sublayer to obtain an embedding vector b2; the embedding vector b2 is inputted into the global attention mechanism sublayer to obtain an embedding vector b3; the embedding vector b3 is inputted into the feedforward neural network to obtain a predicted triple. The first output vector is generated by inputting the wordpiece sequence into the multi-head self-attention mechanism and the feedforward neural network connected in sequence.

2. The method of claim 1, wherein, The first output vector is generated by inputting the wordpiece sequence into the ALBERT model, and the specific process is as follows:

3. The method of claim 1, wherein, each wordpiece is mapped to a corresponding wordpiece index according to the word table of the ALBERT model; the wordpiece index is converted into a low-dimensional embedding vector by inputting the wordpiece index into an embedding layer; the embedding vector is inputted into an encoder layer to obtain the first output vector, and the encoder layer comprises a plurality of identical sublayers, and each sublayer comprises a multi-head self-attention layer and a feedforward neural network layer connected in sequence. The second output vector is generated by inputting the wordpiece sequence into the TKG_TransE knowledge graph embedding model, and the specific process is as follows:

4. The method of claim 1, wherein, filtering meaningless wordpieces in the segmented word sequence, merging continuous nouns into entities based on part-of-speech tagging and pattern matching, and positioning the entities by using a regular expression; ​ In relation extraction, first, verb / case positioning is performed, a set of verbs and cases is predefined, verbs or cases are filtered as relation candidates, and subject-predicate-object structures are extracted through SpaCy dependency parsing; The adjacent entities are combined for the nearest relation, only one-way relations are allowed, a triple list is generated, low-frequency triples are removed, repeated triples are merged, a TKG_TransE loss function is dynamically adjusted, and an initial triple list is output.

5. The method of claim 1, wherein, In the query process, the keys and values come from the output of the GFN gated fusion network, and the query comes from the initialized query vector. The purpose is to integrate the context information of the text sentence into the triple query representation of the non-autoregressive decoder, and fully utilize the semantic and syntactic information of the input sentence.

6. A system implementing the method of any one of claims 1-5, characterized by The method comprises the following steps: The sentence encoder comprises an ALBERT model, a TKG_TransE knowledge graph embedding model, and a GFN gated fusion network. The non-autoregressive decoder comprises N identical Transformer modules stacked, each Transformer module comprising a multi-head self-attention sublayer, a multi-head cross-attention sublayer, a global attention sublayer, and a feedforward neural network, and is used to obtain a prediction vector.

7. A storage medium storing a program, characterized by comprising: The program is executed by the processor to implement the method for jointly extracting entity and relation information in the text according to any one of claims 1-5.

Images