Knowledge enhancement relation extraction method based on multi-modal cross attention

Abstract

According to the knowledge enhancement relation extraction method based on multi-modal cross attention, through a double-encoder architecture and a cross attention mechanism, text and knowledge graph information are effectively fused, and the accuracy and robustness of relation extraction are remarkably improved. Experimental results show that the method achieves performance superior to that of an existing baseline model on multiple data sets, and particularly shows powerful capacity in the aspects of noise data processing and complex relation extraction tasks. Through an ablation experiment, the key contribution of a cross suture mechanism and a local and overall attention mechanism to the model performance is verified. In addition, the method has excellent performance in robustness analysis under different noise proportions, and shows the potential of the method in dealing with the remote supervision data noise problem.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a knowledge-enhanced relationship extraction method based on multimodal cross-attention. Background Art

[0002] Relation extraction is one of the key tasks in information extraction, and its goal is to identify and determine the specific relationship between entity pairs from natural language text. Relation extraction is of great significance in the fields of intelligent retrieval and semantic analysis. It provides basic support for these applications, helps improve search efficiency, promotes the automatic construction of knowledge bases, and generates knowledge triples for downstream tasks such as knowledge graph construction. These structured entity-relationship triples are usually stored in relational databases or graph databases so that computers can better understand and apply knowledge. At present, methods based on supervised learning perform best in relationship extraction tasks. However, this method relies on a large number of high-quality annotated datasets and is limited by the quantity and quality of datasets. In addition, text data in different fields have their own characteristics, and many fields lack public datasets, which increases the difficulty of applying supervised learning methods; The advantages of existing remote supervision technologies include rich data sources, the ability to utilize large-scale text data, and the ability to align and annotate with existing knowledge bases. This not only provides diverse and rich training data for text summary generation, but also helps the model learn text features of different types and topics, improving the accuracy of summary generation. In addition, by combining with the knowledge base, remote supervision technology provides additional knowledge and background information for the text summary generation model, which helps the model better understand the semantics and context of the text, thereby generating more accurate and comprehensive summaries; Distant supervision can also help the model identify common patterns and structures in the text, so as to better grasp the key points and key information of the text, which is crucial for extracting key information in the summary; However, while remote supervision technology reduces the cost of data annotation, it also introduces the problem of incorrect annotation. One of the common methods for dealing with noisy sentences is to use knowledge graphs as a supplementary source of information and encode text and knowledge graphs separately through a dual encoder architecture. However, although this method combines the information of the two types of data, it fails to effectively establish a direct connection between the two. Summary of the invention

[0003] The technical problem to be solved by the present invention is to provide a knowledge-enhanced relationship extraction method based on multimodal cross-attention, which encodes natural language text and knowledge graph information respectively, and realizes the transmission and fusion of two-way information through cross-attention and cross-stitching mechanisms, thereby enhancing the connection between multimodal information and improving the accuracy and robustness of relationship extraction.

[0004] In order to solve the above technical problems, the technical solution adopted by the present invention is: a knowledge-enhanced relationship extraction method based on multimodal cross attention, comprising the following steps: S1. Construct a relationship extraction model, which includes two parts: overall extraction and local extraction. Both parts include an embedding layer, a feature extraction layer, a cross-coordinated attention layer, and a prediction layer. S2, input multiple texts and knowledge graphs into the overall embedding layer and the local embedding layer respectively to obtain text vector representation and knowledge graph vector representation; S3, input the text vector representation and the knowledge graph vector representation obtained from the overall extraction part into two independent and identical attention modules, respectively, and calculate the information between the text as a whole and the information between the graph as a whole; input the text vector representation and the knowledge graph vector representation obtained from the local extraction part into two independent and identical CNN modules, respectively, and calculate the information between the local texts and the information between the local graphs; S4, input the information between the whole text and the whole graph into the cross-coordinated attention module of the whole extraction part to obtain the whole text embedding and the whole graph embedding; input the information between the parts of the text and the parts of the graph into the cross-coordinated attention module of the local extraction part (the same as that in the whole extraction) to obtain the local text embedding and the local graph embedding; S5, concatenate the overall text embedding and the local text embedding, and concatenate the overall graph embedding and the local graph embedding to obtain a sentence embedding set; S6. Perform maximum pooling on all sentence embedding vectors of each dimension to obtain bag-of-sentence embedding. S7, using the Softmax function, calculate the probability that each element represents the corresponding relationship category; S8. Obtain the probability distribution of relationship categories and select the most likely relationship as the prediction result.

[0005] Preferably, in step S2, a word sentence As input, and generate of dimensional context representation; from the last layer through the connection The embeddings of the head and tail entities obtained in and , and by the last layer The word representation obtained by mean pooling and maximum pooling and , construct a pair containing entity pairs Sentences Fixed length representation of ; that is, the final representation of the input sentence is , where “;” indicates vector connection; The knowledge graph encoder is based on triples As input, and generate , of dimensional contextual representation; then, the of , and Use as head entity ,relation and tail entity The embedding of ; the vocabulary of the KG encoder By all entities and relationship , and a mask marker Composition, that is ; For simplicity, assume that the representations of text and KG encoder have the same dimension , that is, setting .

[0006] Preferably, the calculation formula of the attention module is as follows: , , , are the query, key, and value matrices in the overall attention mechanism; these matrices are generated from the input, capturing global contextual dependencies through the overall attention score matrix; the outputs of the local and overall attention mechanisms are combined through weighted combinations to achieve comprehensive modeling of fine-grained and global semantic information: ;in, is a learnable weight parameter that balances the contribution of local and global attention to the final feature representation.

[0007] Preferably, the CNN module uses a one-dimensional convolutional neural network to extract text features. With the graph features ; For each layer of the vectors in the text encoder and knowledge graph encoder , perform convolution operation: ;in, is a learnable parameter matrix, is the bias term, Is the activation function; for the convolution result Perform maximum pooling to obtain the features: ; The CNN layer uses two different filters To extract multiple features, and connect different outputs to form , as the representation of the user's comments; finally, the features are obtained to form a matrix .

[0008] Preferably, the cross-coordinated attention module is given a sentence and the corresponding knowledge graph triples , the text encoder generates a representation of the sentence , while the knowledge graph encoder generates triple representations ; Then calculate the cross attention in two directions: triple to sentence attention ( ) and sentence-to-triplet attention ( ), the calculation formula is as follows: , ;in, and is a trainable linear transformation matrix; triplet to sentence attention Representation is used to update sentence representation The triplet of The embedding weight of each token in : ,in, and is a trainable parameter; next, the gating mechanism determines the original text representation Contribution to the new hidden state of the text encoder: ,in represents the logistic sigmoid function; then, by interpolating the original hidden state and triple table To update the text encoder in Hidden state of a layer: Information sharing in the sentence-to-triplet direction is done in a similar way: , , in, and is the hyperparameter weight.

[0009] Preferably, in step S6, a sentence embedding set is defined , perform maximum pooling on all sentence embedding vectors of each dimension to obtain the sentence bag embedding .

[0010] Preferably, in step S7, the sentence bag embedding vector Perform a linear transformation: in is a vector of length C, representing the unnormalized scores (logits) of each relationship category; Input into the Softmax function to get the probability of each relationship category: in Indicates The probability of a relation category, Indicates The logit value of each category is used to obtain a probability vector of length C. , each element represents the probability of the corresponding relationship category, the probability distribution of the relationship category is obtained, and the most likely relationship is selected as the prediction result.

[0011] The present invention provides a knowledge-enhanced relation extraction method based on multimodal cross attention, which has the following beneficial effects: 1. Through the dual encoder architecture and cross-attention mechanism, text and knowledge graph information are effectively integrated, significantly improving the accuracy and robustness of relationship extraction; 2. It has achieved better performance than existing baseline models on multiple datasets, especially in processing noisy data and complex relationship extraction tasks. Through ablation experiments, we verified the key contributions of the cross-stitching mechanism and the local and global attention mechanisms to the model performance; 3. Excellent performance in robustness analysis under different noise ratios, showing the potential of the model in dealing with the noise problem of remote supervision data; 4. Through the multi-level cross-attention mechanism, deep fusion between text and knowledge graph is achieved, which not only enhances the model's ability to capture fine-grained and global semantic information, but also provides a new method for future knowledge graph construction and relationship extraction tasks; 5. It reduces the cost of entity relationship extraction, improves efficiency and accuracy, and has wide application and promotion value. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The present invention will be further described below in conjunction with the accompanying drawings and embodiments: Figure 1 is a flow chart of the method of the present invention; Figure 2 This is a method model diagram of the present invention. DETAILED DESCRIPTION

[0013] In order to solve the problem that existing supervised learning methods rely heavily on high-quality labeled data, which leads to high labeling costs and data scarcity in many fields, this paper provides a method based on a dual encoder architecture to encode natural language text and knowledge graph information respectively, and realize the transmission and fusion of two-way information through cross-attention and cross-stitching mechanisms, thereby enhancing the connection between multimodal information and improving the accuracy and robustness of relationship extraction. Figure 1 , the method of the present invention comprises the following steps: S1. Build a relation extraction model. The model consists of two parts: overall extraction and local extraction. Both parts include an embedding layer, a feature extraction layer, a cross-coordinated attention layer, and a prediction layer. The structure of the model is referenced Figure 2 shown.

[0014] S2. Input multiple texts and knowledge graphs into the overall embedding layer and the local embedding layer respectively to obtain text vector representation and knowledge graph vector representation.

[0015] Specifically, S2 includes the following steps: S201. Preferably, the text encoder of this example receives a word sentence As input, and generate of Dimensional context representation.

[0016] [1] By connecting The embeddings of the head and tail entities obtained in and , and by the last layer The word representation obtained by mean pooling and maximum pooling and , construct a pair containing entity pairs Sentences The final representation of the input sentence is , where “;” indicates vector connection.

[0017] S202, knowledge graph encoder with triples As input, and generate , of Dimensional contextual representation.

[0018] Will come from the last layer of , and Use as head entity ,relation and tail entity Embedding.

[0019] Vocabulary of Knowledge Graph Encoder By all entities and relationship , and a mask marker Composition, that is .

[0020] For simplicity, this example assumes that the representations of the text and the KG encoder have the same dimension , that is, setting .

[0021] S3. Input the text vector representation and knowledge graph vector representation obtained from the overall extraction part into two independent and identical attention modules respectively, and calculate the information between the text as a whole and the information between the graphs as a whole; input the text vector representation and knowledge graph vector representation obtained from the local extraction part into two independent and identical CNN modules respectively, and calculate the information between the local texts and the information between the local graphs.

[0022] Specifically, S3 includes the following steps: S301. Preferably, the overall attention mechanism aims to capture global semantic information at the text or paragraph level and provide understanding of the context in a larger scope. This part uses the global attention network to perform semantic aggregation within the scope of a paragraph or an entire document. This holistic representation can cover larger context information, thereby supporting the model to extract relations across sentences or paragraphs.

[0023] The calculation formula for overall attention is as follows: . , , are the query, key, and value matrices in the overall attention mechanism. These matrices are generated from the input and capture the global contextual dependencies through the overall attention score matrix.

[0024] The outputs of the local and global attention mechanisms are weighted together to achieve comprehensive modeling of fine-grained and global semantic information: .in, is a learnable weight parameter that balances the contribution of local and global attention to the final feature representation. This combination enables the model to model both details and the whole, thereby improving the ability to understand and extract complex relationships.

[0025] S302. Preferably, this example uses a CNN model to extract local features. The CNN model can extract local features well and has high training efficiency.

[0026] Use one-dimensional convolutional neural network to extract text features With the graph features .

[0027] For each layer of the text encoder and knowledge graph encoder, the vector , perform convolution operation: .in, is a learnable parameter matrix, is the bias term, is the activation function.

[0028] Convolution results By performing maximum pooling, we can get the following features: . The CNN layer uses two different filters To extract multiple features, and connect different outputs to form , as a representation of the graph.

[0029] Finally, the features are obtained and the matrix is ​​formed .

[0030] S4. Input the information between the entire text and the information between the entire graph into the cross-coordinated attention module of the overall extraction part to obtain the overall text embedding and the overall graph embedding; input the information between the local text and the information between the local graph into the cross-coordinated attention module of the local extraction part (the same as that in the overall extraction) to obtain the local text embedding and the local graph embedding.

[0031] Preferably, this example adopts a cross-stitch mechanism, which operates by mixing and updating the intermediate representations of the dual encoders, and dynamically controls the degree of mixing through a cross-attention-based gating mechanism.

[0032] Given a sentence and the corresponding knowledge graph triples , the text encoder generates a representation of the sentence , while the knowledge graph encoder generates triple representations .

[0033] Compute cross attention in two directions: triplet to sentence attention ( ) and sentence-to-triplet attention ( ), the calculation formula is as follows: , .in, and is a trainable linear transformation matrix.

[0034] Triplet-to-Sentence Attention Representation is used to update sentence representation The triplet of The embedding weight of each token in : .in, and is a trainable parameter.

[0035] Gating mechanism determines the original text representation Contribution to the new hidden state of the text encoder: .in Represents the logistic sigmoid function.

[0036] By interpolating the original hidden state and triple table To update the text encoder in Hidden state of a layer: .

[0037] Information sharing in the sentence-to-triplet direction is done in a similar way: , , .in, and is the hyperparameter weight.

[0038] S5. The overall text embedding and the local text embedding are concatenated, and the overall graph embedding and the local graph embedding are concatenated to obtain a sentence embedding set.

[0039] S6. Perform maximum pooling on all sentence embedding vectors of each dimension to obtain bag-of-sentence embedding.

[0040] Define a sentence embedding set ; Perform maximum pooling on all sentence embedding vectors of each dimension to obtain the sentence bag embedding .

[0041] S7. Use the Softmax function to calculate the probability that each element represents the corresponding relationship category.

[0042] Preferably, the sentence bag embedding vector Perform a linear transformation: in is a vector of length C representing the unnormalized scores (logits) of each relationship category.

[0043] Will Input into the Softmax function to get the probability of each relationship category: in Indicates The probability of a relation category, Indicates The logit value of each category.

[0044] Get a probability vector of length C , each element represents the probability of the corresponding relationship category.

[0045] S8. Obtain the probability distribution of relationship categories and select the most likely relationship as the prediction result.

[0046] In order to verify the effect of the knowledge-enhanced relationship extraction method based on multimodal cross-attention provided by the present invention, Tables 1 and 2 are comparisons between the present invention and other existing models on different data sets.

[0047] Table 1. Comparison of the present invention with other existing models on the NYT10 dataset Model Accuracy Precision Recall F1-score PCNN + ATT 0.341 0.636 0.708 0.710 BRE + CE 0.632 0.880 0.737 0.740 Cross-stitched Encoders 0.705 0.944 0.763 0.767 The present invention 0.713 0.953 0.753 0.772 Table 2. Comparison of the present invention with other existing models on the WebNLG dataset Model Accuracy Precision Recall F1-score PCNN + ATT 0.435 0.627 0.730 0.735 BRE + CE 0.645 0.877 0.762 0.765 Cross-stitched Encoders 0.723 0.932 0.785 0.789 MLCA-KRE （ ours ） 0.732 0.941 0.788 0.792 As can be seen from Table 1, the F1-score of the present invention on the NYT10 dataset reaches 0.772. This shows that the present invention performs well in processing noisy remote supervision data and can better combine text and knowledge graph information for effective relationship extraction. On the WebNLG dataset, the F1-score of the present invention is 0.792, which is better than other benchmark models, proving that the model effectively processes multimodal information in the generation task, especially in matching text with knowledge base triples.

[0048] In order to explore the effectiveness of each module in the present invention, the cross stitching mechanism and local and global attention are removed respectively, and ablation experiments are performed. Table 3 shows the results of the ablation experiment of the present invention in the NTY10 dataset.

[0049] Table 3. Results of ablation experiments of the present invention in the NTY10 dataset Model Accuracy Precision Recall F1-score All modules (complete model) 0.711 0.953 0.753 0.772 Remove cross stitching mechanism 0.655 0.915 0.710 0.712 Removing local and global attention 0.637 0.922 0.718 0.720 The experimental results show that the removal of the cross-stitching mechanism leads to a 6.0% performance drop, while the removal of the local and global attention leads to a 5.2% performance drop, indicating that the cross-stitching module and the local and global attention module added in the present invention have an improvement effect.

[0050] In summary, the knowledge-enhanced relation extraction method based on multimodal cross-attention described in the present invention effectively integrates text and knowledge graph information through a dual encoder architecture and a cross-attention mechanism, significantly improving the accuracy and robustness of relation extraction. Experimental results show that the present invention has achieved better performance than existing baseline models on multiple data sets, especially in processing noisy data and complex relation extraction tasks. Through ablation experiments, we verified the key contributions of the cross-stitching mechanism, local and global attention mechanisms to the model performance. In addition, the present invention performs well in the robustness analysis under different noise ratios, showing the potential of the invention in dealing with the noise problem of remote supervision data. The innovation of this study is to achieve deep fusion between text and knowledge graph through a multi-level cross-attention mechanism, which not only enhances the model's ability to capture fine-grained and global semantic information, but also provides a new method for future knowledge graph construction and relation extraction tasks. Future research can further optimize the design of the cross-attention mechanism, explore more efficient noise processing methods, and expand the invention to more application scenarios.

[0051] The above embodiments are only preferred technical solutions of the present invention and should not be regarded as limiting the present invention. The protection scope of the present invention shall be the technical solutions recorded in the claims, including equivalent replacement solutions of the technical features in the technical solutions recorded in the claims. That is, equivalent replacement improvements within this scope are also within the protection scope of the present invention.

Claims

1. A knowledge-enhanced relation extraction method based on multimodal cross-attention, characterized in that: The following steps are involved: S1. Construct a relationship extraction model, which includes two parts: overall extraction and local extraction. Both parts include an embedding layer, a feature extraction layer, a cross-coordinated attention layer, and a prediction layer. S2, input multiple texts and knowledge graphs into the overall embedding layer and the local embedding layer respectively to obtain text vector representation and knowledge graph vector representation; S3, input the text vector representation and the knowledge graph vector representation obtained from the overall extraction part into two independent and identical attention modules, respectively, and calculate the information between the text as a whole and the information between the graph as a whole; input the text vector representation and the knowledge graph vector representation obtained from the local extraction part into two independent and identical CNN modules, respectively, and calculate the information between the local texts and the information between the local graphs; S4, inputting the information between the whole text and the whole graph into the cross-coordinated attention module of the whole extraction part to obtain the whole text embedding and the whole graph embedding; inputting the information between the parts of the text and the parts of the graph into the cross-coordinated attention module of the local extraction part to obtain the local text embedding and the local graph embedding; S5, concatenate the overall text embedding and the local text embedding, and concatenate the overall graph embedding and the local graph embedding to obtain a sentence embedding set; S6. Perform maximum pooling on all sentence embedding vectors of each dimension to obtain bag-of-sentence embedding. S7, using the Softmax function, calculate the probability that each element represents the corresponding relationship category; S8. Obtain the probability distribution of relationship categories and select the most likely relationship as the prediction result.

2. According to claim 1, a knowledge-enhanced relation extraction method based on multimodal cross attention is characterized in that: In step S2, a word sentence As input, and generate of dimensional context representation; from the last layer through the connection The embeddings of the head and tail entities obtained in and , and by the last layer The word representation obtained by mean pooling and maximum pooling and , construct a pair containing entity pairs Sentences Fixed length representation of ; that is, the final representation of the input sentence is , where ";" indicates vector connection; The knowledge graph encoder is based on triples As input, and generate , of Dimensional contextualized representation; Then, the last layer of , and Use as head entity ,relation and tail entity embedding of ; vocabulary of KG encoder By all entities and relationship , and a mask marker Composition, that is ; For simplicity, assume that the representations of text and KG encoder have the same dimension , that is, setting .

3. According to claim 1, a knowledge-enhanced relation extraction method based on multimodal cross attention is characterized in that: The calculation formula of the attention module is as follows: , , , are the query, key, and value matrices in the overall attention mechanism; these matrices are generated from the input, capturing global contextual dependencies through the overall attention score matrix; the outputs of the local and overall attention mechanisms are combined through weighted combinations to achieve comprehensive modeling of fine-grained and global semantic information: ;in, is a learnable weight parameter that balances the contribution of local and global attention to the final feature representation.

4. According to claim 1, a knowledge-enhanced relation extraction method based on multimodal cross attention is characterized in that: The CNN module uses a one-dimensional convolutional neural network to extract text features. With the graph features ; For each layer of the vectors in the text encoder and knowledge graph encoder , perform convolution operation: ;in, is a learnable parameter matrix, is the bias term, Is the activation function; for the convolution result Perform maximum pooling to obtain the features: ; The CNN layer uses two different filters To extract multiple features, and connect different outputs to form , as the representation of the user's comments; finally, the features are obtained to form a matrix .

5. According to claim 1, a knowledge-enhanced relation extraction method based on multimodal cross attention is characterized in that: The cross-cooperative attention module is given a sentence and the corresponding knowledge graph triples , the text encoder generates a representation of the sentence , while the knowledge graph encoder generates triple representations ; Then calculate the cross attention in two directions: triple to sentence attention ( ) and sentence-to-triplet attention ( ), the calculation formula is as follows: , ;in, and is a trainable linear transformation matrix; triplet to sentence attention Representation is used to update sentence representation The triple The embedding weight of each token in : ,in, and is a trainable parameter; next, the gating mechanism determines the original text representation Contribution to the new hidden state of the text encoder: ,in represents the logistic sigmoid function; then, by interpolating the original hidden state and triple table To update the text encoder in Hidden state of a layer: Information sharing in the sentence-to-triplet direction is done in a similar way: , , in, and is the hyperparameter weight.

6. According to claim 1, a knowledge-enhanced relation extraction method based on multimodal cross attention is characterized in that: In step S6, a sentence embedding set is defined , perform maximum pooling on all sentence embedding vectors of each dimension to obtain the sentence bag embedding .

7. According to claim 1, a knowledge-enhanced relation extraction method based on multimodal cross attention is characterized in that: In step S7, the sentence bag embedding vector Perform a linear transformation: in is a vector of length C, representing the unnormalized scores (logits) of each relationship category; Input into the Softmax function to get the probability of each relationship category: in Indicates The probability of a relation category, Indicates The logit value of each category is used to obtain a probability vector of length C. , each element represents the probability of the corresponding relationship category, the probability distribution of the relationship category is obtained, and the most likely relationship is selected as the prediction result.

Images