A knowledge-enhanced relation extraction method based on multimodal cross-attention

Abstract

This invention presents a knowledge-enhanced relation extraction method based on multimodal cross-attention. Through a dual-encoder architecture and a cross-attention mechanism, it effectively fuses textual and knowledge graph information, significantly improving the accuracy and robustness of relation extraction. Experimental results show that this invention outperforms existing baseline models on multiple datasets, particularly demonstrating strong capabilities in handling noisy data and complex relation extraction tasks. Ablation experiments verify the key contributions of the cross-stitching mechanism and local and global attention mechanisms to model performance. Furthermore, this invention exhibits excellent robustness under different noise levels, demonstrating its potential in addressing noise issues in remotely supervised data.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to a knowledge-enhanced relation extraction method based on multimodal cross-attention. Background Technology

[0002] Relation extraction is a key task in information extraction, aiming to identify and determine specific relationships between entity pairs in natural language text. Relation extraction is crucial for intelligent retrieval, semantic analysis, and other fields, providing fundamental support for these applications, improving search efficiency, promoting the automatic construction of knowledge bases, and generating knowledge triples for downstream tasks such as knowledge graph construction. These structured entity-relation triples are typically stored in relational databases or graph databases, enabling computers to better understand and utilize knowledge. Currently, supervised learning-based methods perform best in relation extraction tasks. However, these methods rely on large amounts of high-quality labeled datasets, limiting their application due to the quantity and quality of datasets. Furthermore, text data from different domains has distinct characteristics, and many domains lack publicly available datasets, increasing the difficulty of applying supervised learning methods.

[0003] Existing remote supervision techniques offer advantages such as abundant data sources, the ability to utilize large-scale text data, and alignment and annotation with existing knowledge bases. This not only provides diverse and rich training data for text summarization generation but also helps models learn text features of different types and topics, improving the accuracy of summarization generation.

[0004] Furthermore, by combining a knowledge base, remote supervision technology provides the text summarization generation model with additional knowledge and background information, which helps the model better understand the semantics and context of the text, thereby generating more accurate and comprehensive summaries.

[0005] Remote supervision technology can also help models identify common patterns and structures in text, thereby better grasping the key points and information of the text, which is crucial for extracting key information from summaries;

[0006] However, while remote supervision techniques reduce data annotation costs, they also introduce the problem of mislabeling. One common approach to handling noisy sentences is to utilize knowledge graphs as a supplementary source of information, employing a dual-encoder architecture to encode the text and the knowledge graph separately. However, while this method combines information from both types of data, it fails to effectively establish a direct link between them. Summary of the Invention

[0007] The technical problem to be solved by this invention is to provide a knowledge-enhanced relation extraction method based on multimodal cross-attention. Natural language text and knowledge graph information are encoded separately, and bidirectional information transmission and fusion are achieved through cross-attention and cross-stitching mechanisms, thereby enhancing the connection between multimodal information and improving the accuracy and robustness of relation extraction.

[0008] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is: a knowledge enhancement relation extraction method based on multimodal cross-attention, comprising the following steps:

[0009] S1. Construct a relation extraction model, which includes two parts: global extraction and local extraction. Both parts include an embedding layer, a feature extraction layer, a cross-cooperative attention layer, and a prediction layer.

[0010] S2. Input multiple texts and knowledge graphs into the overall embedding layer and the local embedding layer respectively to obtain text vector representations and knowledge graph vector representations;

[0011] S3. Input the text vector representation and knowledge graph vector representation obtained from the overall extraction into two independent and identical attention modules respectively, and calculate the information between the overall text and the overall graph; input the text vector representation and knowledge graph vector representation obtained from the local extraction into two independent and identical CNN modules respectively, and calculate the information between the local text and the local graph.

[0012] S4. Input the information between the text as a whole and the information between the graph as a whole into the cross-cooperative attention module of the overall extraction part to obtain the overall text embedding and the overall graph embedding; input the information between the text local and the information between the graph local into the cross-cooperative attention module of the local extraction part (the same as in the overall extraction) to obtain the local text embedding and the local graph embedding.

[0013] S5. The overall text embedding and the local text embedding are concatenated, and the overall graph embedding and the local graph embedding are concatenated to obtain the sentence embedding set;

[0014] S6. Perform max pooling on all sentence embedding vectors for each dimension to obtain sentence bag embeddings;

[0015] S7. Use the Softmax function to calculate the probability that each element represents the corresponding relationship category;

[0016] S8. Obtain the probability distribution of the relationship categories and select the most likely relationship as the prediction result.

[0017] Preferably, in step S2, a component containing Sentences with 1 word element As input, and generate of Dimensional context representation; via connections from the last layer Embedding of the head and tail entities obtained and , and through the last layer The word representation obtained by mean pooling and max pooling and Construct a structure containing entity pairs Sentences The fixed-length representation; that is, the final representation of the input sentence is , where “;” indicates vector concatenation;

[0018] Knowledge graph encoders use triples As input, and generate , of Dimensional contextualization representation; then, from the last layer... of , and Used as head entity ,relation Tail-end entity Embedding; KG encoder vocabulary By all entities and relationships and a mask mark Composition, that is For simplicity, we assume that the text and the representation of the KG encoder have the same dimensions. That is, setting .

[0019] Preferably, the calculation formula for the attention module is as follows:

[0020] , , , The query, key, and value matrices in the global attention mechanism are generated from the input and capture global contextual dependencies through the global attention score matrix. The outputs of the local and global attention mechanisms are weighted and combined to achieve a comprehensive modeling of fine-grained and global semantic information. ;in, It is a learnable weight parameter used to balance the contributions of local and global attention to the final feature representation.

[0021] Preferably, the CNN module uses a one-dimensional convolutional neural network to extract text features. With spectral features For the vectors of each layer in the text encoder and knowledge graph encoder Perform convolution operations: ;in, It is a learnable parameter matrix. It is a bias term. The activation function is used for the convolution result. Max pooling is performed to obtain the features: The CNN layer uses two different filters. To extract multiple features and concatenate different outputs to form This serves as a representation of the user's comment; ultimately, the features are obtained, forming a matrix. .

[0022] Preferably, the cross-cooperative attention module is given a sentence. and corresponding knowledge graph triples The text encoder generates the representation of the sentence. The knowledge graph encoder generates the representation of triples. Then, the cross-attention in two directions is calculated: attention from the triple to the sentence ( ) and sentence-to-triple attention ( The calculation formula is as follows: , ;in, and It is a trainable linear transformation matrix; attention from triples to sentences. This indicates that the sentence representation is used to update the sentence representation. triples Embedding weight of each word element: ,in, and These are trainable parameters; next, the gating mechanism determines the representation of the original text. Contribution to the new hidden state of the text encoder: ,in The logical sigmoid function is represented; then, the original hidden state is interpolated. and triplet table To update the text encoder at the Hidden state of the layer: Information sharing from sentence to triplet direction is carried out in a similar manner: , , in, and It refers to the hyperparameter weights.

[0023] Preferably, in step S6, a sentence embedding set is defined. Max pooling is performed on all sentence embedding vectors for each dimension to obtain sentence bag embeddings. .

[0024] Preferably, in step S7, the sentence packet embedding vector is... Perform a linear transformation: in It is a vector of length C, representing the unnormalized score (logits) of each relation class; The input is fed into the Softmax function to obtain the probability of each relation category: in Indicates the first The probability of each relation category. Indicates the first The logit values ​​of each category are used to obtain a probability vector of length C. Each element represents the probability of the corresponding relationship category. The probability distribution of the relationship categories is obtained, and the most likely relationship is selected as the prediction result.

[0025] This invention provides a knowledge-enhanced relation extraction method based on multimodal cross-attention, which has the following beneficial effects:

[0026] 1. By employing a dual-encoder architecture and a cross-attention mechanism, textual and knowledge graph information are effectively integrated, significantly improving the accuracy and robustness of relation extraction;

[0027] 2. The model outperforms existing baseline models on multiple datasets, demonstrating particularly strong capabilities in handling noisy data and complex relation extraction tasks. Through ablation experiments, we validated the key contributions of the cross-stitching mechanism and the local and global attention mechanisms to model performance.

[0028] 3. It performs well in robustness analysis under different noise ratios, demonstrating the potential of the model in dealing with noise problems in remote supervision data;

[0029] 4. By using a multi-layered cross-attention mechanism, deep fusion between text and knowledge graph is achieved. This not only enhances the model's ability to capture fine-grained and global semantic information, but also provides a new method for future knowledge graph construction and relation extraction tasks.

[0030] 5. It reduces the cost of entity relationship extraction, improves efficiency and accuracy, and has broad application and promotion value. Attached Figure Description

[0031] The present invention will be further described below with reference to the accompanying drawings and embodiments:

[0032] Figure 1This is a flowchart of the method of the present invention;

[0033] Figure 2 This is a schematic diagram of the method of the present invention. Detailed Implementation

[0034] To address the problem that existing supervised learning methods heavily rely on high-quality labeled data, leading to high labeling costs and data scarcity in many fields, this invention provides a method based on a dual-encoder architecture. This method encodes natural language text and knowledge graph information separately, and achieves bidirectional information transfer and fusion through cross-attention and cross-stitching mechanisms. This enhances the connections between multimodal information and improves the accuracy and robustness of relation extraction. (Refer to...) Figure 1 The method of the present invention includes the following steps:

[0035] S1. Construct a relation extraction model. This model comprises two parts: overall extraction and local extraction. Both parts include an embedding layer, a feature extraction layer, a cross-cooperative attention layer, and a prediction layer. The structure of the model is described below. Figure 2 As shown.

[0036] S2. Input multiple texts and knowledge graphs into the overall embedding layer and the local embedding layer respectively to obtain text vector representations and knowledge graph vector representations.

[0037] Specifically, S2 includes the following steps:

[0038] S201, Preferably, the text encoder of this example receives a text containing... Sentences with 1 word element As input, and generate of Dimensional context representation.

[0039] [1] By connecting from the last layer Embedding of the head and tail entities obtained and , and through the last layer The word representation obtained by mean pooling and max pooling and Construct a structure containing entity pairs Sentences The fixed-length representation. That is, the final representation of the input sentence is... , where “;” indicates vector concatenation.

[0040] S202, Knowledge Graph Encoder uses triples As input, and generate , of Dimensional contextualization.

[0041] Will come from the last floor of , and Used as head entity ,relation Tail-end entity Embedded.

[0042] Vocabulary of a knowledge graph encoder By all entities and relationships and a mask mark Composition, that is .

[0043] For simplicity, this example assumes that the text and the KG encoder representation have the same dimensions. That is, setting .

[0044] S3. Input the text vector representation and knowledge graph vector representation obtained from the overall extraction into two independent and identical attention modules respectively, and calculate the information between the overall text and the overall graph; input the text vector representation and knowledge graph vector representation obtained from the local extraction into two independent and identical CNN modules respectively, and calculate the information between the local text and the local graph.

[0045] Specifically, S3 includes the following steps:

[0046] S301. Preferably, the holistic attention mechanism aims to capture global semantic information at the text or paragraph level, providing an understanding of context over a wider scope. This part utilizes a global attention network, enabling semantic aggregation across paragraphs or the entire document. This holistic representation can encompass greater contextual information, thereby supporting the model's task of extracting relationships across sentences or paragraphs.

[0047] The formula for calculating overall attention is as follows: . , , This represents the query, key, and value matrices in the overall attention mechanism. These matrices are generated from the input and capture global-scale contextual dependencies through the overall attention score matrix.

[0048] The outputs of local and global attention mechanisms are combined in a weighted manner to achieve a comprehensive modeling of fine-grained and global semantic information: .in, It is a learnable weight parameter used to balance the contributions of local and global attention to the final feature representation. This combination enables the model to model at both the detail and global levels simultaneously, thereby improving the understanding and extraction of complex relationships.

[0049] S302. Preferably, this example uses a CNN model for local feature extraction. The CNN model can extract local features well and has high training efficiency.

[0050] One-dimensional convolutional neural networks are used to extract text features. With spectral features .

[0051] For the vectors of each layer in the text encoder and knowledge graph encoder Perform convolution operations: .in, It is a learnable parameter matrix. It is a bias term. This is the activation function.

[0052] Convolution results Max pooling yields the following features: The CNN layer uses two different filters. To extract multiple features and concatenate different outputs to form , as a representation of the spectrum.

[0053] Finally, the features are obtained, forming a matrix. .

[0054] S4. Input the information between the whole text and the whole graph into the cross-cooperative attention module of the whole extraction part to obtain the whole text embedding and the whole graph embedding; input the information between the local text and the local graph into the cross-cooperative attention module of the local extraction part (the same as in the whole extraction) to obtain the local text embedding and the local graph embedding.

[0055] Preferably, this example employs a cross-stitch mechanism. This mechanism operates by mixing and updating the intermediate representations of the dual encoders, with the degree of mixing dynamically controlled through a cross-attention-based gating mechanism.

[0056] Given a sentence and corresponding knowledge graph triples The text encoder generates the representation of the sentence. The knowledge graph encoder generates the representation of triples. .

[0057] Calculate cross-attention in two directions: attention from triples to the sentence ( ) and sentence-to-triple attention ( The calculation formula is as follows: , .in, and It is a trainable linear transformation matrix.

[0058] Attention from triples to sentences This indicates that the sentence representation is used to update the sentence representation. triples Embedding weight of each word element: .in, and These are trainable parameters.

[0059] Gating mechanisms determine the representation of the original text. Contribution to the new hidden state of the text encoder: .in This represents the logical Sigmoid function.

[0060] By interpolating the original hidden state and triplet table To update the text encoder at the Hidden states of layers: .

[0061] Information sharing from sentence to triplet direction is carried out in a similar manner: , , .in, and It refers to the hyperparameter weights.

[0062] S5. The overall text embedding and the local text embedding are concatenated, and the overall graph embedding and the local graph embedding are concatenated to obtain the sentence embedding set.

[0063] S6. Perform max pooling on all sentence embedding vectors for each dimension to obtain sentence packet embeddings.

[0064] Define a sentence embedding set ;

[0065] Max pooling is performed on all sentence embedding vectors for each dimension to obtain sentence bag embeddings. .

[0066] S7. Use the Softmax function to calculate the probability that each element represents the corresponding relationship category.

[0067] Preferably, the sentence packet embedding vector Perform a linear transformation: in It is a vector of length C, representing the unnormalized score (logits) of each relation class.

[0068] Will The input is fed into the Softmax function to obtain the probability of each relation category: in Indicates the first The probability of each relation category. Indicates the first The logit value for each category.

[0069] We obtain a probability vector of length C. Each element represents the probability of the corresponding relationship category.

[0070] S8. Obtain the probability distribution of the relationship categories and select the most likely relationship as the prediction result.

[0071] To verify the effectiveness of the knowledge enhancement relation extraction method based on multimodal cross-attention provided by this invention, Tables 1 and 2 show the comparison between this invention and other existing models on different datasets.

[0072] Table 1. Comparison of the present invention with other existing models on the NYT10 dataset.

[0073] Model Accuracy Precision Recall F1 score PCNN + ATT 0.341 0.636 0.708 0.710 BRE + CE 0.632 0.880 0.737 0.740 Cross-stitched Encoders 0.705 0.944 0.763 0.767 This invention 0.713 0.953 0.753 0.772

[0074] Table 2. Comparison of the present invention with other existing models on the WebNLG dataset.

[0075] Model Accuracy Precision Recall F1 score PCNN + ATT 0.435 0.627 0.730 0.735 BRE + CE 0.645 0.877 0.762 0.765 Cross-stitched Encoders 0.723 0.932 0.785 0.789 MLCA-KRE （ ours ） 0.732 0.941 0.788 0.792

[0076] As shown in Table 1, the present invention achieved an F1-score of 0.772 on the NYT10 dataset. This indicates that the present invention performs well in handling noisy remote supervision data and can better combine text and knowledge graph information for effective relation extraction. On the WebNLG dataset, the present invention achieved an F1-score of 0.792, outperforming other benchmark models, demonstrating that the model effectively handles multimodal information in generation tasks, especially performing excellently in matching text with knowledge base triples.

[0077] To investigate the effectiveness of each module in this invention, the cross-stitching mechanism and local and global attention mechanisms were removed, and ablation experiments were conducted. Table 3 shows the results of the ablation experiments of this invention on the NTY10 dataset.

[0078] Table 3. Results of ablation experiments using the NTY10 dataset in this invention.

[0079] Model Accuracy Precision Recall F1 score All modules (complete model) 0.711 0.953 0.753 0.772 Remove cross-stitch mechanism 0.655 0.915 0.710 0.712 Remove local and overall attention 0.637 0.922 0.718 0.720

[0080] Experimental results show that removing the cross-stitching mechanism leads to a 6.0% performance decrease, while removing local and global attention results in a 5.2% performance decrease. This indicates that the cross-attention module and the local and global attention module added in this invention have an improving effect.

[0081] In summary, the knowledge-enhanced relation extraction method based on multimodal cross-attention described in this invention effectively integrates textual and knowledge graph information through a dual-encoder architecture and a cross-attention mechanism, significantly improving the accuracy and robustness of relation extraction. Experimental results show that this invention outperforms existing baseline models on multiple datasets, particularly demonstrating strong capabilities in handling noisy data and complex relation extraction tasks. Ablation experiments validated the key contributions of the cross-stitching mechanism and local and global attention mechanisms to model performance. Furthermore, this invention exhibits excellent robustness under different noise ratios, demonstrating its potential in addressing noise issues in remotely supervised data. The innovation of this research lies in achieving deep integration between text and knowledge graphs through a multi-level cross-attention mechanism. This not only enhances the model's ability to capture fine-grained and global semantic information but also provides a new method for future knowledge graph construction and relation extraction tasks. Future research can further optimize the design of the cross-attention mechanism, explore more efficient noise processing methods, and extend the invention to more application scenarios.

[0082] The above embodiments are merely preferred technical solutions of the present invention and should not be considered as limitations on the present invention. The scope of protection of the present invention should be limited to the technical solutions described in the claims, including equivalent substitutions of the technical features described in the claims. That is, equivalent substitutions and improvements within this scope are also within the scope of protection of the present invention.

Claims

1. A knowledge-enhanced relation extraction method based on multimodal cross-attention, characterized in that, Includes the following steps: S1. Construct a relation extraction model, which includes two parts: global extraction and local extraction. Both parts include an embedding layer, a feature extraction layer, a cross-cooperative attention layer, and a prediction layer. S2. Input multiple texts and knowledge graphs into the overall embedding layer and the local embedding layer respectively to obtain text vector representations and knowledge graph vector representations; S3. Input the text vector representation and knowledge graph vector representation obtained from the overall extraction into two independent and identical attention modules respectively, and calculate the information between the overall text and the overall graph; input the text vector representation and knowledge graph vector representation obtained from the local extraction into two independent and identical CNN modules respectively, and calculate the information between the local text and the local graph. S4. Input the information between the overall text and the overall graph into the cross-cooperative attention module of the overall extraction part to obtain the overall text embedding and the overall graph embedding; input the information between the local text and the local graph into the cross-cooperative attention module of the local extraction part to obtain the local text embedding and the local graph embedding. The cross-cooperative attention module is given a sentence and corresponding knowledge graph triples The text encoder generates the representation of the sentence. The knowledge graph encoder generates the representation of triples. Then, the cross-attention in two directions is calculated: attention from the triple to the sentence ( ) and sentence-to-triple attention ( The calculation formula is as follows: , ;in, and It is a trainable linear transformation matrix; attention from triples to sentences. This indicates that the sentence representation is used to update the sentence representation. triples Embedding weight of each word element: ,in, and These are trainable parameters; next, the gating mechanism determines the representation of the original text. Contribution to the new hidden state of the text encoder: ,in The logical sigmoid function is represented; then, the original hidden state is interpolated. and triplet table To update the text encoder at the Hidden states of layers: Information sharing from sentence to triplet direction is carried out in a similar manner: , , in, and These are hyperparameter weights; S5. The overall text embedding and the local text embedding are concatenated, and the overall graph embedding and the local graph embedding are concatenated to obtain the sentence embedding set; S6. Perform max pooling on all sentence embedding vectors for each dimension to obtain sentence bag embeddings; S7. Use the Softmax function to calculate the probability that each element represents the corresponding relationship category; S8. Obtain the probability distribution of the relationship categories and select the most likely relationship as the prediction result.

2. The knowledge enhancement relation extraction method based on multimodal cross-attention according to claim 1, characterized in that, In step S2, a containing Sentences with 1 word element As input, and generate of Dimensional context representation; via connections from the last layer Embedding of the head and tail entities obtained and And through the last layer The word representation obtained by mean pooling and max pooling and Construct a structure containing entity pairs Sentences The fixed-length representation; that is, the final representation of the input sentence is , where ";" indicates vector concatenation; Knowledge graph encoders use triples As input, and generate , of Dimensional contextualization; Then, from the last layer of , and Used as head entity ,relation Tail-end entity Embedding; KG encoder vocabulary By all entities and relationships and a mask mark Composition, that is For simplicity, we assume that the text and the representation of the KG encoder have the same dimensions. That is, setting .

3. The knowledge enhancement relation extraction method based on multimodal cross-attention according to claim 1, characterized in that, The calculation formula for the attention module is as follows: The query, key, and value matrices in the global attention mechanism are generated from the input and capture global contextual dependencies through the global attention score matrix. The outputs of the local and global attention mechanisms are weighted and combined to achieve a comprehensive modeling of fine-grained and global semantic information. ;in, It is a learnable weight parameter used to balance the contributions of local and global attention to the final feature representation.

4. The knowledge enhancement relation extraction method based on multimodal cross-attention according to claim 1, characterized in that, The CNN module uses a one-dimensional convolutional neural network to extract text features. With spectral features For the vectors of each layer in the text encoder and knowledge graph encoder Perform convolution operations: ;in, It is a learnable parameter matrix. It is a bias term. The activation function is used for the convolution result. Max pooling is performed to obtain the features: The CNN layer uses two different filters. To extract multiple features and concatenate different outputs to form This serves as a representation of user reviews; ultimately, features are obtained and a matrix is ​​formed. .

5. The knowledge enhancement relation extraction method based on multimodal cross-attention according to claim 1, characterized in that, In step S6, a sentence embedding set is defined. Max pooling is performed on all sentence embedding vectors for each dimension to obtain sentence bag embeddings. .

6. The knowledge-enhanced relation extraction method based on multimodal cross-attention according to claim 1, characterized in that, In step S7, the sentence packet embedding vector is... Perform a linear transformation: in It is a vector of length C, representing the unnormalized score (logits) of each relation class; The input is fed into the Softmax function to obtain the probability of each relation category: in Indicates the first The probability of each relation category. Indicates the first The logit values ​​of each category are used to obtain a probability vector of length C. Each element represents the probability of the corresponding relationship category. The probability distribution of the relationship categories is obtained, and the most likely relationship is selected as the prediction result.

Images