Multimodal entity relationship extraction method based on hypergraph neural network

Abstract

The invention relates to the technical field of natural language processing, in particular to a hypergraph neural network-based multi-modal entity relationship extraction method, which comprises the following steps of: acquiring a text-image pair and inputting the text-image pair into a preset hypergraph construction model, acquiring node characteristics and a multi-modal hypergraph structure of the text-image pair, the hypergraph structure comprises a group of hyperedge sets, each hyperedge in the hyperedge sets can be connected with a plurality of nodes at the same time, and the number of the nodes connected with the hyperedges is not limited; the node features and the multi-modal hypergraph structure are input into a preset hypergraph neural network model, the multi-modal entity relation in the text-image pair is output, and the hypergraph neural network model is used for analyzing hypergraph node information propagation from the semantic and contextual relation and giving corresponding weights to semantic information between modals in combination with an attention mechanism. According to the method, the multi-modal information can be effectively fused, and the entity category and the entity relationship of the multi-modal information can be identified.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a multimodal entity relationship extraction method based on a hypergraph neural network. Background Art

[0002] In an era of rapid development of information technology and surging data volume, the phenomenon of information explosion has become increasingly prominent. The massive amount of data generated every day covers various forms such as text, images, audio and video. How to extract valuable information efficiently and accurately has become a major challenge in today's society. To address this problem, multimodal information processing technology has emerged to solve the need for information extraction in complex data. Among them, multimodal named entity recognition (MNER) and multimodal relation extraction (MRE) are two core tasks that play an important role. The MNER task aims to accurately identify named entities from inputs containing different modalities (such as text and images), while MRE further identifies the relationships between entities. By combining information from both text and image modalities, it is possible to more accurately identify known category entities in text, such as names of people (PER), places (LOC) and organizations (ORG), and extract relationships between them, such as family relationships and subordinate relationships.

[0003] In recent years, the research on MNER and MRE has mainly focused on extracting information from text and images. Based on the extraction of text entity relations, images are introduced as auxiliary clues to deepen the understanding of text information. Early methods usually encode images and text separately, and then perform cross-modal information fusion and entity relationship extraction. Multimodal pre-training models such as CLIP and BLIP align text and image information, achieving higher accuracy than traditional methods. Although the strategy of aligning image and text information is effective, it is often difficult to fully utilize the rich information in the image and is also affected by noise information. Some studies have proposed a strategy to obtain object-level features by segmenting fine-grained targets within the modality, and then combine graph-based or transformer-based models to fuse multimodal information. Integrating object-level image information with text data can better identify image targets that are aligned with text or multimodal requirements and effectively filter out irrelevant noise.

[0004] However, in real scenarios, images and texts may have complementary or contradictory relationships, which may lead to a decrease in model training effect. In addition, existing methods usually rely on surface similarity or simple feature splicing to achieve cross-modal alignment, which makes it difficult for them to capture the deep semantic correspondence between text and images. Therefore, the present invention proposes a multimodal entity relationship extraction method based on a hypergraph neural network. Summary of the invention

[0005] The purpose of the present invention is to provide a multimodal entity relationship extraction method based on a hypergraph neural network, which can effectively integrate multimodal information, identify entity categories and entity relationships of multimodal information, and improve model accuracy and robustness.

[0006] To achieve the above object, the present invention provides the following solutions:

[0007] A multimodal entity relationship extraction method based on a hypergraph neural network, comprising:

[0008] Get text-image pairs;

[0009] Inputting the text-image pair into a preset hypergraph construction model to obtain node features and a multimodal hypergraph structure of the text-image pair, wherein the hypergraph construction model is used to extract fine-grained node features from text and images, analyze the relationship between nodes through the node features and express it as a multimodal hypergraph structure, the multimodality includes text and image, and the hypergraph structure includes a set of hyperedge sets, each hyperedge in the hyperedge set can connect multiple nodes at the same time, and the number of nodes connected by the hyperedge is not limited;

[0010] The node features and multimodal hypergraph structure are input into a preset hypergraph neural network model, and the multimodal entity relationship in the text-image pair is output, wherein the hypergraph neural network model is used to analyze the hypergraph node information propagation from the perspective of semantics and contextual relationships, and combines the attention mechanism to assign corresponding weights to the semantic information between modalities.

[0011] Optionally, the hypergraph construction model includes a feature extraction module and a hypergraph construction module, the feature extraction module is used to extract node features of the text-image pair, and obtain a multimodal semantic hyperedge set and a contextual hyperedge set based on the node features; the hypergraph construction module is used to construct the multimodal hypergraph structure based on the node features, semantic hyperedges, and contextual hyperedges.

[0012] Optionally, extracting node features of the text-image pair includes:

[0013] The text-image pair is parsed to obtain text node features and image node features respectively, wherein the text node features include words, phrases to which the words belong, and global information, and the image node features include visual targets and visual relationships between visual targets.

[0014] Optionally, acquiring a multimodal semantic hyperedge set and a contextual hyperedge set according to the node features includes:

[0015] Comparing semantic similarity within and between modalities according to the text node features and the image node features, respectively, to construct the semantic hyperedge set;

[0016] The text context structure information is extracted according to the text node features and the text grammar dependency tree, and the context hyperedge set is constructed in combination with the visual targets and the visual relations between the visual targets.

[0017] Optionally, comparing semantic similarities within a modality and between modalities according to the text node features and the image node features, respectively, constructing the semantic hyperedge set includes:

[0018] Calculate the cosine similarity matrix of the text node features and image node features and cross-modalities respectively, and connect semantic hyperedges according to the cosine similarity matrix, wherein each node in the modality is connected to the top k nodes with the highest similarity to itself. n nodes, forming a hyperedge centered on the current node, and obtaining the text and image semantic hyperedge set matrix; cross-modal information is centered on the text node, and the k nodes with the highest similarity to the central node are selected. m image nodes, connecting the image nodes as the similarity hyperedge set matrix of the text-to-image correspondence;

[0019] Based on the text and image semantic hyperedge set matrix and the similarity hyperedge set matrix of the text corresponding image, a semantic hypergraph association matrix is ​​constructed to obtain the semantic hyperedge set.

[0020] Optionally, extracting text context structure information according to the text node features and the text grammar dependency tree, and combining visual targets and visual relationships between visual targets to construct the context hyperedge set includes:

[0021] Take words as text nodes, grammatical relations between words and phrase combinations as the correspondence between nodes and edges, build a text structure hypergraph association matrix, and obtain the text context hyperedge set;

[0022] Taking visual targets as nodes and visual relationships between visual targets as the correspondence between nodes and edges, a visual structure hypergraph association matrix is ​​constructed to obtain the visual context hyperedge set.

[0023] Optionally, constructing the multimodal hypergraph structure based on the node features, semantic hyperedges, and contextual hyperedges includes:

[0024] The semantic hypergraph association matrix, the text structure hypergraph association matrix, and the visual structure hypergraph association matrix are integrated into a hypergraph association matrix, and the multimodal hypergraph structure is constructed according to the hypergraph association matrix, wherein the degree of a node in the multimodal hypergraph structure is the number of hyperedges in which each node participates, and the degree of a hyperedge is the number of nodes connected by each hyperedge.

[0025] Optionally, the hypergraph neural network model includes a feature aggregation module and a relationship classification module, the feature aggregation module is used to obtain the hyperedge feature information in the multimodal hypergraph structure and feed it back to the node to update the node feature; the relationship classification module is used to aggregate the updated node features and extract the relationship between entities.

[0026] Optionally, obtaining hyperedge feature information in the multimodal hypergraph structure includes:

[0027] Aggregate the feature information in nodes using semantic hyperedges as units, and introduce an attention mechanism to adjust the weights of nodes in different contexts to obtain the semantic hyperedge feature matrix;

[0028] The feature information of nodes is aggregated using contextual hyperedges as units, and a spectral convolutional network is used to aggregate the feature information of nodes on hyperedges to obtain the contextual hyperedge feature matrix.

[0029] Optionally, aggregating the updated node features to extract the relationship between entities includes:

[0030] Perform entity category analysis through updated node features to obtain predicted entity categories;

[0031] Aggregate the updated node features to obtain the node features of the predicted entity;

[0032] The relationship between the predicted entity and the target entity is extracted based on the node features of the predicted entity.

[0033] The beneficial effects of the present invention are:

[0034] The present invention proposes a multimodal entity relationship extraction method based on a hypergraph neural network, which connects multimodal nodes from the perspective of semantic and contextual relationships, integrates information in units of hyperedges, uses the attention mechanism to enhance the aggregation ability of semantic hyperedges, flexibly divides the weights of different modal information in tasks, reduces the interference of irrelevant information and noise information, and thus improves the ability of hypergraph networks to extract multimodal entity relationships. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0036] Figure 1 A schematic diagram of a hypergraph construction model workflow according to an embodiment of the present invention;

[0037] Figure 2A schematic diagram of the working process of a feature aggregation module of a hypergraph neural network model according to an embodiment of the present invention;

[0038] Figure 3 Schematic diagram of the working process of the hypergraph neural network model relationship classification module of an embodiment of the present invention. DETAILED DESCRIPTION

[0039] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0041] Some studies have proposed strategies to obtain object-level features by segmenting fine-grained targets within the modality, and then combine graph-based or transformer-based models to fuse multimodal information. Integrating object-level image information with text data can better identify image targets that are aligned with text or multimodal requirements and effectively filter out irrelevant noise. Sun et al. proposed a BERT-based cross-modal classifier that aims to integrate and filter multimodal overall information at the same time to reduce potential interference. At the same time, Cui et al. introduced the information bottleneck theory, which is used to filter irrelevant information to improve the efficiency and accuracy of the model. Chen et al. tried to use a hierarchical multimodal fusion framework to reduce the impact of irrelevant information on the final result.

[0042] In addition, Wang et al. explored methods to collect relevant information from rich text sources such as Wikipedia to further enhance the model's knowledge base and contextual understanding capabilities. This strategy provides more comprehensive information support for multimodal tasks. Hu et al. used cross-modal retrieval (Google Vision APIs and web crawlers) to obtain information, not only retrieving text, but also retrieving visual and textual information related to objects, sentences, and entire images. The retrieved visual and textual information is combined for relational reasoning. This method can effectively select and compare evidence from different modalities, but the disadvantage is that the retrieval results will have a great impact on the model and are restricted in many application scenarios. The dataset tested in this paper is constructed by annotating social media information, which has a great advantage in Google retrieval, and the actual use of the model is limited. Zheng et al. borrowed the cross-language difference problem in machine translation and worked on solving the misalignment problem between text and images. They significantly improved the performance of the model by creating pseudo-parallel corpus pairs using a high-resource corpus and a diffusion-based generative model. These studies demonstrate the great potential of multimodal learning in information extraction, emphasize the importance of combining images with text, and provide new directions for future research.

[0043] From the above content, it can be seen that in actual scenarios, images and texts may have a complementary or contradictory relationship, which may lead to a decrease in the model training effect, and existing methods usually rely on surface similarity or simple feature splicing to achieve cross-modal alignment, which makes it difficult for them to capture the deep semantic correspondence between text and image. In order to address these challenges, this embodiment proposes a method for representing fused multimodal information as a hypergraph structure. Different from traditional binary graph structures and fully connected structures, hypergraphs can reduce the spatiotemporal complexity of information fusion while ensuring information propagation. In this embodiment, images and texts are decomposed into fine-grained nodes containing information at all levels, and the nodes are connected by combining the semantic similarity between modalities and the contextual relationship within the modality to construct a hypergraph to form a multimodal entity and relationship representation with more semantic understanding ability and fine-grained alignment.

[0044] In addition, existing methods lack the ability to flexibly adjust the dynamic weights of different modal information when fusing fine-grained multimodal information, which may cause important visual or textual information to be ignored or underestimated. In order to solve this problem, this embodiment constructs a multimodal hypergraph neural network based on the multimodal hypergraph structure, analyzes the propagation of hypergraph node information from the semantic and contextual relationships, and explores the weight of semantic information between modalities through the attention mechanism. This mechanism can not only improve the accuracy of entity relationship extraction, but also better capture the complex interactive relationships between different modalities, further enhance the robustness and practicality of the model, and reduce the noise interference of irrelevant information.

[0045] This embodiment provides a multimodal entity relationship extraction method based on a hypergraph neural network, including:

[0046] Get text-image pairs;

[0047] Input the text-image pair into a preset hypergraph construction model to obtain the node features and multimodal hypergraph structure of the text-image pair, wherein the hypergraph construction model is used to extract fine-grained node features from the text and the image, analyze the relationship between the nodes through the node features and express it as a multimodal hypergraph structure, the multimodality includes text and image, and the hypergraph structure includes a set of hyperedge sets, each hyperedge in the hyperedge set can connect multiple nodes at the same time, and the number of nodes connected by the hyperedge is not limited;

[0048] The node features and multimodal hypergraph structure are input into the preset hypergraph neural network model, and the multimodal entity relationships in the text-image pairs are output. The hypergraph neural network model is used to analyze the information propagation of hypergraph nodes from the perspective of semantics and contextual relationships, and the attention mechanism is used to assign corresponding weights to the semantic information between modalities.

[0049] Specifically, this embodiment connects multimodal nodes from the perspective of semantics and contextual relationships, integrates information in units of hyperedges, uses the attention mechanism to enhance the aggregation ability of semantic hyperedges, flexibly divides the weights of different modal information in tasks, and reduces the interference of irrelevant and noise information, thereby improving the ability of hypergraph networks to extract multimodal entity relationships.

[0050] The following is a detailed description of the method and process for implementing the functions of each module in the hypergraph construction model and the hypergraph neural network model in this embodiment, as follows:

[0051] The hypergraph construction model includes a feature extraction module and a hypergraph construction module. The feature extraction module is used to extract node features of text-image pairs and obtain multimodal semantic hyperedge sets and contextual hyperedge sets based on node features; the hypergraph construction module is used to construct a multimodal hypergraph structure based on node features, semantic hyperedges, and contextual hyperedges. The specific workflow is as follows: Figure 1 shown.

[0052] Furthermore, extracting node features of text-image pairs includes:

[0053] The text-image pair is parsed to obtain text node features and image node features respectively. The text node features include words, phrases to which the words belong, and global information, and the image node features include visual targets and visual relationships between visual targets.

[0054] Specifically, this embodiment extracts fine-grained node features from text-image. The text takes words as the basic unit, including the features of words and their phrases and sentences. The image takes target entities as the basic unit, and extracts the features of the candidate boxes of the cut target entity images.

[0055] In order to make full use of the information in text and images, targets that can be used as entities are extracted from text and images as nodes. The original text and image are parsed using the text scene graph and scene graph generation network RelTR algorithm to generate ordinary graph structures TSG and VSG containing fine-grained nodes and original contextual relationships. TSG and VSG contain the preliminary parsed node divisions and contextual relationships between nodes.

[0056] TSG nodes are words in the original text. Considering that the semantics of words depend on contextual information, the initial features of text nodes include not only words but also the phrases to which the words belong and global information. The text feature matrix X is constructed. t VSG contains the obvious visual targets identified from the original image and the visual relationships between them. This embodiment extracts the image features of the visual targets as fine-grained nodes of the image and constructs the visual feature matrix X v . Connect the text and image node features to form a hypergraph node feature matrix X = [X t ,X v ] T .

[0057] Furthermore, based on the node features, semantic hyperedges are constructed by comparing the semantic similarity between modalities. At the same time, this module can also obtain contextual hyperedges based on the context of text nodes or the visual context of image target nodes, including:

[0058] Compare semantic similarity within and between modalities based on text node features and image node features, and construct semantic hyperedge sets;

[0059] The text context structure information is extracted based on the text node features and the text grammatical dependency tree, and the context hyperedge set is constructed by combining the visual targets and the visual relations between the visual targets.

[0060] Among them, the semantic similarity is compared within the modality and between the modalities according to the text node features and the image node features, and the semantic hyperedge set is constructed, including:

[0061] Calculate the cosine similarity matrix of text node features and image node features and cross-modal features respectively, and connect semantic hyperedges according to the cosine similarity matrix. In which, each node in the modality is connected to the top k nodes with the highest similarity to itself. nnodes, forming a hyperedge centered on the current node, and obtaining the text and image semantic hyperedge set matrix; cross-modal information is centered on the text node, and the k nodes with the highest similarity to the central node are selected. m image nodes, connecting the image nodes as the similarity hyperedge set matrix of the text-to-image correspondence;

[0062] Based on the text and image semantic hyperedge set matrix and the similarity hyperedge set matrix of the text corresponding image, a semantic hypergraph association matrix is ​​constructed to obtain the semantic hyperedge set.

[0063] Among them, the text context structure information is extracted according to the text node features and the text grammar dependency tree, and the context hyperedge set is constructed by combining the visual targets and the visual relationship between the visual targets, including:

[0064] Take words as text nodes, grammatical relations between words and phrase combinations as the correspondence between nodes and edges, build a text structure hypergraph association matrix, and obtain the text context hyperedge set;

[0065] Taking visual targets as nodes and visual relationships between visual targets as the correspondence between nodes and edges, a visual structure hypergraph association matrix is ​​constructed to obtain the visual context hyperedge set.

[0066] Specifically, we use the pre-trained CLIP as an encoder to encode the fine-grained feature nodes in text and images and project them into the same embedding space. We construct hyperedge sets from the perspectives of semantic similarity and context structure. In order to ensure semantic interaction between modalities, we compare semantic similarity within and between modalities, and construct semantic hyperedge sets within and between modalities.

[0067] Calculate the cosine similarity matrix of text, image and cross-modality respectively, connect semantic hyperedges according to the cosine similarity matrix, and connect each node in the modality to the top k nodes with the highest similarity to itself. n nodes, forming a hyperedge centered on the current node, and obtaining the text and image semantic hyperedge set matrix and Cross-modal information is centered on the text node, and the k nodes with the highest similarity to the central node are selected. m image nodes, connecting the image nodes as the similarity hyperedge set matrix of the text-to-image

[0068] Finally, the three hypergraph matrices are integrated together to obtain the semantic hypergraph association matrix The distance between nodes in each hyperedge is converted into weights through the Gaussian kernel algorithm.

[0069] The contextual structure information of text and image comes from TSG and VSG, which converts the ordinary graph structure into a hypergraph connection matrix, represented as ntsg is the number of hyperedges transformed in the semantic dependency tree.

[0070] The structural information of image nodes is converted into scene graphs in the same way. The visual targets are used as nodes, and the visual relationships between nodes are converted into the corresponding relationships between nodes and edges, which are expressed as and m vsg is the number of hyperedges converted. Integrate the two hypergraph matrices to obtain the structural hypergraph association matrix The weight of each hyperedge is an adjustable parameter α, and the hyperedge weight matrix is ​​represented by W g =diag[α].

[0071] Furthermore, constructing a multimodal hypergraph structure based on node features, semantic hyperedges, and contextual hyperedges includes:

[0072] The semantic hypergraph association matrix, text structure hypergraph association matrix, and visual structure hypergraph association matrix are integrated into a hypergraph association matrix, and a multimodal hypergraph structure is constructed based on the hypergraph association matrix. The degree of a node in the multimodal hypergraph structure is the number of hyperedges in which each node participates, and the degree of a hyperedge is the number of nodes connected by each hyperedge.

[0073] Specifically, in this embodiment, the multimodal hypergraph is defined as follows: Given an original text-image pair (T, V), the multimodal hypergraph constructed by the model is represented as HG = (X, E, H), where X = {x1, x2, ..., x n} represents the node set of the hypergraph, n represents the number of nodes, E = {e1,e2,...,e m} represents the hyperedge set, m represents the number of hyperedges, and the incidence matrix H∈R n×m Represents the connection between nodes and hyperedges. The elements in H are defined as follows:

[0074]

[0075] Based on the node information parsed by TSG and VSG, further connect the nodes to construct a hypergraph. According to the semantic hypergraph association matrix and the structural hypergraph association matrix, and the hyperedges and hyperedge weights from the semantic and contextual relationships are integrated to achieve a complete hypergraph structure and realize multimodal information fusion. The hypergraph association matrix is ​​finally expressed as H = [H s ,H g ], the degree of a hypergraph node is the number of hyperedges that each node participates in. Suppose node x i Connected on multiple hyperedges {e1,e2,...,e k} Considering the weight of the hyperedge to calculate the node degree, it is defined as The degree of each hyperedge is the number of nodes connected by the hyperedge, defined as D X and D E are diagonal matrices representing node degrees and edge degrees respectively.

[0076] Furthermore, the hypergraph neural network model includes a feature aggregation module and a relationship classification module. The feature aggregation module is used to obtain the hyperedge feature information in the multimodal hypergraph structure and feed it back to the node to update the node features; the relationship classification module is used to aggregate the updated node features and extract the relationship between entities. The specific workflow is as follows: Figure 2 , Figure 3 shown.

[0077] Furthermore, obtaining the hyperedge feature information in the multimodal hypergraph structure includes:

[0078] Aggregate the feature information in nodes using semantic hyperedges as units, and introduce an attention mechanism to adjust the weights of nodes in different contexts to obtain the semantic hyperedge feature matrix;

[0079] The feature information of nodes is aggregated using contextual hyperedges as units, and a spectral convolutional network is used to aggregate the feature information of nodes on hyperedges to obtain the contextual hyperedge feature matrix.

[0080] Specifically, the feature aggregation module in this embodiment uses two convolutional layers, a node-to-hyperedge convolutional layer and a hyperedge-to-node convolutional layer. Hyperedge aggregation originates from node information, can capture high-order hidden representations of original data, and backpropagate this information back to the node.

[0081] The way information is propagated determines the principle of information aggregation through hyperedges. The hypergraph constructed in this embodiment fully models the cross-modal hypergraph from the perspectives of semantics and contextual relationships, and explores the potential interactions between the two modalities. Semantic hyperedges aggregate the representation information from nodes, while contextual hyperedges combine their own transmission paths to provide a structured information propagation path within the same modality.

[0082] When aggregating these two types of hyperedge information, different methods are used to separate and process them. Since the semantic contribution of a node often depends on the context, an attention mechanism is introduced when aggregating semantic hyperedge features to adaptively adjust the weight of each node in different contexts. The semantic hyperedge feature is defined as:

[0083]

[0084] Aggregate semantic hyperedge feature matrix:

[0085]

[0086]

[0087]

[0088] in, and X l are the semantic hyperedge and node features of the lth layer, W K , W Q and W V It is a trainable attention matrix that can more accurately reflect the semantic contribution of nodes by using semantic attention.

[0089] Contextual relationship hyperedges use spectral convolutional networks to aggregate the features of adjacent nodes to extract global contextual relationships:

[0090]

[0091]

[0092] in, and X l It is the contextual relationship hyperedge and node features of the lth layer.

[0093] In order to obtain complete hyperedge feature information, the two types of hyperedge features are concatenated. Subsequently, this information is back-propagated back to the node through the hyperedge, defined as:

[0094]

[0095] The hypergraph separation process serves a dual purpose: extracting semantic consistency across modal instances and modality-specific semantics from different hypergraphs. At the same time, it ensures that complex data from these changing hypergraphs is preserved by vertices, minimizing the possibility of significant information loss.

[0096] Furthermore, the updated node features are aggregated to extract the relationship between entities including:

[0097] Perform entity category analysis through updated node features to obtain predicted entity categories;

[0098] Aggregate the updated node features to obtain the node features of the predicted entity;

[0099] The relationship between the predicted entity and the target entity is extracted based on the node features of the predicted entity.

[0100] Specifically, this embodiment constructs different classifier layers for MNER and MRE respectively based on the latest hypergraph node features and high-order aggregation information.

[0101] The model uses CRF as a decoder to perform named entity recognition tasks. CRF models the dependency between input features and labels, and finds the most likely label y′ based on the predicted probability of node features and the transition probability between entity labels:

[0102]

[0103] Among them, P(y|x) is the joint probability of the input feature x and the label sequence y. During the training process, the model calculates the loss through the CRF log_likelihood method to measure the difference between the label sequence output by the model and the actual label sequence:

[0104] Loss_mner=-logP(y ture |X) (10)

[0105] The goal of the relationship extraction task is to predict the relationship between the main entity and the target entity. The classifier extracts the entity node features that need to predict the relationship, and aggregates the node features to form the entity matrix X E , use the softmax function to calculate the probability distribution of relationship categories:

[0106] y′=soft(X E ) (11)

[0107] Define the cross quotient loss function:

[0108]

[0109] Among them, y ture is the encoding of the true relationship category, y′ is the classification probability output by the model, the gradient of the loss relative to the model parameters is calculated, and the parameters are updated.

[0110] Experimental verification:

[0111] (1) Indicators and datasets:

[0112] Three publicly available datasets are used to evaluate the multimodal named entity recognition (MNER) and multimodal relation extraction (MRE) methods proposed in this embodiment, including two MNER datasets: Twitter-2015 and Twitter-2017, and a manually annotated MRE task dataset MNRE. These datasets are collected from multimodal posts on Twitter, each Twitter post consists of a text and a picture to simulate real-life usage scenarios. Both Twitter-2015 and Twitter-2017 contain four types of entities: person (PER), location (LOC), organization (ORG), and miscellaneous (MISC). Table 1 shows the number of entities of each type and the division of multimodal tweets in the training, development, and test sets of the two datasets. Table 2 shows the statistics of the MNRE dataset, which contains 9201 sentence-image pairs and 23 relation categories.

[0113] Table 1

[0114]

[0115] Table 2

[0116]

[0117] We use precision, recall, and F1-score as evaluation metrics and compare these results in the subsequent sections.

[0118] (2) Implementation details:

[0119] Experiments were conducted using the PyTorch framework on two Nvidia 4090 GPUs. The node feature dimension size was set to 1536. All optimizations used the AdamW optimizer with a decay rate of 0.1, a learning rate of 1e-4, and a batch size of 32. Since the number of nodes depends on the multimodal data, the number of semantic node connections k n and k m It is not fixed, but is determined by the ratio of the number of nodes extracted from the current dataset. The final result is based on a 50% ratio, which means that each central node is connected to the top 50% of the nodes in the similarity ranking. In addition, in the experimental results and subsequent ablation experiments, the weight of the contextual hyperedge is set to α = 1.

[0120] (3) Results:

[0121] To demonstrate the effectiveness of our method and model, we compare them with several state-of-the-art text-based and multimodal methods.

[0122] Text-based models: Classic text-based models such as CNN-BiLSTM-CRF and BERT-CRF, as well as the BERT-based MTB framework are evaluated for named entity recognition and relation extraction, respectively.

[0123] Text-Image Models: UMT aligns fine-grained visual object features with textual representations, which is suitable for multimodal tasks. UMGF introduces a unified multimodal graph fusion method for MNER, while MEGA adopts dual scene graphs for MRE. MoRe utilizes relevant documents retrieved from Wikipedia as auxiliary representations of images and text. MMIB introduces a multimodal information bottleneck to address modal noise and gap issues by utilizing refinement and alignment regularizers. SMNER improves multimodal alignment under global and local strategies, while TMR enhances entity and relation extraction using back translation and divergence estimation, treating multimodal alignment as cross-lingual translation. In addition, the effectiveness of pre-trained visual-text models such as CLIP, BLIP, and UPA on multimodal tasks is evaluated.

[0124] The performance summary of this model and the baseline model on the three test sets is shown in Table 3.

[0125] First, our model significantly outperforms text-only models by 15-20 percentage points on the MNER task and more than 20 percentage points on the MRE task, highlighting the value of incorporating visual information, especially when dealing with short text and ambiguous content commonly found in social media.

[0126] Table 3

[0127]

[0128] Second, our model outperforms traditional fine-grained multimodal methods such as UMT, UMGF, and MEGA. Unlike those methods that filter information, our model and TMR both retain comprehensive multimodal data and achieve higher MRE scores (88.42 and 89.05, respectively), while filtering-based methods such as HVPNeT (81.85) and MMIB (83.23) perform lower. By enhancing the interaction of fine-grained information nodes, the performance of sequence labeling tasks is improved, and our model outperforms TMR on two MNER task datasets, demonstrating its advantage in node information propagation.

[0129] Finally, the results are compared. Using CLIP as the encoder, the proposed model improves by 11.65, 4.01, and 9.56 on the three datasets, respectively, proving the effectiveness of the proposed model in enhancing the integration of multimodal information. In addition, compared with widely recognized multimodal pre-trained models such as BLIP and fine-tuned multimodal models such as UPA, the proposed model also performs well on each dataset.

[0130] In summary, our model achieves SOTA results on Twitter-2015 and performs competitively on other datasets. This success is attributed to: 1) the hypergraph structure preserves the integrity of multimodal information while revealing hidden high-order interaction information; 2) the separation of cross-modal interactions enables more accurate filtering of the correct answer from the provided information.

[0131] (4) Ablation experiment:

[0132] Ablation studies are conducted to comprehensively evaluate the effectiveness and importance of various modules in the model.

[0133] First, we discuss two key innovations in hypergraph construction: the necessity of distinguishing semantic hyperedges from contextual hyperedges. To this end, we design two experimental conditions: one containing only semantic hyperedges and the other reducing semantic connections, to compare their performance.

[0134] Under the condition of retaining only semantic hyperedges, the context information embedded by CLIP in the feature encoding stage already provides sufficient coverage for text data. However, this setting significantly weakens the visual connection between images, especially in the high-order contextual interaction between images and text. As shown in Table 3, the performance of the model drops significantly under this configuration, showing that the importance of context information cannot be ignored.

[0135] Subsequently, the complete contextual hyperedges are retained, while the role of semantic hyperedges is tested in depth. In this model, the semantically connected node method is to select by comparing the cosine similarity between nodes, ensuring that each central node is connected to its most similar node. This process is applicable to connections within the modality as well as cross-modal interactions. In order to evaluate the impact of different connection ratios on model performance, connection ratios of 10%, 30%, 50% and 70% are set to observe the specific role of semantic node connections on the final results. At the same time, considering that contextual relationships can only be constructed within each modality, it is ensured that at least one pair of semantically related nodes is connected between modalities. The experimental results are shown in Table 4. The connection mode of semantic hyperedges has a significant impact on performance, especially in reasoning entity categories and relationships, which further emphasizes the key role of semantic information in the model.

[0136] Table 4

[0137]

[0138] Finally, the separation operation is removed and the hypergraph neural network is simplified to the standard hypergraph convolutional network HGNN. The training mechanism of HGNN mainly relies on the dynamic update of node features and the effective aggregation of hyperedges. During the training process, HGNN optimizes the model parameters through the back-propagation algorithm to ensure that the information transfer and aggregation between nodes and hyperedges can achieve the best effect.

[0139] Specifically, HGNN first captures the relationship between nodes through the adjacency matrix, and then applies the aggregation function to fuse the features of adjacent nodes. This process is repeated in each layer, updating the feature representation of the node layer by layer. The node feature update is defined as:

[0140]

[0141] θ is a trainable weight matrix. In the design of the loss function, HGNN usually combines cross entropy loss and regularization terms to improve the generalization ability of the model. Through iterative training, HGNN can gradually learn richer feature representations and provide more accurate support for downstream tasks. This training mechanism enables HGNN to achieve efficient information extraction and feature learning in complex multimodal environments. However, compared with the original complete method, this simplification significantly leads to a decrease in performance, and the results are shown in Table 3.

[0142] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A multimodal entity relationship extraction method based on hypergraph neural network, characterized in that: include: Get text-image pairs; Inputting the text-image pair into a preset hypergraph construction model to obtain node features and a multimodal hypergraph structure of the text-image pair, wherein the hypergraph construction model is used to extract fine-grained node features from text and images, analyze the relationship between nodes through the node features and express it as a multimodal hypergraph structure, the multimodality includes text and image, and the hypergraph structure includes a set of hyperedge sets, each hyperedge in the hyperedge set can connect multiple nodes at the same time, and the number of nodes connected by the hyperedge is not limited; The node features and multimodal hypergraph structure are input into a preset hypergraph neural network model, and the multimodal entity relationship in the text-image pair is output, wherein the hypergraph neural network model is used to analyze the hypergraph node information propagation from the perspective of semantics and contextual relationships, and combines the attention mechanism to assign corresponding weights to the semantic information between modalities.

2. According to claim 1, the multimodal entity relationship extraction method based on hypergraph neural network is characterized in that: The hypergraph construction model includes a feature extraction module and a hypergraph construction module. The feature extraction module is used to extract the node features of the text-image pair and obtain a multimodal semantic hyperedge set and a contextual hyperedge set based on the node features; the hypergraph construction module is used to construct the multimodal hypergraph structure based on the node features, semantic hyperedges, and contextual hyperedges.

3. The multimodal entity relationship extraction method based on hypergraph neural network according to claim 2 is characterized in that: Extracting node features of the text-image pair includes: The text-image pair is parsed to obtain text node features and image node features respectively, wherein the text node features include words, phrases to which the words belong, and global information, and the image node features include visual targets and visual relationships between visual targets.

4. The multimodal entity relationship extraction method based on hypergraph neural network according to claim 3 is characterized in that: Acquiring a multimodal semantic hyperedge set and a contextual hyperedge set according to the node features includes: Comparing semantic similarity within and between modalities according to the text node features and the image node features, respectively, to construct the semantic hyperedge set; The text context structure information is extracted according to the text node features and the text grammar dependency tree, and the context hyperedge set is constructed in combination with the visual targets and the visual relations between the visual targets.

5. The multimodal entity relationship extraction method based on hypergraph neural network according to claim 4 is characterized in that: Comparing semantic similarity within and between modalities according to the text node features and the image node features, respectively, constructing the semantic hyperedge set comprises: Calculate the cosine similarity matrix of the text node features and image node features and cross-modalities respectively, and connect semantic hyperedges according to the cosine similarity matrix, wherein each node in the modality is connected to the top k nodes with the highest similarity to itself. n nodes, forming a hyperedge centered on the current node, and obtaining the text and image semantic hyperedge set matrix; cross-modal information is centered on the text node, and the k nodes with the highest similarity to the central node are selected. m image nodes, connecting the image nodes as the similarity hyperedge set matrix of the text-to-image correspondence; Based on the text and image semantic hyperedge set matrix and the similarity hyperedge set matrix of the text corresponding image, a semantic hypergraph association matrix is ​​constructed to obtain the semantic hyperedge set.

6. The multimodal entity relationship extraction method based on hypergraph neural network according to claim 5 is characterized in that: Extracting text context structure information according to the text node features and the text grammar dependency tree, and combining visual targets and visual relationships between visual targets to construct the context hyperedge set includes: Take words as text nodes, grammatical relations between words and phrase combinations as the correspondence between nodes and edges, build a text structure hypergraph association matrix, and obtain the text context hyperedge set; Taking visual targets as nodes and visual relationships between visual targets as the correspondence between nodes and edges, a visual structure hypergraph association matrix is ​​constructed to obtain the visual context hyperedge set.

7. The multimodal entity relationship extraction method based on hypergraph neural network according to claim 6 is characterized in that: Constructing the multimodal hypergraph structure based on the node features, semantic hyperedges, and contextual hyperedges includes: The semantic hypergraph association matrix, the text structure hypergraph association matrix, and the visual structure hypergraph association matrix are integrated into a hypergraph association matrix, and the multimodal hypergraph structure is constructed according to the hypergraph association matrix, wherein the degree of a node in the multimodal hypergraph structure is the number of hyperedges in which each node participates, and the degree of a hyperedge is the number of nodes connected by each hyperedge.

8. The multimodal entity relationship extraction method based on hypergraph neural network according to claim 1 is characterized in that: The hypergraph neural network model includes a feature aggregation module and a relationship classification module, wherein the feature aggregation module is used to obtain hyperedge feature information in the multimodal hypergraph structure and feed it back to the node to update the node feature; The relationship classification module is used to aggregate updated node features and extract relationships between entities.

9. The method for extracting multimodal entity relations based on a hypergraph neural network according to claim 8, characterized in that: Acquiring hyperedge feature information in the multimodal hypergraph structure includes: Aggregate the feature information in nodes using semantic hyperedges as units, and introduce an attention mechanism to adjust the weights of nodes in different contexts to obtain the semantic hyperedge feature matrix; The feature information of nodes is aggregated using contextual hyperedges as units, and a spectral convolutional network is used to aggregate the feature information of nodes on hyperedges to obtain the contextual hyperedge feature matrix.

10. The multimodal entity relationship extraction method based on hypergraph neural network according to claim 8, characterized in that: Aggregating the updated node features and extracting the relationship between entities includes: Perform entity category analysis through updated node features to obtain predicted entity categories; The updated node features are aggregated to obtain the node features of the predicted entity; and the relationship between the predicted entity and the target entity is extracted based on the node features of the predicted entity.

Images