Joint multi-modal entity relationship extraction method and system based on information representation and semantic alignment

Abstract

The invention discloses a joint multi-modal entity relationship extraction method and system based on information representation and semantic alignment, and the method comprises the steps: obtaining sample data composed of an original image and a text, and obtaining the feature representation of visual information and text information based on the sample data; aligning the feature representation of the visual information and the feature representation of the text information by using a progressive modal semantic alignment strategy; by introducing a multi-layer correlation mapping mechanism guided by fine granularity, the correlation coefficient represented by the features of the aligned visual information and text information is judged, and features irrelevant to a task core are filtered; performing visual representation and text interaction by using a multi-modal interaction module to obtain multi-modal semantic features; carrying out weighted mapping on the multi-modal semantic features by utilizing a routing weighting function, and finally obtaining multi-modal feature representation; and sending the multi-modal feature representation into a word pair relation label extractor, and extracting an entity, an entity relation and an entity attribute quintuple.

Description

Technical Field

[0001] This invention belongs to the field of natural language processing in computer intelligent information processing, specifically involving a joint multimodal entity relation extraction method and system based on information representation and semantic alignment. Background Technology

[0002] In the field of Natural Language Processing (NLP), Named Entity Recognition (NER) and Relation Extraction (RE) aim to extract structured information from unstructured text. With the increasing prevalence of multimedia content, traditional text-based methods are struggling to meet the demand for comprehensive information understanding due to the simplicity and non-standard nature of text in multimodal scenarios. Multimodal Named Entity Recognition (MNER) and Relation Extraction (MRE) enhance semantic understanding by utilizing visual information as additional input, supplementing the semantic information missing from the text.

[0003] Despite significant progress in MNER and MRE, independent task modeling paradigms still have limitations. Relation extraction largely depends on the results of named entity recognition, thus leading to error propagation problems. Considering these issues, as well as the tight semantic coupling and bidirectional dependency between MNER and MRE tasks, Joint Entity Relation Extraction (JERE) is becoming increasingly important.

[0004] Joint Multimodal Entity Relation Extraction (JMERE) aims to enhance task performance by leveraging the bidirectional interaction between MNER and MRE tasks. As a pioneering work in JMERE, EEGA designed an edge-enhanced graph alignment network to capture relationships between entities and objects, and devised a word pair relation labeling method to avoid error propagation caused by pipelined information extraction. Subsequent work, FGCL, further extended the word pair relation labeling scheme, proposing a multi-word decoding method that iteratively predicts the relationships between word pairs by fusing the prediction results of neighboring word pairs, thus improving the extraction capability for multi-word entities. More recently, researchers have favored introducing external knowledge enhancement to further supplement the missing semantic information in text-image pairs. For example, methods such as CAG, KEJME, and Shape-CA utilize GPT and LLaVA as knowledge sources, performing fine-grained selection of knowledge features based on specific visual objects and text entities to import relevant knowledge.

[0005] However, the core of the JMERE task lies in how to effectively utilize the excellent visual features in images and integrate them into text representations to supplement the missing semantic information in the text. Existing methods ignore the fact that different representations of visual information can be obtained from the original image, and each representation can help with the task. Furthermore, there are shortcomings in utilizing multiple information representations; different information features contribute differently to the task, and treating them as equal makes it difficult for the model to distinguish the importance of features in task decision-making, affecting the accuracy of the model's decisions.

[0006] At the same time, while different visual information can provide richer semantic information for task decision-making, it may also introduce interference that is irrelevant to the task, leading the model to make incorrect decisions.

[0007] Finally, since images and text originate from vision and language respectively, the semantic representations of these two semantic spaces differ significantly, making cross-modal interaction modeling difficult. Although existing methods in the JMERE domain attempt to align entities in text with localized regions in images using graph structure methods, these methods exhibit explicit correlations, potentially leading to errors in image-text alignment. Summary of the Invention

[0008] To address the problems existing in the prior art, this invention provides a joint multimodal entity relationship extraction method and system based on information representation and semantic alignment. By using multi-angle visual representation, fine-grained relevance filtering, and progressive modal alignment, the accuracy of joint multimodal entity relationship extraction is improved.

[0009] To achieve the above objectives, the present invention provides the following solution: A joint multimodal entity relation extraction method based on information representation and semantic alignment, the method comprising: Step 1: Obtain sample data consisting of the original image and text, use multiple methods to obtain multi-angle visual information representation of the image, and use multiple encoders to encode the input sample data and visual information representation to obtain feature representations of visual and text information; Step 2: Use a progressive modal semantic alignment strategy to align the feature representations of visual and textual information; Step 3: By introducing a fine-grained guided multi-layer correlation mapping mechanism, the correlation coefficient between the feature representations of aligned visual and textual information is determined, and features irrelevant to the core task are filtered out. Step 4: Utilize the multimodal interaction module to perform visual representation and text interaction to obtain multimodal semantic features; use the routing weighting function to perform weighted mapping on the multimodal semantic features to finally obtain the multimodal feature representation; Step 5: Feed the multimodal feature representations into the word pair relation annotation extractor to extract entity, entity relation, and entity attribute quintuples.

[0010] Preferably, in step one, the method of obtaining multi-angle visual information representations of images using multiple methods and encoding the input sample data and visual information representations using multiple encoders to obtain feature representations of visual and textual information includes: The pre-trained language model BERT is used as a text encoder to encode the original text information to obtain the feature representation of the text. T represents the encoded text features; Multiple strategies are employed to extract multi-angle visual representations from the original image: The original image I is encoded using a pre-trained visual encoder model to extract multi-level visual features perceived by ViT. , ,in This indicates that the initial coding layer of ViT can encode the original image into a feature representation. Let m represent the ViT hierarchical block and m represent the ViT level. The features obtained from the 12 ViT levels are concatenated to obtain the original image encoding representation. , This represents the features mapped from the original image I through the initial coding layer. The features of ViT at layer m; The visual grounding toolkit is used to detect local regions containing objects in an image, and the regions most relevant to the semantics of the original text are extracted from the original image I. Each fine-grained image region is encoded using a pre-trained ViT encoder to obtain an encoded feature representation. ; The original image is fed into the BLIP model and the Faster-RCNN model to generate global descriptive text (Image caption) and image tags (Image tags) for highlighting regions of the image, respectively. Feature representations are then obtained by encoding using a pre-trained BERT language model. , ,in and These represent the encoded image descriptive text and object labels, respectively.

[0011] Preferably, in step two, the method for aligning the feature representations of visual and textual information using a progressive modal semantic alignment strategy includes: The descriptive text and object labels of the image are concatenated and then fed into a trainable Transformer block for representation. This yields the semantic features of the descriptive text and object labels; The obtained semantic features are used to perform semantic enhancement on the original image, fine-grained image regions, and text features. The expression is as follows: ; ; ,in This represents the cross-attention mechanism. As a query As keys and values, Manually defined keys Scaling factor and The coefficient weights represent the semantic enhancement of textual and visual information; The enhanced textual and visual semantic representations are brought closer together by using a contrastive learning mechanism.

[0012] Preferred methods for further bridging the gap between enhanced textual and visual semantic representations using contrastive learning mechanisms include: The original image-text pairs are divided into multiple batches. N image-text pairs from each batch are used to construct positive and negative samples, where the same image-text pair... As positive samples, different image and text pairs Considered as negative samples, the text-to-image similarity within a batch is defined as: ,in express Cosine similarity between them Let T and V represent the original text information encoding and the original image encoding after semantic enhancement in a batch, respectively. T represents the text modality, V represents the image modality, and i and j are the i-th and j-th sample pairs in a batch of N samples, respectively. , It is a learnable temperature parameter; image-to-text similarity within a batch: ,in express Cosine similarity between them; Based on the text-to-image similarity and image-to-text similarity within a batch, calculate the image-to-text contrast loss. Contrast loss between text and image : ; Based on image-to-text contrastive loss and text-to-image contrastive loss, a batch of image-text loss values ​​is obtained to further enhance the semantic representation consistency between images and text. .

[0013] Preferably, in step three, the method for determining the correlation coefficient between visual and textual information and filtering features irrelevant to the core task by introducing a fine-grained guided multi-layer correlation mapping mechanism includes: ; In the formula, This means merging the visual features of 12 layers into the features of 1 layer. This indicates that the h-th part aggregates features from 12 layers. Indicates the first The characteristics of each part are derived from their convergence.

[0014] Preferably, step four involves using a multimodal interaction module to perform visual representation and text interaction to obtain multimodal semantic features; and using a routing weighting function to perform weighted mapping on the multimodal semantic features to finally obtain the multimodal feature representation. ; In the formula, G represents the original image, fine-grained image, region descriptive text, and multimodal features resulting from the interaction between object labels and text features T, respectively. Indicates weighted routing. express and The linear mapping function, , Indicates the 0th, 1st to... The weights of the features, where T represents the original text encoded features. express , [ ; ; ; ] indicates concatenation operation.

[0015] Preferably, in step five, the method for feeding the multimodal feature representation into the word pair relation annotation extractor to extract entities, entity relations, and entity attribute quintuples includes: Terms in multimodal features and The resulting word pair representations are fed into a linear prediction layer to obtain the probability distribution of the word pair labels. Used to predict the quintuple <subject, subject attribute, object, object attribute, subject-object relation>, where... Learnable mapping matrix This represents the learnable initial bias vector.

[0016] The present invention also provides a joint multimodal entity relation extraction system based on information representation and semantic alignment. The system is used to implement the aforementioned method and includes: an encoding module, an alignment module, a discrimination module, an interaction and weighting module, and an extraction module. The encoding module is used to acquire sample data consisting of original images and text, acquire multi-angle visual information representations of the images using multiple methods, and encode the input sample data and visual information representations using multiple encoders to acquire feature representations of visual and text information. The alignment module is used to align the feature representations of visual information and text information using a progressive modal semantic alignment strategy; The discrimination module is used to determine the correlation coefficient between the feature representations of aligned visual information and text information by introducing a fine-grained guided multi-layer correlation mapping mechanism, and to filter features that are irrelevant to the core of the task. The interaction and weighting are used to perform visual representation and text interaction using the multimodal interaction module to obtain multimodal semantic features; the multimodal semantic features are then weighted and mapped using a routing weighting function to finally obtain a multimodal feature representation. The extraction module is used to feed the multimodal feature representation into the word pair relation annotation extractor to extract entities, entity relations, and entity attribute quintuples.

[0017] Compared with the prior art, the beneficial effects of the present invention are as follows: Existing joint multimodal entity relation extraction methods suffer from several issues regarding image visual representation, modal semantic alignment, and interference from irrelevant visual information. For example, they neglect the importance of the rich semantic information inherent in the image itself for task decision-making; existing image-text semantic alignment methods lack explicit relevance, leading to poor modal alignment results; and irrelevant visual information may mislead task decisions. This invention proposes a joint multimodal entity relation extraction method based on information representation and semantic alignment. This method utilizes multi-angle visual representation of images and explicit modal alignment to improve the accuracy of joint multimodal entity relation extraction tasks.

[0018] Image information contains various visual representations, including the original global image, local image regions, and linguistic descriptions. Therefore, this invention fully utilizes multi-angle visual features to extract fine-grained local regions from the original image and generate descriptive text. A text-guided prefix-guided modal interaction mechanism then facilitates modal interaction for each visual representation. Furthermore, considering the significant differences in visual information perception across different levels of visual encoding, this invention extracts multi-layered visual information using features generated at different levels of the visual encoder. Addressing the issue of varying contributions of visual information from different perspectives to the task, this invention employs dynamic routing to determine the importance of modal features to the task, dynamically adjusting the weight of each feature to collectively contribute to task decision-making.

[0019] Existing methods only actively utilize image modalities to supplement text semantic information. However, visual information may contain interfering factors that are irrelevant to task decision-making. Inspired by attention mechanisms, this invention designs a fine-grained guided multi-layer relevance mapping mechanism. Starting from semantics that are relevant to task decision-making, it makes fine-grained judgments on features that are related to text semantics and filters out visual information that is not strongly relevant to the core of the task.

[0020] Traditional image-text comparison learning often fails to generate significant similarity in semantic representations. This invention observes that textual descriptions of images have a high semantic relevance to the images themselves and, to some extent, are similar to the semantic representations of the textual information. Therefore, a progressive modality alignment module is designed. It first enhances modal similarity with descriptive text of the image, narrowing the modal representation gap and mitigating modal heterogeneity. Then, it further aligns the modalities through image-text comparison learning. Compared to traditional direct image-text alignment, this progressive alignment strategy with explicit cues is more effective. Attached Figure Description

[0021] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 : A schematic diagram of the joint multimodal entity relation extraction method based on information representation and semantic alignment provided by this invention; Figure 2 The present invention provides a fine-grained guided multilayer correlation mapping structure diagram; Figure 3 : A schematic diagram of progressive modal semantic alignment provided by this invention; Figure 4 : A schematic diagram of the text-guided prefix-guided modal interaction mechanism provided by this invention; Figure 5 : A schematic diagram of the routing weighted fusion mechanism provided by this invention; Figure 6 : A schematic diagram of the multi-angle visual representation provided by this invention; Figure 7 The experimental results of variations involving the deletion or alteration of different visual information provided by this invention are illustrated in the figure. Figure 8 The diagram shows the effect of the coefficients for enhancing text in the progressive alignment provided by this invention. Figure 9 The experimental results of the progressive alignment ablation method provided by this invention are illustrated in the following diagram; Figure 10Visual comparison chart of the modal distribution of this invention and other methods. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0025] Example 1 This invention provides a joint multimodal entity relation extraction method based on information representation and semantic alignment, comprising: Step 1: Multi-angle visual information extraction. This step utilizes various strategies to extract diverse and multi-layered visual representations from the original image, including local visual regions, textual descriptions of the image, and image feature labels. A multi-layered visual encoder is used to obtain these multi-level visual representations. Since existing methods overuse information introduced by large language models while neglecting the various information inherent in the image itself, this approach deeply considers the multi-angle implicit information of the image, providing diverse and multi-layered visual information to supplement textual semantics and improve the accuracy of joint multimodal entity relation extraction.

[0026] Step Two: Progressive Modal Semantic Alignment. Image and text modalities originate from two different semantic spaces and exhibit significant differences in their representations, posing challenges in cross-modal interaction modeling and making it difficult to associate images with text. Most existing methods ignore the modal heterogeneity problem caused by the gap in modal representations. This invention proposes progressive semantic alignment to achieve explicit semantic representation alignment across modalities. Since the textual description of an image has semantic relevance in both the image and the text, the image and text are first enhanced using descriptive text to establish explicit relevance. Then, further contrastive learning is used to align the image and text. Alignment with explicit cues makes cross-modal interaction modeling more effective.

[0027] Step 3: Fine-grained guided multi-layer correlation mapping. Existing methods only consider that image information can supplement text semantic information in a positive direction. However, visual information may exist in interfering factors that are irrelevant to task decision-making. Therefore, this invention proposes a fine-grained guided multi-layer correlation mapping mechanism. Through a multi-head attention mechanism, the correlation coefficient between visual information and text is judged in a fine-grained manner, reducing visual features that are irrelevant to the core of the task, thereby filtering out features that mislead the model to make wrong decisions.

[0028] Step 4: Modal Interaction and Routing Weighted Fusion. Existing joint multimodal entity relationship extraction methods do not pay enough attention to modal interaction design, adopting shallow modeling approaches and ignoring the different contributions of different information features to task decision-making. Therefore, this invention utilizes a multi-layered prefix-guided modal interaction mechanism to perform deep cross-modal interaction modeling of multi-angle visual features, so as to effectively integrate visual information into the text. Furthermore, a routing weighting mechanism is proposed, which sends multiple feature information into the routing to obtain the contribution of different features relative to the core of the task. The coefficients of the routing feedback are used to weight and fuse multiple features, ultimately contributing to the correct task decision.

[0029] Step 5: Input the multimodal semantic features obtained in Step 4 into the word pair relation annotation extractor to extract entities, entity relations, and entity attribute quintuples.

[0030] like Figure 1 As shown, the following steps are specifically disclosed: Step 1: Obtain sample data consisting of the original image and text, use multiple methods to obtain multi-angle visual information representation of the image, and use multiple encoders to encode the input sample data and visual information representation to obtain feature representations of visual and text information; Step 2: Use a progressive modal semantic alignment strategy to align the feature representations of visual and textual information; Step 3: By introducing a fine-grained guided multi-layer correlation mapping mechanism, the correlation coefficient between the feature representations of aligned visual and textual information is determined, and features irrelevant to the core task are filtered out. Step 4: Utilize the multimodal interaction module to perform visual representation and text interaction to obtain multimodal semantic features; use the routing weighting function to perform weighted mapping on the multimodal semantic features to finally obtain the multimodal feature representation; Step 5: Feed the multimodal feature representations into the word pair relation annotation extractor to extract entity, entity relation, and entity attribute quintuples.

[0031] Step one involves extracting visual information from multiple angles, which includes the following specific operations: After acquiring sample data consisting of original images and text, multiple strategies are used to extract multi-angle visual representations of the images. The original information of the input image-text pairs, as well as the multi-angle visual representations of the images, are encoded using visual and text encoders to obtain feature representations of visual and textual information from multiple perspectives, such as... Figure 6 As shown.

[0032] For text information, this invention uses the pre-trained language model BERT as a text encoder to encode the original text information and obtain the feature representation of the text. T represents the encoded text features; For multi-angle visual representation of images, various methods are used to extract information from the images. First, a pre-trained visual encoder, the Vision Transformer (ViT) model, is used to encode the original image I. Since ViT adopts a multi-layered stacked structure similar to Transformer blocks, each block level has a different perceptual effect on visual information. This invention extracts the multi-level visual features perceived by ViT. , ,in This represents the initial coding layer of ViT, which encodes the image into features. This represents the features mapped from the original image I through the initial coding layer. This represents the hierarchical block of ViT, m This indicates the hierarchy of ViT. The features of the m-th ViT layer are used; the features obtained from a total of 12 ViT layers are concatenated to obtain the original image encoding representation. ; Meanwhile, this invention utilizes visual grounding toolkit localization technology to detect local regions containing objects in an image, and extracts the most semantically relevant regions from image I. A fine-grained image region (object image), Each of these fine-grained image regions is also encoded using a pre-trained ViT encoder. , ,in The features representing the initial coding layer mapping, This represents the feature of the m-th layer of ViT. express This is a fine-grained image region. First, the features from the 12 ViT layers are concatenated, and then... The fine-grained image region features are concatenated to obtain the final encoded feature representation. ; Furthermore, this invention feeds the original image into the BLIP model and the Faster-RCNN model, which generate global descriptive text (image caption) and image tag descriptions of highlighted regions, respectively. The pre-trained BERT language model is then used for encoding to obtain feature representations. , ,in and These represent the encoded image descriptive text and object labels, respectively.

[0033] Step two, progressive modal semantic alignment, includes the following specific operations: like Figure 3 As shown, after extracting text and multi-angle visual features, the present invention adopts a two-stage progressive alignment strategy. First, the present invention uses image caption and image tag that are relevant to both image and text semantics to enhance the semantics of text and image, so that they both obtain a certain degree of similarity, making the subsequent alignment have explicit relevance. First, the descriptive text and object labels of the image are concatenated and then fed into a trainable Transformer block for representation. Semantic features of descriptive text and object labels are obtained, and then these semantic features are used to enhance the original image, fine-grained image regions, and text features. ; ; in This represents the cross-attention mechanism. As a query As keys and values, Manually defined keys The scaling factor, T, represents the encoded text features obtained by the BERT encoder from the original input text in the image-text pair. , Representing text features T and original image encoded features respectively. Fine-grained image region features As queries, the descriptive text of the image and the semantic features of the object labels. As keys and values, and The coefficient weights represent the semantic enhancement of textual and visual information; After semantic enhancement, the text and visual representations acquire a certain semantic relevance. A contrastive learning mechanism is then used to further narrow the semantic gap between the text and visual representations. Specifically, this invention divides the original semantically enhanced text information encoding and the corresponding semantically enhanced original image encoding into multiple batches. N image-text pairs in each batch are used to construct positive and negative samples, where the same image-text pair... As positive samples, different image and text pairs Negative samples are considered as examples. The goal of semantic alignment is to maximize the similarity between positive samples while minimizing the similarity between negative samples, thereby reducing the semantic representation gap between text and images. The text-to-image similarity within a batch is defined as: ,in express Cosine similarity between them Let T and V represent the original text information encoding and the original image encoding after semantic enhancement in a batch, respectively. Here, T represents the text modality, V represents the image modality, and i and j are the i-th and j-th sample pairs in a batch of N samples, respectively. , It is a learnable temperature parameter; and similarly, the similarity between an image and text within a batch can be obtained: ; Therefore, the contrast loss from image to text can be calculated. Contrast loss between text and image : ; By combining the two loss functions mentioned above, a batch of image-text loss values ​​is obtained, which can be used to further enhance the semantic representation consistency of images and text.

[0034] .

[0035] The fine-grained guided multilevel correlation mapping mechanism in step three includes the following specific operations. like Figure 2 As shown, the original image and the fine-grained image region are processed by the visual encoder ViT to obtain 12 layers of visual features. To filter out visual information that is ineffective for the task, this invention introduces a fine-grained guided multi-layer relevance mapping mechanism. Utilizing a multi-head attention mechanism, each visual feature can be assigned a more granular and accurate relevance coefficient to the text. Specifically, the original image and the fine-grained image region are connected together using the 12 layers of visual features generated by ViT as the key (K), and the text information is used as the query (Q). The relevance between the two is determined through multi-head attention. Where T and V represent the encoding features of the original text information of the result. This represents the encoded features obtained after passing the original image and fine-grained image regions through 12 layers of ViT. For the m-th layer features, Subsequently, a multi-head attention mechanism was used to... Calculate the weighting coefficients: , and This represents the h-th part after the partitioning of Q and K. This is the attention weight of the h-th head; concatenating multiple heads yields the complete attention coefficient. , The scaling factor for key k is defined manually; The original text information encoding feature T is used as the query Q, and the original image and fine-grained image regions are encoded through a 12-layer ViT. Each is used as a key K, and fine-grained computation is achieved through a corresponding multi-head attention mechanism. T and V are uniformly divided into H according to the hidden dimension. Each part By manually defining, the first The attention head coefficient is calculated as follows: , The h-th part of the text information encoding feature T is used as the query. This invention involves dividing the encoded features of the corresponding 12-layer original image / fine-grained image regions into H parts, and then concatenating each h-th part of the 12 layers as a key. A relevance weight coefficient is then calculated using an attention mechanism. Specifically, this invention divides all the encoded features of the corresponding 12-layer original image / fine-grained image regions into H parts, and multiplies each part by its corresponding attention score. We obtain the h-th part of the aggregated 12-layer features, where M=12 represents a 12-layer ViT. This represents the attention weight coefficient for the h-th head. This represents the characteristics of the h-th part of the 12-layer ViT convergence. Corresponding to the h-th part; Finally, by splicing together all H parts, the visual features of the 12 layers can be merged into 1 layer. ,in This means that the original image and fine-grained image region encoding feature distributions from 12 layers are aggregated into a single layer of features. , Indicates the first Each part is characterized by its convergence. This means dividing each of the 12-layer feature encoding into... Each part Corresponding to the first This allows for the filtering of information that is irrelevant to the semantics of the text.

[0036] Step four, modal interaction and route weighted fusion, includes the following specific operations: like Figure 4 As shown, this invention designs an independent multimodal interaction module for each visual information. Each module consists of a text-guided prefix-guided interaction module. This module treats the visual information representation of each image as a prefix input of text features and uses a self-attention mechanism to perform modal interaction modeling to obtain complementary image features for the text representation. Specifically, in the Transformer block with a total of L layers, for the ... The layer first obtains prefix-guided representations of text feature keys and values. ,in The image is represented by a multi-angle visual representation, including original image features and fine-grained image region features derived from a fine-grained guided multi-layer correlation mapping mechanism, image description text, and object label features; then, modal interaction operations are performed to obtain a multi-modal semantic representation of each visual information to generate a token-aware representation. , ,in No. The features of the layers after the text-image modal interaction, the first layer T represents the encoded features of the original text. , This represents the original image features, fine-grained image region features, image description text, and object label features. This indicates a concatenation operation based on length, d= , for manually defined keys scaling factor, ( Indicates the first Layer attention weights, This serves as the next layer of features, from which we can obtain the multimodal semantic features of each visual representation and text interaction; Fine-grained guided multilayer correlation mapping mechanism for original image features With fine-grained image region features Image-description text features and object label features As key and value respectively Modal interaction is performed by performing prefix-guided interaction module operations with the text information encoding features T to obtain the multimodal features of the Lth layer. , respectively corresponding to , , and ,like Figure 5 As shown, a routing weighting function is used to perform weighted mapping on multimodal features. The mapping function is defined as follows: ,in =4, T represents the text information encoding feature. express , Calculate routes for weights. It is a linear mapping function; All multimodal semantic features are concatenated and fed into the routing algorithm, which consists of a multilayer perceptron (MLP) with a softmax activation function. Routing weighting operations are used to assign corresponding weights to text and multimodal features, thereby quantifying their importance in task decision-making. ,in These represent the multimodal features resulting from the interaction between the original image (after fine-grained guided multi-layer correlation mapping), fine-grained image region features, region descriptive text, and object labels with text feature T. Considering the greater importance of text for the joint multimodal entity relation extraction task, features T from the original text are added to enhance semantic information and reduce interference from irrelevant visual information. The final multimodal feature representation is obtained. , where G= Indicates weighted routing. express and The linear mapping function, , Indicates the 0th, 1st to... The weights of the features, where T represents the original text encoded features. express [a;b;c;d] represents the concatenation operation.

[0037] The word pair relation tagging extractor in step five includes the following specific operations: Word pair relation annotation methods extract named entities and their relations by transforming the joint multimodal entity relation extraction task into a classification task of predicting the relation labels between each word pair, thus avoiding the error propagation problem present in pipelined methods. Specifically, it extracts the word terms from multimodal features. and The resulting word pair representations are fed into a linear prediction layer to obtain the probability distribution of the word pair labels. ,in Learnable mapping matrix This represents a learnable initial bias vector used to predict the quintuple <subject, subject attribute, object, object attribute, subject-object relation>, and uses cross-entropy loss to measure the predicted distribution. With the true distribution Differences between them: Where S represents the total number of entity and relation label categories, and s is the number of the first category. There are tag category indices, where n represents the length of the encoded sentence from the original input text, and i and j are the index positions of two terms in the sequence within a word pair. , Let be the true label of the word pair (i, j) in the s-th label category. The value is 1 if the word pair (i, j) belongs to the s-th label category, and 0 otherwise. This represents the probability that the model predicts a word pair (i, j) belonging to the s-th category label. This represents the set of all learnable parameters in the model; finally, the final loss function can be obtained. ,in The image-text comparison learning loss value is calculated for each batch. The loss of all batches in the sample is calculated through iterative calculation, and the model is optimized by backpropagation using the loss.

[0038] This invention conducted extensive experiments on the only publicly available dataset for the JMERE task to verify the effectiveness of the proposed joint multimodal entity relation extraction method combining information representation and semantic alignment. The JMERE dataset is primarily used for the JMERE task. This dataset mainly combines the MNER and MRE datasets, and each sample in the dataset contains the original sentence, the corresponding image, and a set of entity-relation quintuples. Precision (P), Recall (R), and F1-score are used as evaluation metrics for model performance. The extracted quintuples are considered correct only when the predicted span exactly matches the span of the ground truth.

[0039] To ensure the fairness and comparability of the evaluation, this invention selects a comprehensive comparison with currently advanced and representative methods.

[0040] Textual methods: PFN and UniRel. Multimodal pipeline methods: This invention selects the best-performing models from MNER and MRE studies and combines them with MEGA to form multimodal pipeline methods, including AdapCoAt, OCSGA+, and UMGF+. Multimodal-JERE methods: Using the word pair relation tagging method proposed by EEGA to improve AdapCoAtt, OCSGA, AGBAN, UMGF, MEGA, MAF, and HVPNeT, several new JMERE models are obtained, including AdapCoAtt. OCSGA AGBAN UMGF MEGA MAF and HVPNeT This invention also selected the mainstream JMERE methods EEGA, Shape-CA, FGCL, KEJME, CAG, CITR, and TESGA as the baseline for the model.

[0041] Table 1. Experimental results (%) on the JMERE task and named entity recognition results (#MNER) on the JMERE dataset. Table 1 shows the experimental results of the model of this invention and other representative methods on the JMERE task. This invention allows us to observe: First, UMGF+ performs well in the multimodal pipelined approach, validating the effectiveness of the semantic interaction strategy among multimodal semantic units. However, it exhibits poorer performance compared to joint extraction methods because the pipelined approach first performs MNER to identify entities, and then performs MRE to identify relationships between entities. This leads to error propagation, affecting the accuracy of information extraction.

[0042] Secondly, in the multimodal joint extraction method, the HVPNeT method, which improves upon the word pair relation tagging, is utilized. Compared to text-based methods, this approach utilizes image information to supplement missing semantics in the text, resulting in a 1.07% improvement in the F1 score compared to PFN on the JMERE task. Furthermore, it employs a joint extraction method to achieve end-to-end extraction, suppressing the error propagation problem present in pipelined methods. Compared to the best-performing UMGF+, it achieves a 6.29% improvement in the F1 score, demonstrating the effectiveness of the joint extraction method.

[0043] Finally, current mainstream JMERE methods largely employ strategies that utilize supplementary information from large language models, such as CAG and KEJME, to enhance the model's understanding of the task. However, they fail to fully utilize the semantic information within images, resulting in suboptimal performance. FGCL and TESGA emphasize mitigating modal heterogeneity and the mismatch between image and text information through feature alignment, highlighting the importance of consistent image-text representations. However, the final performance of these methods still falls short of expectations, indicating a need for more effective approaches to address this issue.

[0044] The method of this invention effectively mines multi-angle visual representations of images and reduces the interference of irrelevant visual information on task decisions through a filtering mechanism. Simultaneously, the progressive modal alignment strategy constructs explicit cues, which is more effective than direct semantic representation alignment. The proposed MAVPA model achieves state-of-the-art performance on the JMERE dataset, improving the F1 score by 1.46% compared to the current top-performing TESGA method. In the multimodal named entity recognition (#MNER) subtask, the F1 score is improved by 0.48%, demonstrating the effectiveness of the proposed method on the JMERE task.

[0045] To verify the effectiveness of each part of the present invention, the present invention conducted a verification experiment on the model variants, and the results are shown in Table 2.

[0046] Table 2 Ablation Experiment Results w / o RW indicates the removal of the routing fusion mechanism, directly concatenating the text and multimodal semantic features, and fusing them through simple feature addition. As shown in the experimental results, the F1 score decreased by 1.27%, indicating that the routing weighting strategy adopted in this invention can effectively fuse semantic features of multiple modalities and coordinate the contributions of various modal features to task decision-making. w / o RW-T indicates the removal of textual information enhancement during fusion. Because the richness of multimodal semantic features can excessively mask the semantic information of the text, the F1 score decreases to some extent.

[0047] `w / o CL` removes the progressive modal alignment module, which, as shown in the results, leads to a significant 2.52% drop in the F1 score. The lack of modal alignment makes subsequent modal interaction modeling difficult. `w / o CL-P` removes the enhancements to images and text, and `w / o CL-I` removes the contrastive learning mechanism. Removing either part results in performance degradation; only a two-stage semantic alignment strategy can better mitigate modal heterogeneity.

[0048] `w / o FMRM` means removing fine-grained guided multi-layer correlation mappings and using only simple addition to achieve feature aggregation. Since irrelevant features exist in visual information, they can interfere with task decisions, and the lack of a filtering mechanism leads to performance degradation.

[0049] To further explore the effectiveness of different visual information representations of images in joint multimodal entity relation extraction tasks, this invention demonstrates, through variant experiments involving the deletion or alteration of different visual information, that diverse and multi-layered visual representations facilitate task decision-making.

[0050] Figure 7This study demonstrates the importance of different visual representations for the joint multimodal entity relation extraction task. Experimental results show that each visual representation influences task decisions. Diverse visual representations consider image features from different perspectives, supplementing semantics from multiple angles. Multi-layered visual features focus more on features at different levels of the image, providing the model with richer semantic meaning. Furthermore, fine-grained regions of the image have the greatest impact on model performance, indicating that details in the image play a crucial role in entity and relation extraction. In contrast, descriptive text for images only provides a general overview, potentially leading to the loss of some details and insufficient mining of image depth information, thus having a smaller impact on performance.

[0051] To demonstrate the effectiveness of the progressive modal alignment strategy proposed in this invention, this invention compares the removal of the progressive modal alignment strategy with the adoption of the progressive alignment strategy, such as... Figure 9 The heatmap shown represents the image and text features in a batch, with the horizontal and vertical axes representing the image and text features respectively. The diagonal elements represent the same image-text pair, and the other elements in the matrix represent different image-text pairs. The more similar the image-text pairs are, the darker the color. Figure 9 The comparison shows that the diagonal elements of the heatmap using the progressive alignment strategy have significantly different similarities from other elements. Only when the same text and image pair is more similar to other elements can modeling be more effective in subsequent cross-modal interactions.

[0052] like Figure 8 As shown, the feature distribution of the model of this invention and the TESGA model are visualized and compared. This invention randomly selected a batch of samples from the JMERE test set and used the t-SNE dimensionality reduction tool to map high-dimensional image and text features to a two-dimensional plane. The TESGA model, due to the lack of an alignment strategy, exhibits significant modal heterogeneity, with a clear gap between images and text. This makes it difficult to capture the correlation between different modalities in subsequent modal interactions. In contrast, the progressive modal alignment proposed in this invention creates a stronger correlation between the two modalities, making it easier to capture image-text relationships in subsequent modal interactions.

[0053] Simultaneously, this invention also tested the degree of enhancement that progressive alignment provides to images and text, such as... Figure 8 As shown in the line graph, if the image text enhancement is too weak, there will be no enhancement effect. On the contrary, if the enhancement coefficient is too large, the enhanced semantics will cover the semantics of the original image text. Both of these will lead to a decrease in performance. If a learnable enhancement coefficient is used, it will be affected by the imbalance fluctuation of a few samples, affecting the model's judgment of the overall sample group and weakening the model performance.

[0054] like Figure 10The image shown is a visual comparison of the modal distribution of the present invention and other methods.

[0055] Example 2 The present invention also provides a joint multimodal entity relation extraction system based on information representation and semantic alignment. The system is used to implement the method described in Embodiment 1. The system includes: an encoding module, an alignment module, a discrimination module, an interaction and weighting module, and an extraction module. The encoding module is used to acquire sample data consisting of original images and text, and to encode the input sample data using multiple encoders to obtain feature representations of visual and textual information. The alignment module is used to align the feature representations of visual and textual information using a progressive modal semantic alignment strategy. The discrimination module is used to determine the correlation coefficient between the feature representations of aligned visual information and text information by introducing a fine-grained guided multi-layer correlation mapping mechanism. Interaction and weighting are used to perform visual representation and text interaction using the multimodal interaction module to obtain multimodal semantic features; the routing weighting function is used to perform weighted mapping on the multimodal semantic features to finally obtain the multimodal feature representation; The extraction module is used to feed the multimodal feature representations into the word pair relation annotation extractor to extract entities, entity relations, and entity attribute quintuples.

[0056] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A method for joint multi-modal entity relation extraction based on information representation and semantic alignment, characterized in that, The method includes: Step 1: Obtain sample data consisting of the original image and text, use multiple methods to obtain multi-angle visual information representation of the image, and use multiple encoders to encode the input sample data and visual information representation to obtain feature representations of visual and text information; Step 2: Use a progressive modal semantic alignment strategy to align the feature representations of visual and textual information; Step 3: By introducing a fine-grained guided multi-layer correlation mapping mechanism, the correlation coefficient between the feature representations of aligned visual and textual information is determined, and features irrelevant to the core task are filtered out. Step 4: Utilize the multimodal interaction module to perform visual representation and text interaction to obtain multimodal semantic features; use the routing weighting function to perform weighted mapping on the multimodal semantic features to finally obtain the multimodal feature representation; Step 5: Feed the multimodal feature representations into the word pair relation annotation extractor to extract entity, entity relation, and entity attribute quintuples.

2. The method according to claim 1, characterized in that, In step one, multiple methods are used to obtain multi-angle visual information representations of the image, and multiple encoders are used to encode the input sample data and visual information representations. The methods for obtaining feature representations of visual and textual information include: The pre-trained language model BERT is used as a text encoder to encode the original text information to obtain the feature representation of the text. T represents the encoded text features; Multiple strategies are employed to extract multi-angle visual representations from the original image: The original image I is encoded using a pre-trained visual encoder model to extract multi-level visual features perceived by ViT. , ,in This indicates that the initial coding layer of ViT can encode the original image into a feature representation. Let m represent the ViT hierarchical block and m represent the ViT level. The features obtained from the 12 ViT levels are concatenated to obtain the original image encoding representation. , This represents the features mapped from the original image I through the initial coding layer. The features of ViT at layer m; The visual grounding toolkit is used to detect local regions containing objects in an image, and the regions most relevant to the semantics of the original text are extracted from the original image I. Each fine-grained image region is encoded using a pre-trained ViT encoder to obtain an encoded feature representation. ; The original image is fed into the BLIP model and the Faster-RCNN model to generate global descriptive text (Image caption) and image tags (Image tags) for highlighting regions of the image, respectively. Feature representations are then obtained by encoding using a pre-trained BERT language model. , ,in and These represent the encoded image descriptive text and object labels, respectively.

3. The method according to claim 2, characterized in that, In step two, the method for aligning the feature representations of visual and textual information using a progressive modal semantic alignment strategy includes: The descriptive text and object labels of the image are concatenated and then fed into a trainable Transformer block for representation. This yields the semantic features of the descriptive text and object labels; The obtained semantic features are used to perform semantic enhancement on the original image, fine-grained image regions, and text features. The expression is as follows: ; ; ,in This represents the cross-attention mechanism. As a query As keys and values, Manually defined keys Scaling factor and The coefficient weights represent the semantic enhancement of textual and visual information; The enhanced textual and visual semantic representations are brought closer together by using a contrastive learning mechanism.

4. The method according to claim 3, characterized in that, Methods that leverage contrastive learning mechanisms to further bridge the gap between enhanced textual and visual semantic representations include: The original image-text pairs are divided into multiple batches. N image-text pairs from each batch are used to construct positive and negative samples, where the same image-text pair... As positive samples, different image and text pairs Considered as negative samples, the text-to-image similarity within a batch is defined as: ,in express Cosine similarity between them Let T and V represent the original text information encoding and the original image encoding after semantic enhancement in a batch, respectively. T represents the text modality, V represents the image modality, and i and j are the i-th and j-th sample pairs in a batch of N samples, respectively. , It is a learnable temperature parameter; image-to-text similarity within a batch: ,in express Cosine similarity between them; Based on the text-to-image similarity and image-to-text similarity within a batch, calculate the image-to-text contrast loss. Contrast loss between text and image : ; Based on image-to-text contrastive loss and text-to-image contrastive loss, a batch of image-text loss values ​​is obtained to further enhance the semantic representation consistency between images and text. .

5. The method according to claim 4, characterized in that, In step three, the method for determining the correlation coefficient between visual and textual information and filtering features irrelevant to the core task by introducing a fine-grained guided multi-layer correlation mapping mechanism includes: ； In the formula, This means merging the visual features of 12 layers into the features of 1 layer. This indicates that the h-th part aggregates features from 12 layers. Indicates the first The characteristics of each part are derived from their convergence.

6. The method according to claim 5, characterized in that, Step 4: Utilize the multimodal interaction module to perform visual representation and text interaction to obtain multimodal semantic features; Methods for obtaining multimodal feature representations by weighting multimodal semantic features using routing weighting functions include: ； In the formula, G represents the original image, fine-grained image, region descriptive text, and multimodal features resulting from the interaction between object labels and text features T, respectively. Indicates weighted routing. express and The linear mapping function, , Indicates the 0th, 1st to... The weights of the features, where T represents the original text encoded features. express , [ ; ; ; ] indicates concatenation operation.

7. The method according to claim 6, characterized in that, In step five, the multimodal feature representations are fed into the word pair relation annotation extractor, and the method for extracting entities, entity relations, and entity attribute quintuples includes: Terms in multimodal features and The resulting word pair representations are fed into a linear prediction layer to obtain the probability distribution of the word pair labels. Used to predict the quintuple <subject, subject attribute, object, object attribute, subject-object relation>, where... Learnable mapping matrix This represents the learnable initial bias vector.

8. A joint multimodal entity relation extraction system based on information representation and semantic alignment, the system being used to implement the method described in any one of claims 1-7, characterized in that, The system includes: an encoding module, an alignment module, a discrimination module, an interaction and weighting module, and an extraction module; The encoding module is used to acquire sample data consisting of original images and text, acquire multi-angle visual information representations of the images using multiple methods, and encode the input sample data and visual information representations using multiple encoders to acquire feature representations of visual and text information. The alignment module is used to align the feature representations of visual information and text information using a progressive modal semantic alignment strategy; The discrimination module is used to determine the correlation coefficient between the feature representations of aligned visual information and text information by introducing a fine-grained guided multi-layer correlation mapping mechanism, and to filter features that are irrelevant to the core of the task. The interaction and weighting are used to perform visual representation and text interaction using the multimodal interaction module to obtain multimodal semantic features; the multimodal semantic features are then weighted and mapped using a routing weighting function to finally obtain a multimodal feature representation. The extraction module is used to feed the multimodal feature representation into the word pair relation annotation extractor to extract entities, entity relations, and entity attribute quintuples.

Images