A multi-modal relation extraction method based on media-based attention mechanism
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- UNIV OF ELECTRONICS SCI & TECH OF CHINA
- Filing Date
- 2024-09-13
- Publication Date
- 2026-07-03
Smart Images
Figure CN119129732B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of multimodal fusion and relation extraction technology, specifically to a multimodal relation extraction method based on a media attention mechanism. Background Technology
[0002] Relation extraction (RE) is a core task aimed at identifying relationships between entities from unstructured text corpora. This work is crucial for building large-scale knowledge graphs and advancing downstream applications. Early methods largely relied on statistical techniques and embedding methods; however, with the rise of neural networks, sequence models and dependency-driven frameworks have significantly improved performance.
[0003] Despite these advances, over-reliance on textual data reveals limitations, particularly in short texts and noisy social media texts where contextual cues are often scarce. Combining visual content with textual information offers a powerful solution by supplementing missing semantic information and enhancing the accuracy of relation extraction, especially in environments dominated by short texts and pervasive noise. This has driven the development of Multimodal Relation Extraction (MRE), a form of information extraction specifically designed to discover relationships between entities from the intersection of textual and visual data. MRE tasks are characterized by sifting through multimedia posts, news articles, and various mixed-media documents to extract structured knowledge to seed or enrich existing knowledge graphs.
[0004] Taking the widely used Multimodal Named Entity Relation Extraction (MNRE) dataset as an example, it was found that the average length of each sentence is relatively short. The MNRE v2 dataset not only provides multimodal data (including images), but also significantly increases the diversity of entity and relation types, while maintaining a reasonable sample size suitable for training and evaluating complex multimodal relation extraction models.
[0005] Existing multimodal relation extraction techniques face four main challenges: First, modal alignment noise: Non-task-related information in images and text can interfere with the model, reducing the accuracy and discriminative power of encoded information. Second, cross-modal fusion differences: The model needs to effectively capture complementary relationships between different modalities while preserving the unique information characteristics of each modality. Third, relative positional information in text: Accurately understanding the positional and semantic relationships between entities in natural language sequences is crucial for relation extraction. Fourth, the uniqueness of classification labels: Reflecting the semantic differences behind different relation labels during model training is essential for accurate classification.
[0006] Although recent research advances have addressed some of the aforementioned challenges and made significant contributions to the field, limitations still exist when dealing with highly complex multimodal relation extraction tasks, especially when fine-grained cross-modal information matching and deep semantic consistency are required. Summary of the Invention
[0007] The technical problem to be solved by this invention is to propose a deep learning model that can effectively improve the performance of multimodal relation extraction. This model is specifically designed for relation extraction tasks in short text and noisy environments, and aims to improve the accuracy and recall of relation extraction by fusing textual and visual information.
[0008] The technical solution adopted by the present invention to solve the above-mentioned technical problems is as follows:
[0009] A multimodal relation extraction method based on media attention mechanism is proposed, including the following steps:
[0010] First, a multimodal relation extraction model is established, and then this model is used to process text and images:
[0011] A1. For text and image samples that require relation extraction, encode them using a pre-trained encoder to extract feature representations of the text and images.
[0012] A2. Use an autoencoder to perform coarse-grained autoencoding on the feature representations of pre-trained text and images;
[0013] A3. Construct a cross-encoder based on an intermediate mediating cross-attention mechanism to fuse text and image representations;
[0014] A4. Decoding of relation extraction tasks is performed using a relation extraction encoder and a conditional random field, where the relation extraction encoder is used to predict relation categories and the conditional random field is used to optimize entity boundary detection.
[0015] Then, a multimodal relation extraction model is used for relation extraction training and specific relation extraction tasks.
[0016] B1. When training the model, use the different representations and prediction results obtained in the previous steps to set different joint learning sub-tasks and loss functions for joint training, and optimize the model parameters to improve the model performance through gradient descent.
[0017] B2. When performing multimodal relation extraction tasks, the trained model is used directly to predict new multimodal data. The output of the relation extraction decoder is used as the relation probability, and the highest score is taken as the relation extraction prediction result.
[0018] Furthermore, the pre-trained encoder includes a Transformer-based bidirectional transformer encoder (BERT) and an image encoder; for a given sentence T, it is input into the BERT model to obtain its hidden state representation x. t For a given image I, convert it into a representation x that is compatible with text features. v ;
[0019] Furthermore, the autoencoder is specifically a variational autoencoder, and the autoencoding process is as follows:
[0020] For text encoding:
[0021] μ t =f(x) t )
[0022] σ t =f(x) t )
[0023] Where f is the self-attention module, μ t It is the mean of the text encoding, σ t It is the standard deviation of text encoding;
[0024] For image encoding:
[0025] μ v =f(x) v )
[0026] σ v =f(x) v )
[0027] Where, μ v It is the mean of the image encoding, σ v It is the standard deviation of image coding;
[0028] In variational autoencoders, reparameterization techniques are used to sample from the latent spaces of text and images to ensure that the latent variables follow a normal distribution. The sampled latent variables are:
[0029]
[0030] Where z t The latent variable representing text encoding, z v Represents the latent variables of image encoding.
[0031] Furthermore, the cross-encoder based on the intermediary cross-attention mechanism employs a unique observer structure as an intermediary for cross-modal attention, updating information through two layers of alternating propagation; the specific process of step A3 is as follows:
[0032] Step S3.1: First, define the initial observer w:
[0033] w = [x t ;x v ]
[0034] During the first propagation, modal information is extracted using latent variables sampled from the initial observers, and this is used as the first update for the observers, represented as:
[0035] w0 = g(z0, w)
[0036] Where, z0 = [z t ;z v ]; g represents cross-attention mechanism, using z0 as the query in the cross-attention mechanism, and w0 as the key and value;
[0037] Step S3.2: Observer-guided hierarchical coding;
[0038] In the hierarchical coding process, the observer is repeatedly updated using multimodal information, and then the observer is used as a guide to promote cross-modal information interaction. This process is formalized as a series of equations, as follows:
[0039] Begin with the following:
[0040] w1 = f(w0)
[0041] w1 = [w v1 w t1 ]
[0042] Where f represents a self-attention mechanism, using the same input as a query, key, and value;
[0043] Then:
[0044] h t1 =h(z) t w v1 , z v )
[0045] h v1 =h(z) v w t1 , z t )
[0046] Among them, h t1 and h v1 Encoding the text and image obtained from the first observer attention propagation; h() represents the observer attention mechanism, z v w t1 , z t These are respectively used as the query, key, and value in the attention mechanism;
[0047] Next is:
[0048] w2=f([h t1 h v1 ])
[0049] h t2 =h(h t1 w v2 h v1 )
[0050] h v2 =h(h v1 w t2 h t1 )
[0051] o1 = [h t2 h v2 ]
[0052] Where w2 is the observer code of the second observer mechanism, h t2 and h v2 This encodes the text and image obtained from the second observer attention propagation; o1 is the output of the first-round mutual encoder.
[0053] Based on the first-round observer mechanism, a variational autoencoder is used to encode the latent variable output of the single-modality system, namely:
[0054] z′ t =VAE(x t h t2 )
[0055] z′ v =VAE(x v h v2 )
[0056] z1=[z′ t ;z′ v ]
[0057] Where, z′ t , z′ v The text and image modal encodings after sampling by the second variational autoencoder are used to form the final output z1 of the second variational autoencoder.
[0058] Subsequently, a second round of the observer mechanism was conducted:
[0059] w3 = g(z1, o1, o1)
[0060] h t3 =h(z′) t w v3 , z′ v )
[0061] h v3 =h(z′)v w t3 , z′ t )
[0062] w4=f([h t3 h v3 ])
[0063] h t4 =h(h t3 W v4 h v3 )
[0064] h v4 =h(h v3 W t4 h t3 )
[0065] o2=[h t4 h v4 ]
[0066] Here, w3 is the observer code initialized in the second round of the observer mechanism, and the update process of the other variables is the same as that in the first round of the observer mechanism.
[0067] Furthermore, the relation extraction encoder is a multilayer perceptron (MLP), and the output of the relation extraction encoder is the final prediction result.
[0068] The conditional random field is used as an auxiliary task during training, and supervised training is performed using the existing entity location annotations in the dataset, as specifically shown below:
[0069] Y crf =crf(mlp(h t4 ))
[0070] Here, mlp represents a multilayer perceptron (MLP), and crf represents a conditional random field.
[0071] During model training, the model's total loss function L total as follows:
[0072] L total =θL VAE +αL align +βL watcher +γL RE +δL InfoNCE +∈L CRF +ζL recon
[0073] Where θ, α, β, γ, δ, ∈, ζ are all weights of the loss function for each optimization objective, and L VAE L represents the loss of a single-modal autoencoder, measuring the difference between the original and reconstructed variables in a variational autoencoder;align L represents the alignment loss, which enhances the correlation between modalities by maximizing the similarity score of matched pairs and minimizing the score of unmatched pairs; watch Represents observer loss, maintaining high similarity between the final entity embedding and the original observer embedding during observer attention propagation; L RE This represents the relation extraction loss, which is used to supervise the comparison between the model's predictions and the true labels to ensure accurate relation predictions; L InfoNCE Represents InfoNCE loss, enhancing the consistency between entity embedding and label embedding; L CRF L represents the CRF loss, which optimizes entity boundary detection to improve the accuracy of entity recognition; recon This represents the reconstruction loss, ensuring that the model can accurately reconstruct the masked modal information.
[0074] The beneficial effects of this invention are:
[0075] 1) By combining textual and visual information, this invention effectively supplements the missing contextual cues in short texts, and this method significantly improves the accuracy of relation extraction, especially in social network environments. Specifically, by utilizing visual information to enhance the semantic connections between entities, the model can more accurately identify relationships between entities, thereby improving the overall performance of multimodal relation extraction.
[0076] 2) This invention proposes a joint learning framework that integrates multiple loss functions, including alignment loss and relation extraction loss. These loss functions work together at different levels of the model, enhancing its ability to handle complex multimodal data. This comprehensive learning strategy enables the model to better understand and integrate information from different modalities, thereby demonstrating excellent performance in multimodal relation extraction tasks. Attached Figure Description
[0077] Figure 1 This is a flowchart of the overall relationship extraction method in this invention.
[0078] Figure 2 This is a schematic diagram illustrating the construction and coordination of training objectives in this invention. Detailed Implementation
[0079] This invention aims to propose a deep learning model that can effectively improve the performance of multimodal relation extraction. This model is specifically designed for relation extraction tasks in short text and noisy environments, and aims to improve the accuracy and recall of relation extraction by fusing textual and visual information. Its overall implementation process is as follows: Figure 1 As shown:
[0080] S1. For text and image samples that need relation extraction, the raw information is converted into tensor representations required by deep learning by encoding them through a pre-trained encoder trained on a large number of data samples.
[0081] S2. Use variational autoencoders (VAE) to coarsely encode text and images in their respective modalities to make the sample distribution more reasonable;
[0082] S3. By using a cross-attention mechanism with an intermediary, text and image representations are fused to give the fused representation richer semantic information.
[0083] S4. The prediction task of relation extraction using fully connected layers and the prediction task of entity boundaries using conditional random fields (CRF) are respectively used.
[0084] S5. When training the model, use the different representations and prediction results obtained in the previous steps to set different joint learning sub-tasks and loss functions for joint training. Optimize the model parameters and improve the model performance through gradient descent. When performing a specific prediction task, use the trained model parameters to make predictions directly and obtain the prediction results directly based on the scores.
[0085] Example:
[0086] The following example uses a pair of entities (head entity e) with m candidate relation types. h Tail entity e t Using the example of the entity pair and the text containing it, the solution of the present invention will be described in detail below. The specific implementation process is as follows:
[0087] Step 1: First, preprocess the sample data. Data preprocessing specifically includes two sub-steps: pre-training encoding and VAE encoding.
[0088] Step 11: For the text and image samples that require relation extraction, encode them using a pre-trained encoder to extract feature representations of the text and images, and map the features of the text and images into a shared representation space:
[0089] In the model, a pre-trained BERT (Transformer-based bidirectional transformer encoder) is used as the text encoder to obtain the contextual representation of the sentence. For image I, an image encoder ResNet (Residual Network Image Encoder) is used to extract image features.
[0090] For a given sentence T, input it into the BERT model to obtain its hidden state representation x. tThese hidden representations are passed through multiple encoder layers. Each encoder layer typically contains two sub-layers: a multi-head self-attention mechanism and a feedforward neural network. Each layer is followed by residual connections and layer normalization. Ultimately, we can choose to use the hidden states of different layers, such as the last layer or a combination of multiple layers, to obtain richer text representations. Formally, this process can be represented as:
[0091] x t =BERT(T)
[0092] An image encoder processes image I, converting it into a representation x that is compatible with text features. v This can be expressed as:
[0093] x v =ImageEncoder(I)
[0094] To demonstrate the model's effectiveness, common pre-trained encoders such as BERT and ResNet are used. Following this module, the original task input is transformed into embeddings for subsequent stages. This transformation process can be symbolically represented as:
[0095] (T,I)→(x t x v )
[0096] Step 12: The pre-trained encoder was refined and fine-tuned using an autoencoder:
[0097] The autoencoder component employs a variational autoencoder (VAE), a type of autoencoder that adds probability distortions during the autoencoding process. Its main goal is to extract and encode the internal information of unimodal features based on a pre-trained encoder.
[0098] For text encoding:
[0099] μ t =f(x) t )
[0100] σ t =f(x) t )
[0101] Where f is the self-attention module. μ t It is the mean of the text encoding, σ t It is the standard deviation of text encoding.
[0102] Similarly, for image encoding:
[0103] μ v =f(x) v )
[0104] σ v =f(x)v )
[0105] Where, μ v It is the mean of the image encoding, σ v It is the standard deviation of image encoding.
[0106] In VAEs, reparameterization techniques are used to sample from the latent spaces of text and images to ensure that the latent variables follow a normal distribution. The sampled latent variables are:
[0107]
[0108] These variables will be used in subsequent stages of processing; μ and σ in the formula represent the mean and standard deviation, respectively.
[0109] Step 2: Construct the intermediate medium and update and integrate the image, text, and intermediate medium:
[0110] The cross-encoder is designed to merge feature representations of text and images. It employs a unique structure called the "observer" as a mediator for cross-modal attention. The cross-encoder uses a stacked structure to update information through alternating propagation in two layers.
[0111] Step 21: To preserve the most original text and image information after pre-training, first define the initial observer:
[0112] w = [x t ;x v ]
[0113] During the first propagation, latent variables sampled from the initial observers are used to extract the main modal information, which serves as the observer's first update and can be represented as:
[0114] W0 = g(z0, w)
[0115] Where, z0 = [z t ;z v ]
[0116] g represents the cross-attention mechanism, using z0 as the query and W0 as the key and value to supplement the original modal information after sampling.
[0117] Step 22, Observer-guided hierarchical coding:
[0118] In the hierarchical coding process, the observer is repeatedly updated using multimodal information, and then the observer is used as a guide to promote cross-modal information interaction, forming the basis for cross-modal coding. This process can be represented as a series of equations, starting with the following:
[0119] w1 = f(W0)
[0120] w1 = [w v1 w t1 ]
[0121] Unlike the previous 'g', 'f' represents a self-attention mechanism that uses the same input as both query, key, and value to supplement the feature depth of the representation information and the learnability of the encoding process. Here, w... v1 w t1 This indicates the embedding of the corresponding image and text in w1.
[0122] Then:
[0123] h t1 =h(z) t w v1 , z v )
[0124] h v1 =h(z) v w t1 , z t )
[0125] Among them, h t1 and h v1 The text and image obtained from the first observer attention propagation are encoded, h() represents the observer attention mechanism, and z v w t1， z t These are respectively used as the query, key, and value in the attention mechanism.
[0126] Next is:
[0127] w2=f([h t1 h v1 ])
[0128] h t2 =h(h t1 w v2 h v1 )
[0129] h v2 =h(h v1 w t2 h t1 )
[0130] o1 = [h t2 h v2 ]
[0131] Where w2 is the observer code of the second observer mechanism, h t2 and h v2This is used to encode the text and image obtained from the second observer attention propagation. During each update, the observer encoding is fused with encodings from different modalities, and then fed back to guide the interaction between these modalities. Finally, o1 is the output of the first round of the inter-encoder.
[0132] Based on the first round of the observer mechanism, the VAE from step 12 is used again to encode the latent variable output of the single modality, that is:
[0133] z′ t =VAE(x t h t2 )
[0134] z′ v =VAE(x v h v2 )
[0135] z1=[z′ t ;z′ v ]
[0136] Where, z′ t , z′ v The text and image modal codes after the second VAE sampling are used to form the final output z1 of the second VAE.
[0137] Subsequently, a second round of the observer mechanism was conducted:
[0138] w3 = g(z1, o1, o1)
[0139] h t3 =h(z′) txt w v3 , z′ img )
[0140] h v3 =h(z′) img w t3 , z′ txt )
[0141] w4=f([h t3 h v3 ])
[0142] h t4 =h(h t3 w v4 h v3 )
[0143] h v4 =h(h v3 W t4 h t3 )
[0144] o2=[ht4 h v4 ]
[0145] Here, w3 is the observer code initialized in the second round of the observer mechanism, and the update process of the other variables is the same as that in the first round of the observer mechanism.
[0146] Step 3: After obtaining the final encoding result from the previous step, the decoder will be used to complete the final prediction. The decoder consists of two multilayer perceptrons (MLPs), one for relation extraction (RE) and the other for conditional random field (CRF) tasks. The output of the relation extraction encoder is the final prediction result, while the output of the conditional random field serves as an auxiliary task during training. Supervised training is performed using the existing entity location annotations in the dataset to improve the model's semantic segmentation ability of entity boundaries.
[0147] Step 31, Relation Extraction (RE) Decoder:
[0148] The RE decoder aims to extract relations from merged textual and visual feature embeddings. This is accomplished using an MLP that learns a complex mapping between input features and target relations. Let e h and e t Each represents the final representation h. t4 The embeddings corresponding to the head and tail entities are given in the original embedding. These embeddings have been interacted with multiple times to ensure that relative positional information is not lost while effectively propagating cross-modal information. Formally, the RE prediction process can be represented as:
[0149] Y re =mlp(e h e t )
[0150] Step 32, Conditional Random Field (CRF) Decoder:
[0151] The CRF decoder is designed for sequence labeling tasks, aiming to optimize entity boundary detection based on entity location annotations provided by the RE dataset. This is implemented using another MLP, whose output serves as input to the CRF layer. The CRF layer considers the context within the sequence to more accurately delineate boundaries around entities, which is crucial for relation extraction tasks. The CRF prediction process can be represented as:
[0152] Y crf =crf(mlp(h t4 ))
[0153] By simultaneously supervising RE and CRF performance during training, annotation information in the dataset can be utilized more effectively, thereby improving its performance in multimodal relation extraction tasks.
[0154] Step 4: When addressing modal differences and alignment challenges in multimodal relation extraction, such as... Figure 2 As shown, a joint learning training framework with multiple optimization objectives was constructed. By utilizing the representations and prediction results generated in previous steps, new auxiliary tasks and loss functions were built to help improve model training performance. Specifically, the following training objectives were established:
[0155] Step 41: For single-modal autoencoders, a variational autoencoder method is employed. Two specific objectives are pursued: minimizing the Kullback-Leibler divergence loss (KDL) and the L2 distance loss, which together constitute the VAE loss. These objectives ensure that the encoded samples conform to a Gaussian distribution while limiting the variation between the original and encoded samples.
[0156] L2 distance loss measures the difference between the original variable and the reconstructed variable, and the formula is as follows:
[0157]
[0158] Among them, z i x represents the VAE encoded vector of a certain mode. i This represents the VAE pre-encoding vector for a certain modality, where N is the number of samples in the batch. The norm ||·|| represents the Euclidean distance between vectors, providing a measure of dissimilarity between the original and reconstructed data points.
[0159] KDL loss aims to regularize the distribution of latent variables to a standard normal distribution, as shown in the following formula:
[0160]
[0161] in, σ is the expected value symbol, representing the expectation of a random variable z with respect to its posterior distribution q(z|x). Here, q(z|x) represents the distribution of the latent variable z given data x. μ and σ 2 These are the mean and standard deviation of the coding distribution, respectively.
[0162] Step 42: To enhance the correlation between modal embeddings after unimodal encoding, adversarial training is introduced to optimize alignment. Specifically, the goal is to increase the similarity score for matched image-text pairs and decrease the score for mismatched pairs.
[0163] Let s ij Let represent the similarity score between the i-th image embedding and the j-th text embedding. The optimization objective is to maximize s when i = j (positive). ij Minimize s when i ≠ j (negative pair) ij .
[0164] These scores are computed using a scoring function S(·,·) learned through fully connected layers. The loss function for adversarial training is defined as follows:
[0165] The direct loss is:
[0166] L pos =-log(p pos )
[0167] Negative loss is:
[0168] L neg = -log(1-p neg )
[0169] Where, probability p pos and p neg Defined as:
[0170] p pos =σ(S(z) t , z v ))
[0171] p neg =σ(S(z) t , z′ v ))
[0172] Where, z′ v To avoid being with z t Other corresponding image samples. The total alignment loss is the sum of the positive and negative alignment losses:
[0173] L align =L pos +L neg
[0174] Step 43: To effectively integrate visual information into the text representation without losing the inherent structure of the text, an observer mechanism is introduced to supplement the traditional cross-attention mechanism. The goal is to maintain high similarity between the final entity embedding and the original observer embedding to preserve relative positional information.
[0175] Let p watcher This represents the probability score generated by applying the scoring function to the concatenation of the initial observer embedding w0 and the final output o2:
[0176] p watcher =σ(S(w0, o2))
[0177] Where S(·,·) is the scoring function learned through the fully connected layer, and σ(·) is the sigmoid activation function.
[0178] Given probability scores, the observer loss is defined as p watcherBinary cross-entropy loss between the target label (y=1) (indicating high similarity):
[0179] L watcher =-[ylog(p watcher )+(1-y)log(1-p watcher )]
[0180] In practice, since the target label is always set to 1, the observer loss simplifies to:
[0181] L watch =-log(p watch )
[0182] This formula encourages the model to produce high probability scores when comparing the final entity embedding with the original observer embedding, thereby preserving relative positional information during the fusion process.
[0183] Step 44: For relation extraction, use traditional cross-entropy loss to supervise the comparison between the model's predictions and the true labels:
[0184]
[0185] Among them, y i It is a real label, p(y) i ) is the predicted probability.
[0186] Furthermore, to encourage the learning of semantically consistent embeddings of entities with the same labels, a learnable label matrix is utilized. Inspired by InfoNCE, a cosine-based scoring function is used to calculate the similarity score between entity embeddings and label embeddings:
[0187]
[0188] Where fcos(·,·) represents the cosine similarity function, l true It corresponds to [e] h :e t The true tag embedding of [e], l is a representation of [e] h :e t The InfoNCE loss function compares all labels or samples, including both positive and negative examples, to each other. By maximizing the score of positive examples relative to all other samples, the InfoNCE loss function enables the model to learn high similarity between positive examples while reducing similarity with other negative examples.
[0189] The InfoNCE loss aims to maximize the similarity between an entity embedding and its true label embedding, relative to other label embeddings. This encourages the model to learn representations that make entities with similar labels closer together in the embedding space.
[0190] Step 45: To enhance the model's sensitivity to entity locations, train it to predict entity locations using Conditional Random Field (CRF) loss. The CRF loss is as follows:
[0191]
[0192] Among them, s true It is the score of the labeled sequence of real entities in the dataset, ∑ s exp(s) is applied to all possible entity sequences. CRF is a commonly used sequence labeling model in natural language processing for tasks such as named entity recognition and part-of-speech tagging. The loss function measures the difference between the model's predicted sequence and the true sequence. Using this loss function, the goal of the CRF model is to minimize L... CRF This means maximizing the score of the true sequence relative to all other sequences. In other words, the model adjusts its parameters to ensure that the score of the correctly labeled sequence is as high as possible compared to all other sequences, thereby improving the model's performance on sequence labeling tasks.
[0193] NER (Named Entity Recognition) tags are derived from relation extraction datasets by identifying the start and end positions of entities in sentences. These tags are encoded as IDs to ensure compatibility with the model's input format, facilitating the CRF layer's learning from structured entity information.
[0194] Step 46: To promote robust cross-modal representation learning, a reconstruction loss mechanism is adopted. This loss encourages the model to accurately reconstruct the masked parts of the text and visual input based on the unmasked context. The loss formula is as follows:
[0195] L recon =-S(x) unmasked x reconstructed )
[0196] Where S(·,·) is a scoring function used to measure the pre-masked embedding x. unmasked and its refactored version x reconstructed The similarity between them. Wherein, we use z i The output vector of the first VAE encoding is represented as x. unmasked h i4 The output reconstruction vector of the second-round observer mechanism is represented as x. reconstructed .
[0197] In practice, this involves masking specific markers from text input and masking specific channels from visual input. The scoring function module evaluates the reconstruction quality by comparing the reconstructed result with the original unmasked data. A binary cross-entropy loss is used to penalize the difference between the predicted score and the true label (1 for positive examples), aiming to maximize the model's ability to accurately reconstruct missing information.
[0198] Step 5: Integrating the objectives described in this paper, the model's total loss function prioritizes the variational autoencoder (VAE) component, followed by the losses from multimodal feature fusion, relation extraction, and entity recognition.
[0199] L total =θL VAE +αL align +βL watcher +γL RE +δL InfoNCE +∈L CRF +ζL recon
[0200] Where θ, α, β, γ, δ, ∈, and ζ are the weights of the loss function for each optimization objective. VAE It includes the VAE loss described in detail in previous chapters, serving as the fundamental loss component for optimizing the distribution of latent variables. Subsequent terms are task- and modality-specific, each weighted by corresponding hyperparameters. By balancing these losses through hyperparameter tuning, the model can excel across multiple tasks simultaneously, from latent variable modeling to complex multimodal understanding and relational reasoning.
[0201] When performing a prediction task, steps 4 and 5 are omitted. The output of the relation extraction decoder in step 31 is used directly as the relation probability, and the one with the highest score is taken as the relation extraction prediction result.
[0202] Although the present invention has been described herein with reference to embodiments thereof, the above embodiments are merely preferred embodiments of the present invention, and the implementation of the present invention is not limited to the above embodiments. It should be understood that those skilled in the art can design many other modifications and implementations, which will fall within the scope and spirit of the principles disclosed in this application.
Claims
1. A method for multi-modal relation extraction based on media-based attention mechanism, characterized in that, Includes the following steps: First, a multimodal relation extraction model is established, and then this model is used to process text and images: A1. For text and image samples that require relation extraction, encode them using a pre-trained encoder to extract feature representations of the text and images. A2. Use an autoencoder to perform coarse-grained autoencoding on the feature representations of pre-trained text and images; A3. Construct a cross-encoder based on an intermediate mediating cross-attention mechanism to fuse text and image representations; A4. Decoding of relation extraction tasks is performed using a relation extraction encoder and a conditional random field, where the relation extraction encoder is used to predict relation categories and the conditional random field is used to optimize entity boundary detection. Then, a multimodal relation extraction model is used for relation extraction training and specific relation extraction tasks. B1. When training the model, use the different representations and prediction results obtained in the previous steps to set different joint learning sub-tasks and loss functions for joint training, and optimize the model parameters to improve the model performance through gradient descent. B2. When performing multimodal relation extraction tasks, the trained model is used directly to predict new multimodal data. The output of the relation extraction decoder is used as the relation probability, and the highest score is taken as the relation extraction prediction result. The cross-encoder based on the intermediary cross-attention mechanism employs a unique observer structure as an intermediary for cross-modal attention, updating information through two layers of alternating propagation; step A3 is specifically as follows: Step A3.1: First define the initial observer : in, The hidden state representation for a given sentence; A representation of a given image; During the first propagation, modal information is extracted using latent variables sampled from the initial observers, and this is used as the first update for the observers, represented as: in, , Potential variables representing text encoding Represents the latent variables of image encoding; g represents the cross-attention mechanism, using As a query in the cross-attention mechanism As keys and values; Step A3.2: Observer-guided hierarchical coding; In the hierarchical coding process, the observer is repeatedly updated using multimodal information, and then the observer is used as a guide to promote cross-modal information interaction.
2. The multimodal relation extraction method based on media attention mechanism according to claim 1, characterized in that, The pre-trained encoders include the Transformer-based bidirectional transformer encoder BERT and an image encoder; for a given sentence T, it is input into the BERT model to obtain its hidden state representation. For a given image I, convert it into a representation compatible with text features. .
3. The multimodal relation extraction method based on media attention mechanism according to claim 2, characterized in that, The autoencoder is specifically a variational autoencoder, and the autoencoding process is as follows: For text encoding: Where f is the self-attention module, It is the average of the text encoding. It is the standard deviation of text encoding; For image encoding: in, It is the mean of the image encoding. It is the standard deviation of image coding; In variational autoencoders, reparameterization techniques are used to sample from the latent spaces of text and images to ensure that the latent variables follow a normal distribution. The sampled latent variables are: in Potential variables representing text encoding Represents the latent variables of image encoding.
4. The multimodal relation extraction method based on media attention mechanism according to claim 3, characterized in that, The observer-guided hierarchical coding is formalized as a series of equations, as follows: Begin with the following: Where f represents the self-attention mechanism, which uses the same input as the query, key and value; Then: in, and The text and image obtained from the first observer attention propagation are encoded; h() represents the observer attention mechanism. These are respectively used as the query, key, and value in the attention mechanism; Next is: in, Observer coding for the second observer mechanism, and Encoding of text and images obtained from the second observer's attention propagation; This is the output of the first round of the mutual encoder; Based on the first-round observer mechanism, a variational autoencoder is used to encode the latent variable output of the single-modality system, namely: in, , The text and image modal codes sampled by the second variational autoencoder are used to form the final output of the second variational autoencoder. ; Subsequently, a second round of the observer mechanism was conducted: in, The observer encoding is initialized in the second round of the observer mechanism, and the update process for the remaining variables is the same as that in the first round of the observer mechanism.
5. The multimodal relation extraction method based on media attention mechanism according to claim 4, characterized in that, The relation extraction encoder is a multilayer perceptron (MLP), and the output of the relation extraction encoder is the final prediction result. The conditional random field is used as an auxiliary task during training, and supervised training is performed using the existing entity location annotations in the dataset, as specifically shown below: in, This represents a multilayer perceptron (MLP). This represents a conditional random field.
6. The multimodal relation extraction method based on media attention mechanism according to claim 5, characterized in that, The total loss function of the model during training. as follows: in, These are all weights of the loss function for each optimization objective. This represents the loss of a single-modal autoencoder, which measures the difference between the original and reconstructed variables of the variational autoencoder. This represents the alignment loss, which enhances the correlation between modalities by maximizing the similarity score of matched pairs and minimizing the score of unmatched pairs. This represents the observer loss, maintaining a high similarity between the final entity embedding and the original observer embedding during observer attention propagation; This represents the relation extraction loss, which is used to supervise the comparison between the model's predictions and the true labels to ensure accurate relation predictions. This represents the InfoNCE loss, enhancing the consistency between entity embedding and label embedding; This represents the CRF loss, which optimizes entity boundary detection to improve the accuracy of entity recognition. This represents the reconstruction loss, ensuring that the model can accurately reconstruct the masked modal information.
7. The multimodal relation extraction method based on media attention mechanism according to claim 6, characterized in that, The specific loss function is as follows: VAE Loss: For the input and output of the variational autoencoder, divergence loss and L2 distance loss are used, and these two losses together constitute the VAE loss. Specifically, L2 distance loss The formula for measuring the difference between the original variable and the reconstructed variable is as follows: in, This represents the encoded vector of a single-mode variational autoencoder. This represents the vector before encoding of a single-modal variational autoencoder, where N is the number of samples in the batch, and the norm is... Represents the Euclidean distance between vectors; Divergence loss The distribution of the latent variables is regularized to a standard normal distribution, as shown in the following formula: in, It is the expected value symbol, representing the expected value with respect to the posterior distribution. The random variable z takes the expectation; here, This represents the distribution of the latent variable z given data x; and These are the mean and standard deviation of the coding distribution, respectively; Alignment Loss: Adversarial training is introduced to optimize alignment; specifically, let... This represents the similarity score between the i-th image embedding and the j-th text embedding. The optimization objective is to maximize the similarity score when i = j. Minimize when negative pair i ≠ j These scores are calculated using a scoring function S(·,·) learned through fully connected layers. The loss function for adversarial training is defined as follows: The direct loss is: Negative loss is: Among them, the positive probability and negative pair probability Defined as: in, It is the sigmoid activation function. To avoid For the corresponding other image samples, the total alignment loss is the sum of the positive and negative alignment losses: Observer Loss: Observer loss based on an intermediary-mediated cross-attention mechanism; assuming... This indicates that the scoring function is applied to the initial observer embedding. and final output The probability scores generated by the connection: Observer loss Specifically: Relation extraction loss: For relation extraction, cross-entropy loss is used to supervise the comparison between the model's predictions and the true labels. in, It's a real label. It is a predicted probability; InfoNCE Loss: Calculates a similarity score between entity embeddings and label embeddings using a cosine-based scoring function. Where fcos(·,·) denotes the cosine similarity function. It corresponds Real tag embedding, Yes All labels or samples compared, including both positive and negative examples; Conditional Random Field Loss: Using Conditional Random Field Loss To predict entity location, the following is a detailed method: in, It is the score of the labeled sequence of real entities in the dataset. It is performed on all possible sequences of entities; Reconstruction Loss: A reconstruction loss mechanism is employed, which encourages the model to accurately reconstruct the masked parts of the text and visual input based on the unmasked context. The formula is as follows: The output vector encoded by the first variational autoencoder is used as... The output reconstruction vector of the second-round observer mechanism is used as .