Medium-based attention mechanism multi-modal relation extraction method

Abstract

The invention relates to the technical field of multi-modal relation extraction, discloses a multi-modal relation extraction method of an attention mechanism based on media, and solves the problems of modal alignment noise, cross-modal fusion difference, retention of text relative position information, uniqueness of classification labels and the like in the existing method. By integrating comparative learning and variational auto-encoder constraints, the method minimizes irrelevant noise, and preferentially considers semantic data which is crucial to extraction of a multi-modal relationship. In order to improve the performance and training effect of the model, the model is trained by combining a plurality of optimization targets, including single-mode self-encoding, alignment loss, observer loss, relation extraction loss, InfoNCE loss, CRF loss, reconstruction loss and the like. And when a multi-modal relation extraction task is executed, the trained model is directly utilized to predict new multi-modal data, scores of all candidate relations are calculated through a scoring function, and finally the candidate relation with the highest score is selected as a relation extraction result.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal fusion and relationship extraction, and in particular to a multimodal relationship extraction method based on a medium attention mechanism. Background Art

[0002] Relation Extraction (RE) is a core task that aims to identify the relationships between entities from an unstructured text corpus. This work is crucial for building large-scale knowledge graphs and advancing downstream applications. Early methods mostly relied on statistical techniques and embedding methods; however, with the rise of neural networks, sequence models and dependency-driven frameworks have significantly improved the performance.

[0003] Despite these advances, over-reliance on textual data has revealed some limitations, especially in short and noisy social media texts where contextual clues are often scarce. Combining visual content with textual information offers a powerful solution by supplementing the missing semantic information and enhancing the accuracy of relation extraction, especially in environments dominated by short texts and pervasive noise. This has motivated the development of Multimodal Relation Extraction (MRE), a form of information extraction specifically designed to discover relations between entities from the intersection of textual and visual data. The MRE task is characterized by sifting through multimedia posts, news articles, and various mixed-media documents to extract structured knowledge to seed or enrich existing knowledge graphs.

[0004] Considering the widely used Multimodal Named Entity Relation Extraction (MNRE) dataset as an example, it is found that the average length of each sentence is shorter. The MNRE v2 dataset not only provides multimodal data (including images), but also significantly increases the diversity of entity and relation types, while maintaining a reasonable sample size suitable for training and evaluating complex multimodal relation extraction models.

[0005] Existing multimodal relationship extraction technology faces the following four challenges: First, modal alignment noise: non-task-related information in images and texts will interfere with the model, reducing the accuracy and distinguishing ability of the encoded information. Second, cross-modal fusion differences: the model needs to retain the unique information characteristics of each modality while effectively capturing the complementary relationship between different modalities. Third, text relative position information: Accurately understanding the position and semantic relationship between entities in natural language sequences is crucial for relationship extraction. Fourth, the uniqueness of classification labels: Reflecting the semantic differences behind different relationship labels during model training is crucial for accurate classification.

[0006] Although recent research advances have addressed some of the above challenges and made significant contributions to the development of the field, there are still limitations when dealing with high-complexity multimodal relation extraction tasks, especially when sophisticated cross-modal information matching and deep semantic consistency are required. Summary of the invention

[0007] The technical problem to be solved by the present invention is to propose a deep learning model that can effectively improve the performance of multimodal relationship extraction. The model is particularly aimed at relationship extraction tasks in short texts and noisy environments, and aims to improve the accuracy and recall of relationship extraction by integrating text and visual information.

[0008] The technical solution adopted by the present invention to solve the above technical problems is:

[0009] A multimodal relation extraction method based on medium attention mechanism is proposed, which includes the following steps:

[0010] First, we build a multimodal relationship extraction model and use it to process text and images:

[0011] A1. For text and image samples that need to be extracted, encode them through a pre-trained encoder to extract feature representations of text and images.

[0012] A2. Use autoencoders to perform coarse-grained autoencoding on the feature representations of pre-trained encoded text and images.

[0013] A3. Construct a cross-encoder based on the cross-attention mechanism of the intermediate medium to fuse text and image representations;

[0014] A4. Decode the relation extraction task through the relation extraction encoder and conditional random field, where the relation extraction encoder is used to predict the relationship category and the conditional random field is used to optimize the entity boundary detection;

[0015] Then use the multimodal relation extraction model to perform relation extraction training and specific relation extraction tasks;

[0016] B1. When training the model, use the different representations and prediction results obtained in the previous steps, set different joint learning subtasks and loss functions for joint training, and optimize the model parameters through the gradient descent method to improve the model performance;

[0017] B2. When performing multimodal relationship extraction tasks, directly use the trained model to predict new multimodal data, use the output of the relationship extraction decoder as the relationship probability, and take the one with the highest score as the relationship extraction prediction result.

[0018] Furthermore, the pre-trained encoder includes a Transformer-based bidirectional transformer encoder BERT and an image encoder; for a given sentence T, it is input into the BERT model to obtain its hidden state representation x t ; For a given image I, convert it to a representation x that is compatible with text features v ;

[0019] Furthermore, the autoencoder is specifically a variational autoencoder, and the autoencoding process is specifically as follows:

[0020] For text encoding:

[0021] μ t =f(x t )

[0022] σ t =f(x t )

[0023] Among them, f is the self-attention module, μ t is the mean of the text encoding, σ t is the standard deviation of the text encoding;

[0024] For image encoding:

[0025] μ v =f(x v )

[0026] σ v =f(x v )

[0027] Among them, μ v is the mean of the image encoding, σ v is the standard deviation of the image coding;

[0028] In the variational autoencoder, a reparameterization trick is used to sample from the latent space of text and images to ensure that the latent variables follow a normal distribution. The sampled latent variables are:

[0029]

[0030] where z t represents the latent variable for text encoding, z v Latent variables representing image encoding.

[0031] Furthermore, the cross-encoder based on the intermediate-medium cross-attention mechanism adopts the unique structure of the observer as the intermediary of cross-modal attention and updates information through two-layer alternating propagation; the specific process of step A3 is as follows:

[0032] Step S3.1: First define the initial observer w:

[0033] w=[x t ;x v ]

[0034] In the first propagation process, the latent variables sampled from the initial observer are used to extract the modal information as the first update of the observer, which is expressed as:

[0035] w 0 =g(z 0 , w)

[0036] Among them, z 0 =[z t ; z v ]; g represents the cross-attention mechanism, using z 0 As the query in the cross-attention mechanism, w 0 as keys and values;

[0037] Step S3.2: Observer-guided hierarchical coding;

[0038] In the hierarchical encoding process, the observer is repeatedly updated by using multimodal information, and then the observer is used as a guide to promote cross-modal information interaction. This process is formalized as a series of equations, as follows:

[0039] Start with:

[0040] w 1 =f(w 0 )

[0041] w 1 =[w v1 , w t1 ]

[0042] Among them, f represents the self-attention mechanism, using the same input as query, key and value;

[0043] Then:

[0044] h t1 =h(z t , w v1 , z v )

[0045] h v1 =h(z v , w t1 , z t )

[0046] Among them, h t1 and h v1 is the text and image encoding obtained by the first observer attention propagation; h() represents the observer attention mechanism, z v , wt1 , z t As query, key, and value in the attention mechanism respectively;

[0047] Then:

[0048] w 2 =f([h t1 ;h v1 ])

[0049] h t2 =h(h t1 , w v2 ,h v1 )

[0050] h v2 =h(h v1 , w t2 ,h t1 )

[0051] o 1 =[h t2 ;h v2 ]

[0052] Among them, w 2 The observer code for the second observer mechanism, h t2 and h v2 The text and image encoding obtained for the second observer attention propagation; o 1 is the output of the first round of interencoder;

[0053] Based on the first round of observer mechanism, a variational autoencoder is used to encode the unimodal latent variable output, namely:

[0054] z′ t =VAE(x t ,h t2 )

[0055] z′ v =VAE(x v ,h v2 )

[0056] z 1 =[z′ t ; z′ v ]

[0057] Among them, z′ t , z′ v It is the text and picture modality encoding after the second variational autoencoder sampling, and constitutes the final output z of the second variational autoencoder 1 .

[0058] Then, the second round of observer mechanism is carried out:

[0059] w 3 =g(z 1 , o 1 , o 1 )

[0060] h t3 =h(z′ t , w v3 , z′ v )

[0061] h v3 =h(z′ v , w t3 , z′ t )

[0062] w 4 =f([h t3 ;h v3 ])

[0063] h t4 =h(h t3 , W v4 ,h v3 )

[0064] h v4 =h(h v3 , W t4 ,h t3 )

[0065] o 2 =[h t4 ;h v4 ]

[0066] Among them, w 3 Encode the observer initialized in the second round of observer mechanism. The update process of other variables is the same as that of the first round of observer mechanism.

[0067] Furthermore, the relation extraction encoder is a multi-layer perceptron MLP, and the output of the relation extraction encoder is the final prediction result;

[0068] The conditional random field is used as an auxiliary task in the training process, and supervised training is performed using the original entity position annotations in the dataset, which is specifically expressed as follows:

[0069] Y crf =crf(mlp(h t4 ))

[0070] Among them, mlp represents multi-layer perceptron MLP, and crf represents conditional random field.

[0071] When training the model, the total loss function L total as follows:

[0072] L total =θL VAE +αL align +βL watcher +γL RE +δL InfoNCE +∈L CRF +ζL recon

[0073] Among them, θ, α, β, γ, δ, ∈, ζ are the loss function weights of each optimization objective, L VAE represents the unimodal autoencoder loss, which measures the difference between the original variable and the reconstructed variable of the variational autoencoder; L align represents the alignment loss, which enhances the correlation between modalities by maximizing the similarity scores of matching pairs and minimizing the scores of non-matching pairs; L watch represents the observer loss, which maintains a high similarity between the final entity embedding and the original observer embedding when the observer attention is propagated; L RE represents the relation extraction loss, which supervises the comparison between the model prediction and the true label to ensure accurate relation prediction; L InfoNCE represents InfoNCE loss, which enhances the consistency between entity embedding and label embedding; L CRF represents CRF loss, which optimizes entity boundary detection to improve the accuracy of entity recognition; L recon Represents the reconstruction loss, which ensures that the model can accurately reconstruct the masked modal information.

[0074] The beneficial effects of the present invention are:

[0075] 1) By combining text and visual information, the present invention effectively supplements the missing context clues in short texts, especially in social network environments, and this method significantly improves the accuracy of relationship extraction. Specifically, by using visual information to enhance the semantic connection between entities, the model can more accurately identify the relationship between entities, thereby improving the overall multimodal relationship extraction performance.

[0076] 2) This paper proposes a joint learning framework that integrates multiple loss functions, including alignment loss, relationship extraction loss, etc. These loss functions work together at different levels of the model to enhance the model's ability to process complex multimodal data. This comprehensive learning strategy enables the model to better understand and integrate information from different modalities, thereby demonstrating outstanding performance in multimodal relationship extraction tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 This is an overall flow chart of the relationship extraction method in the present invention.

[0078] Figure 2A schematic diagram of the construction and coordination of the training objectives in the present invention. DETAILED DESCRIPTION

[0079] The present invention aims to propose a deep learning model that can effectively improve the performance of multimodal relation extraction. This model is particularly targeted at relation extraction tasks in short texts and noisy environments. It aims to improve the accuracy and recall of relation extraction by integrating text and visual information. The overall implementation process is as follows: Figure 1 As shown:

[0080] S1. For text and image samples that need to be extracted, the pre-trained encoder trained on a large number of data samples is used to encode the original information into the tensor representation required for deep learning.

[0081] S2. Use variational autoencoder (VAE) to perform coarse encoding of text and pictures in their respective modes to make the sample distribution more reasonable;

[0082] S3, the text and image representations are fused through the cross-attention mechanism with an intermediate medium, so that the fused representation has richer semantic information;

[0083] S4, using the fully connected layer for relationship extraction prediction and the conditional random field (CRF) for entity boundary prediction;

[0084] S5. When training the model, use the different representations and prediction results obtained in the previous steps to set different joint learning subtasks and loss functions for joint training, optimize the model parameters through the gradient descent method to improve the model performance, and when performing specific prediction tasks, use the trained model parameters to directly perform predictions, and directly obtain the prediction results based on the scores.

[0085] Example:

[0086] Below is an entity pair with m candidate relation types (head entity e h , tail entity e t ) and the text containing the entity pair are taken as examples to elaborate on the scheme of the present invention in detail. The specific implementation process is as follows:

[0087] Step 1: First, preprocess the sample data. The data preprocessing operation specifically includes two sub-steps: pre-training encoding and VAE encoding:

[0088] Step 11: For the text and image samples that need to be extracted, encode them through the pre-trained encoder, extract the feature representation of the text and image, and map the features of the text and image to the shared representation space:

[0089] In the model, the pre-trained BERT (Bidirectional Transformer Encoder based on Transformer) is used as the text encoder to obtain the contextual representation of the sentence. For image I, the image encoder ResNet (Residual Network Image Encoder) is used to extract image features.

[0090] For a given sentence T, input it into the BERT model to obtain its hidden state representation x t These hidden representations are passed through multiple layers of encoders. Each layer of encoders usually contains two sublayers: a multi-head self-attention mechanism and a feed-forward neural network. Each layer is followed by residual connections and layer normalization. Finally, we can choose to use the hidden states of different layers, such as the last layer or a combination of multiple layers, to obtain richer text representations. Formally, this process can be expressed as:

[0091] x t =BERT(T)

[0092] The image encoder processes the image I and converts it into a representation x that is compatible with text features. v . This can be expressed as:

[0093] x v =ImageEncoder(I)

[0094] To demonstrate the effectiveness of the model, common pre-trained encoders such as BERT and ResNet are used. After this module, the original task input is converted into embeddings for subsequent stages. This conversion process can be symbolically represented as:

[0095] (T, I)→(x t , x v )

[0096] Step 12: Use the autoencoder to refine and fine-tune the pre-trained encoder:

[0097] The autoencoder component adopts a variational autoencoder (VAE), which is a type of autoencoder that adds a probabilistic twist to the autoencoding process, and its main goal is to extract and encode the internal information of unimodal features based on the pre-trained encoder.

[0098] For text encoding:

[0099] μ t =f(x t )

[0100] σ t =f(x t )

[0101] Among them, f is the self-attention module. tis the mean of the text encoding, σ t is the standard deviation of the text encoding.

[0102] Similarly, for image encoding:

[0103] μ v =f(x v )

[0104] σ v =f(x v )

[0105] Among them, μ v is the mean of the image encoding, σ v is the standard deviation of the image encoding.

[0106] In VAE, a reparameterization trick is used to sample from the latent space of text and images to ensure that the latent variables follow a normal distribution. The sampled latent variables are:

[0107]

[0108] These variables will be used for processing in the subsequent stages, and μ and σ in the formula represent the mean and standard deviation.

[0109] Step 2: Construct the intermediate media and update and integrate the images, texts and intermediate media:

[0110] The cross-encoder is designed to merge the feature representations of text and images, and it adopts a unique structure called "Observer" as an intermediary for cross-modal attention. The cross-encoder uses a stacked structure to update information through two layers of alternating propagation.

[0111] Step 21. In order to retain the most original text and image information after pre-training, first define the initial observer:

[0112] w=[x t ;x v ]

[0113] In the first propagation process, the latent variables sampled from the initial observer are used to extract the main modal information as the first update of the observer, which can be expressed as:

[0114] W 0 =g(z 0 , w)

[0115] Where z0=[z t ;z v ]

[0116] g represents the cross-attention mechanism, using z 0 As a query, W 0As keys and values, to supplement the original modality information after sampling.

[0117] Step 22: Observer-guided hierarchical coding:

[0118] In the hierarchical encoding process, the observer is repeatedly updated by using multimodal information, and then the observer is used as a guide to promote cross-modal information interaction, forming the basis of cross-modal cross-coding. This process can be expressed as a series of equations, starting with the following:

[0119] w 1 =f(W 0 )

[0120] w 1 =[w v1 , w t1 ]

[0121] Different from the previous g, f represents a self-attention mechanism that uses the same input as query, key, and value to complement the feature depth of the representation information and the learnability of the encoding process. v1 , w t1 Indicates w 1 The corresponding images and texts are embedded in .

[0122] Then:

[0123] h t1 =h(z t , w v1 , z v )

[0124] h v1 =h(z v , w t1 , z t )

[0125] Among them, h t1 and h v1 is the text and image encoding obtained by the first observer attention propagation, h() represents the observer attention mechanism, z v , w t1， z t They serve as query, key, and value in the attention mechanism respectively;.

[0126] Then:

[0127] w 2 =f([h t1 ;h v1 ])

[0128] h t2 =h(h t1 , w v2 ,h v1 )

[0129] h v2 =h(h v1 , w t2 ,h t1 )

[0130] o 1 =[h t2 ;h v2 ]

[0131] Among them, w 2 The observer code for the second observer mechanism, h t2 and h v2 The text and image encodings obtained by the second observer attention propagation. In each update process, the observer encoding is combined with the encodings of different modalities for fusion, and then feedback is used to guide the interaction between different modalities. Finally, o 1 is the output of the first round of interencoder.

[0132] Based on the first round of observer mechanism, the VAE in step 12 is used again to encode the unimodal latent variable output, namely:

[0133] z′ t =VAE(x t ,h t2 )

[0134] z′ v =VAE(x v ,h v2 )

[0135] z 1 =[z′ t ; z′ v ]

[0136] Among them, z′ t , z′ v It is the text and picture modality encoding after the second VAE sampling, and constitutes the final output z of the second VAE 1 .

[0137] Then, the second round of observer mechanism is carried out:

[0138] w 3 =g(z 1 , o 1 , o 1 )

[0139] h t3 =h(z′ txt , w v3 , z′ img )

[0140] hv3 =h(z′ img , w t3 , z′ txt )

[0141] w 4 =f([h t3 ;h v3 ])

[0142] h t4 =h(h t3 , w v4 ,h v3 )

[0143] h v4 =h(h v3 , W t4 ,h t3 )

[0144] o 2 =[h t4 ;h v4 ]

[0145] Among them, w 3 Encode the observer initialized in the second round of observer mechanism. The update process of other variables is the same as that of the first round of observer mechanism.

[0146] Step 3: After obtaining the final encoding result of the previous step, the decoder will be used to complete the final prediction. The decoder part includes two multi-layer perceptrons (MLPs), one for the relation extraction (RE) task and the other for the conditional random field (CRF) task. Among them, the output of the relation extraction encoder is the final prediction result, and the output of the conditional random field is used as an auxiliary task in the training process. It is supervised by using the original entity position annotations in the dataset to improve the model's semantic segmentation ability for entity boundaries.

[0147] Step 31, Relation Extraction (RE) Decoder:

[0148] The RE decoder aims to extract relations from the combined textual and visual feature embeddings. This is done via an MLP that learns a complex mapping between input features and target relations. Let e h and e t Respectively represent the final expression h t4 The embeddings corresponding to the head entity and the tail entity have been interacted with the original embeddings many times, ensuring that the relative position information is not lost while the cross-modal information is effectively propagated. Formally, the RE prediction process can be expressed as:

[0149] Y re = mlp(e h , e t )

[0150] Step 32, Conditional Random Field (CRF) decoder:

[0151] The CRF decoder targets sequence tagging tasks and aims to optimize the detection of entity boundaries based on the entity location annotations provided by the RE dataset. This is achieved through another MLP, whose output is used as the input of the CRF layer. The CRF layer takes into account the context in the sequence to more accurately draw boundaries around the entity, which is crucial for the relation extraction task. The CRF prediction process can be expressed as:

[0152] Y crf =crf(mlp(h t4 ))

[0153] By supervising both RE performance and CRF performance during training, the annotation information in the dataset can be more effectively utilized, thereby improving its performance in the multimodal relation extraction task.

[0154] Step 4: When solving the challenges of modality differences and alignment in multimodal relation extraction, Figure 2 As shown in the figure, a joint learning training framework with multiple optimization objectives is constructed. By using the representation and prediction results generated in the previous steps, new auxiliary tasks and loss functions are constructed to help model training improve performance. Specifically, the following training objectives are built:

[0155] Step 41. For unimodal autoencoders, a variational autoencoder approach is used. Two specific goals are pursued: minimizing the Kullback-Leibler divergence loss (KDL) and the L2 distance loss, which are combined into the VAE loss. These goals ensure that the encoded samples conform to the Gaussian distribution while limiting the variation between the original and encoded samples.

[0156] The L2 distance loss measures the difference between the original variable and the reconstructed variable, and the formula is as follows:

[0157]

[0158] Among them, z i Represents the VAE encoded vector of a certain modality, x i represents the VAE encoding vector of a certain modality, and N is the number of samples in the batch. The norm ||·|| represents the Euclidean distance between vectors, providing a measure of dissimilarity between the original data points and the reconstructed data points.

[0159] The KDL loss aims to regularize the distribution of latent variables to a standard normal distribution, as follows:

[0160]

[0161] in, is the expected value symbol, which means taking the expectation of the random variable z under the posterior distribution q(z|x). Here, q(z|x) represents the distribution of the latent variable z given the data x. μ and σ 2 are the mean and standard deviation of the encoding distribution, respectively.

[0162] Step 42: In order to enhance the correlation between modality embeddings after unimodal encoding, adversarial training is introduced to optimize alignment. Specifically, the goal is to improve the similarity score for matching image-text pairs and reduce the score for mismatched pairs.

[0163] Assume ij represents the similarity score between the i-th image embedding and the j-th text embedding. The optimization goal is to maximize s when i=j (positive pair) ij , minimize s when i≠j (negative pair) ij .

[0164] These scores are calculated using the scoring function S(·,·) learned through the fully connected layers. The loss function for adversarial training is defined as follows:

[0165] The positive loss is:

[0166] L pos = -log(p pos )

[0167] The negative pair loss is:

[0168] L neg = -log(1-p neg )

[0169] Among them, the probability p pos and p neg Defined as:

[0170] p pos =σ(S(z t , z v ))

[0171] p neg =σ(S(z t , z′ v ))

[0172] Among them, z′ v For not with z t Corresponding other image samples. The total alignment loss is the sum of the positive and negative pair losses:

[0173] L align =L pos +L neg

[0174] Step 43. In order to effectively integrate visual information into text representation without losing the inherent structure of the text, an observer mechanism is introduced to supplement the traditional cross-attention mechanism. The goal is to maintain a high similarity between the final entity embedding and the original observer embedding to preserve the relative position information.

[0175] Let p watcher denotes the scoring function applied to the initial observer embedding w 0 And the final output o 2 The probability score generated by the connection of:

[0176] p watcher =σ(S(w 0 , o 2 ))

[0177] Here, S(·,·) is the scoring function learned by the fully connected layer and σ(·) is the sigmoid activation function.

[0178] Given the probability score, the observer loss is defined as p watcher Binary cross entropy loss with the target label (y=1) (indicating high similarity):

[0179] L watcher =-[ylog(p watcher )+(1-y)log(1-p watcher )]

[0180] In practice, since the target label is always set to 1, the observer loss simplifies to:

[0181] L watch = -log(p watch )

[0182] This formulation encourages the model to produce high probability scores when comparing the final entity embeddings to the original observer embeddings, thereby preserving relative position information during the fusion process.

[0183] Step 44: For relation extraction, use the traditional cross entropy loss to supervise the comparison between the model's prediction and the true label:

[0184]

[0185] Among them, y i is the true label, p(y i ) is the predicted probability.

[0186] In addition, to encourage learning semantically consistent embeddings for entities with the same label, a learnable label matrix is ​​utilized. Inspired by InfoNCE, a cosine-based scoring function is used to calculate the similarity score between entity embeddings and label embeddings:

[0187]

[0188] Where fcos(·,·) represents the cosine similarity function, l true is corresponding to [e h :e t ]’s true label embedding, l is the embedding of [e h :e t ], including both positive and negative examples. By maximizing the score of the positive example relative to all other examples, the InfoNCE loss function encourages the model to learn high similarity between positive examples while reducing the similarity with other negative examples.

[0189] The InfoNCE loss aims to maximize the similarity between entity embeddings and their true label embeddings, relative to other label embeddings. This encourages the model to learn representations such that entities with similar labels are closer in the embedding space.

[0190] Step 45. To enhance the model’s sensitivity to entity locations, train it to predict entity locations using the Conditional Random Field (CRF) loss. The CRF loss is as follows:

[0191]

[0192] Among them, s true is the score of the true entity annotation sequence annotated in the dataset, ∑ s exp(s) is performed on all possible entity sequences. CRF is a commonly used sequence labeling model, which is used in natural language processing for named entity recognition, part-of-speech tagging and other tasks. This loss function is used to measure the difference between the model's predicted sequence and the actual sequence. Through this loss function, the goal of the CRF model is to minimize L CRF , that is, maximize the score of the true sequence relative to all other sequences. This means that the model will adjust its parameters so that the score of the correct label sequence is as high as possible compared to the scores of all other sequences, thereby improving the performance of the model on the sequence labeling task.

[0193] NER (Named Entity Recognition) tags are derived from the relation extraction dataset by identifying the start and end positions of entities in sentences. These tags are encoded as IDs to be compatible with the input format of the model, facilitating the CRF layer to learn from structured entity information.

[0194] Step 46: To promote robust cross-modal representation learning, a reconstruction loss mechanism is used. This loss encourages the model to accurately reconstruct the masked parts of the text and visual inputs based on the unmasked context. The loss formula is as follows:

[0195] Lrecon =-S(x unmasked , x reconstructed )

[0196] Among them, S(·,·) is a scoring function used to measure the embedding x before masking. unmasked and its refactored version x reconstructed Among them, we use z i Represents the output vector of the first VAE encoding as x unmasked ,h i4 Denote the output reconstruction vector of the second round of observer mechanism as x reconstructed .

[0197] In practice, this involves masking specific tokens from textual input and specific channels from visual input. The scoring function module evaluates the quality of reconstruction by comparing the reconstruction result with the original unmasked data. Binary cross entropy loss is used to penalize the difference between the predicted score and the true label (positive examples are set to 1), aiming to maximize the model's ability to accurately reconstruct the missing information.

[0198] Step 5. Integrating the objectives described in this article, the total loss function of the model prioritizes the variational autoencoder (VAE) component, followed by the losses of multimodal feature fusion, relation extraction, and entity recognition:

[0199] L total =θL VAE +αL align +βL watcher +γL RE +δL InfoNCE +∈L CRF +ζL recon

[0200] Among them, θ, α, β, γ, δ, ∈, ζ are the loss function weights of each optimization objective. VAE It includes the VAE loss described in detail in the previous section as the basic loss component to optimize the distribution of latent variables. Subsequent terms are task- and modality-specific, and each term is weighted by a corresponding hyperparameter. Balancing these losses through hyperparameter tuning enables the model to perform well in multiple tasks simultaneously, from latent variable modeling to complex multimodal understanding and relational reasoning.

[0201] When performing a prediction task, steps 4 and 5 are omitted, and the output of the relation extraction decoder in step 31 is directly used as the relation probability, and the one with the highest score is taken as the relation extraction prediction result.

[0202] Although the present invention is described herein with reference to the embodiments of the present invention, the above embodiments are only preferred embodiments of the present invention, and the embodiments of the present invention are not limited to the above embodiments. It should be understood that those skilled in the art can design many other modifications and implementations, which will fall within the scope and spirit of the principles disclosed in this application.

Claims

1. A multimodal relation extraction method based on a medium attention mechanism, characterized in that: The following steps are involved: First, we build a multimodal relationship extraction model and use it to process text and images: A1. For text and image samples that need to be extracted, encode them through a pre-trained encoder to extract feature representations of text and images. A2. Use autoencoders to perform coarse-grained autoencoding on the feature representations of pre-trained encoded text and images. A3. Construct a cross-encoder based on the cross-attention mechanism of the intermediate medium to fuse text and image representations; A4. Decode the relation extraction task through the relation extraction encoder and conditional random field, where the relation extraction encoder is used to predict the relationship category and the conditional random field is used to optimize the entity boundary detection; Then use the multimodal relation extraction model to perform relation extraction training and specific relation extraction tasks; B1. When training the model, use the different representations and prediction results obtained in the previous steps, set different joint learning subtasks and loss functions for joint training, and optimize the model parameters through the gradient descent method to improve the model performance; B2. When performing multimodal relationship extraction tasks, directly use the trained model to predict new multimodal data, use the output of the relationship extraction decoder as the relationship probability, and take the one with the highest score as the relationship extraction prediction result.

2. According to claim 1, a multimodal relationship extraction method based on a medium attention mechanism is characterized in that: The pre-trained encoder includes a Transformer-based bidirectional transformer encoder BERT and an image encoder; for a given sentence T, it is input into the BERT model to obtain its hidden state representation x t ; For a given image I, convert it to a representation x that is compatible with text features v .

3. According to claim 2, a multimodal relationship extraction method based on a medium attention mechanism is characterized in that: The autoencoder is specifically a variational autoencoder, and the autoencoding process is specifically as follows: For text encoding: m t =f(x t ) s t =f(x t ) Among them, f is the self-attention module, μ t is the mean of the text encoding, σ t is the standard deviation of the text encoding; For image encoding: m v =f(x v ) s v =f(x v ) Among them, μ v is the mean of the image encoding, σ v is the standard deviation of the image coding; In the variational autoencoder, a reparameterization trick is used to sample from the latent space of text and images to ensure that the latent variables follow a normal distribution. The sampled latent variables are: where z t represents the latent variable for text encoding, z v Latent variables representing image encoding.

4. The multimodal relationship extraction method based on the medium attention mechanism according to claim 3 is characterized in that: The cross-encoder based on the intermediate cross-attention mechanism adopts the unique structure of the observer as the intermediary of cross-modal attention and updates information through two-layer alternating propagation; the specific process of step A3 is as follows: Step S3.1: First define the initial observer w: w=[x t ;x v ] In the first propagation process, the latent variables sampled from the initial observer are used to extract the modal information as the first update of the observer, which is expressed as: w0=g(z0,w) Where z0=[z t ;z v ]; g represents the cross-attention mechanism, using z0 as the query in the cross-attention mechanism and w0 as the key and value; Step S3.2: Observer-guided hierarchical coding; In the hierarchical encoding process, the observer is repeatedly updated by using multimodal information, and then the observer is used as a guide to promote cross-modal information interaction. This process is formalized as a series of equations, as follows: Start with: w1=f(w0) w1=[w v1 ,In t1 ] Among them, f represents the self-attention mechanism, using the same input as query, key and value; Then: h t1 =h(z t ,w v1 ,z v ) h v1 =h(z v ,w t1 ,z t ) Among them, h t1 and h v1 is the text and image encoding obtained by the first observer attention propagation; h() represents the observer attention mechanism, z v ,w t1 ,z t As query, key, and value in the attention mechanism respectively; Then: w2=f([h t1 ;h v1 ]) h t2 =h(h t1 ,w v2 ,h v1 ) h b2 =h(h v1 ,w t2 ,h t1 ) o1=[h t2 ;h v2 ] Among them, w2 is the observer code of the second observer mechanism, h t2 and h v2 is the text and image encoding obtained by the second observer attention propagation; o1 is the output of the first round of mutual encoder; Based on the first round of observer mechanism, a variational autoencoder is used to encode the unimodal latent variable output, namely: z′ t =VAE(x t ,h t2 ) z′ v =VAE(x v ,h v2 ) z1=[z′ t ;With' v ] Among them, z′ t , z′ v It is the text and picture modality encoding after sampling by the second variational autoencoder, and constitutes the final output z1 of the second variational autoencoder. Then, the second round of observer mechanism is carried out: w3=g(z1,o1,o1) h t3 =h(z′ t ,w v3 ,z′ v ) h v3 =h(z′ v ,w t3 ,z′ t ) w4=f([h t3 ;h v3 ]) h t4 =h(h t3 ,w v4 ,h v3 ) h v4 =h(h v3 ,w t4 ,h t3 ) o2=[h t4 ;h v4 ] Among them, w3 is the observer code initialized in the second round of observer mechanism, and the update process of other variables is the same as that of the first round of observer mechanism.

5. The multimodal relationship extraction method based on the medium attention mechanism according to claim 4 is characterized in that: The relation extraction encoder is a multi-layer perceptron MLP, and the output of the relation extraction encoder is the final prediction result; The conditional random field is used as an auxiliary task in the training process, and supervised training is performed using the original entity position annotations in the dataset, which is specifically expressed as follows: Y crf =crf(mlp(h t4 )) Among them, mlp represents multi-layer perceptron MLP, and crf represents conditional random field.

6. The multimodal relationship extraction method based on the medium attention mechanism according to claim 5, characterized in that: When training the model, the total loss function L total as follows: L total =θL VAE +αL align +βL watcher +γL RE +δL InfoNCE +∈L CRF +ζL recon Among them, θ, α, β, γ, δ,∈, ζ are the loss function weights of each optimization objective, L VAE represents the unimodal autoencoder loss, which measures the difference between the original variable and the reconstructed variable of the variational autoencoder; L align represents the alignment loss, which enhances the correlation between modalities by maximizing the similarity scores of matching pairs and minimizing the scores of non-matching pairs; L watch represents the observer loss, which maintains a high similarity between the final entity embedding and the original observer embedding when the observer attention is propagated; L RE represents the relation extraction loss, which supervises the comparison between the model prediction and the true label to ensure accurate relation prediction; L lnfoNCE represents InfoNCE loss, which enhances the consistency between entity embedding and label embedding; L CRF represents CRF loss, which optimizes entity boundary detection to improve the accuracy of entity recognition; L recon Represents the reconstruction loss, which ensures that the model can accurately reconstruct the masked modal information.

7. The multimodal relationship extraction method based on the medium attention mechanism according to claim 6, characterized in that: The loss function is as follows: VAE loss: For the input and output of the variational autoencoder, divergence loss and L2 distance loss are used. These two losses together constitute the VAE loss L VAE ; Specifically, the L2 distance loss L L2 Measures the difference between the original variable and the reconstructed variable, as follows: Among them, z i Represents the unimodal variational autoencoder encoded vector, x i represents the unimodal variational autoencoder encoding vector, N is the number of samples in the batch, and the norm ∥·∥ represents the Euclidean distance between vectors; Divergence loss L KDL Regularize the distribution of latent variables to a standard normal distribution, as follows: in, is the expected value symbol, which means taking the expectation of the random variable z under the posterior distribution q(z|x); here, q(z|x) represents the distribution of the latent variable z given the data x; μ and σ 2 are the mean and standard deviation of the encoding distribution, respectively; Alignment loss: adversarial training is introduced to optimize alignment; specifically, let s ij represents the similarity score between the i-th image embedding and the j-th text embedding. The optimization goal is to maximize s when i=j. ij , minimize s when negative pair i≠j ij ; These scores are calculated using the scoring function S(·,·) learned through the fully connected layer. The loss function for adversarial training is defined as follows The positive loss is: L pos =-log(p pos ) The negative pair loss is: L neg =-log(1-p neg ) Among them, the probability of being right is p pos and negative pair probability p neg Defined as: p pos =σ(S(z t ,With v )) p neg =σ(S(z t ,With' v )) Where σ(·) is the sigmoid activation function, z′ v For not with z t For other corresponding image samples, the total alignment loss is the sum of the positive and negative pair losses: L align =L pos +L neg Observer loss: Observer loss based on the intermediate cross-attention mechanism; let p watcher represents the probability score generated by the concatenation of the scoring function applied to the initial observer embedding w0 and the final output o2: p watcher =σ(S(w0,o2)) Observer loss L watcher Specifically: L watcher =-log(p watche ) Relation extraction loss: For relation extraction, cross entropy loss is used to supervise the comparison between the model's prediction and the true label: Among them, y i is the true label, p(y i ) is the predicted probability; InfoNCE loss: Calculates the similarity score between entity embeddings and label embeddings using a cosine-based scoring function: Where fcos(·,·) represents the cosine similarity function, l true is the corresponding entity pair [e h :e t ]’s true label embedding, l is the embedding of [e h :e t ]All labels or samples compared, including positive and negative examples; Conditional Random Field Loss: Use the conditional random field loss L CRF To predict the entity position, as follows: Among them, s true is the score of the true entity annotation sequence annotated in the dataset, ∑ s exp(s) is performed on all possible entity sequences; Reconstruction loss: A reconstruction loss mechanism is adopted, which encourages the model to accurately reconstruct the masked parts of the text and visual input based on the unmasked context. The reconstruction loss L recon The formula is as follows: L recon =-S(x unmasked ,x reconstructed ) Among them, the output vector encoded by the first variational autoencoder is x unmasked , the output reconstruction vector of the second round of observer mechanism is taken as x reconstructed .

Images