An Adaptive Multimodal Relation Extraction Method Based on Mutual Attention Mechanism

Abstract

This invention discloses an adaptive multimodal relation extraction method based on a mutual attention mechanism, relating to the field of multimodal relation extraction. First, a pyramid module extracts hierarchical feature information from the image, and then uses this information to generate an adaptive feature vector. The extracted text information and the top-level visual information are denoised and aligned. The processed two modal information are then fused using a multimodal mutual attention fusion module. A loss function is calculated, and the model is trained by minimizing the loss using stochastic gradient descent. When performing the multimodal relation extraction task, after obtaining the text-image pair, a classifier obtains the original score for each sample, which is then normalized to a probability distribution to obtain the probability. Finally, the multimodal relation extraction task is completed based on the probability ranking results. This invention avoids the negative impact of auxiliary images with low contribution value on model prediction and avoids the modality loss problem that may occur with ordinary encoders.

Description

Technical Field

[0001] This invention relates to the field of relation extraction technology, and specifically to an adaptive multimodal relation extraction method based on a mutual attention mechanism. Background Technology

[0002] Relation extraction (RE) aims to identify the semantic relationships between two given entities in textual information, playing a crucial role in tasks such as question answering and knowledge base retrieval. Most existing text-based RE methods can be broadly categorized into sequence-based models and dependency-based models. These models can only extract information from text, and their predictive performance significantly degrades when the text lacks sufficient context. Therefore, leveraging visual information to assist in entity pair extraction from text has become a major research focus, namely multimodal relation extraction (MRE). MRE uses image information to help models identify entity pairs in text, thereby more effectively improving prediction accuracy.

[0003] For visual information extraction, existing methods introduce adaptive co-attention networks, which extend the combination of bidirectional long short-term memory networks and conditional random fields, using global image information as a cue and incorporating an adaptive co-attention mechanism. Other methods propose novel neural network models that utilize visual objects in the image as fine-grained image representations, preserving more local information. However, these methods focus solely on visual information extraction and do not consider how to fuse information from two different modalities.

[0004] The existing approaches to solving this problem can be broadly divided into two categories:

[0005] 1. Extract visual information as visual prefixes to aid in text information extraction;

[0006] This method extracts visual information from images and uses the extracted visual information as a prefix for text entity extraction.

[0007] 2. Extract information from the two modalities separately, and perform modality fusion using an attention encoder;

[0008] This approach can utilize visual information to the greatest extent to complete text information, but it requires an effective method for extracting visual information.

[0009] The MRE dataset contains the global image and auxiliary images obtained by cropping the overall image. However, for each text-image pair, not all auxiliary images contained in the text necessarily improve prediction performance. Existing methods ignore this problem, and the attention-based encoding model used in the fusion stage also leads to the loss of modality information. Summary of the Invention

[0010] The purpose of this invention is to propose an adaptive multimodal relation extraction method based on a mutual attention mechanism, so as to better complete the task of multimodal relation extraction when faced with complex text-image pairs. This addresses the technical problems of existing technologies failing to effectively fuse text and visual information, and the loss of modal information caused by the attention-based encoding model used in the fusion stage.

[0011] The technical solution adopted by the present invention to solve the above-mentioned technical problems is as follows:

[0012] An adaptive multimodal relation extraction method based on mutual attention mechanism includes the following steps:

[0013] A. Training a multimodal relation extraction model:

[0014] A1. Extract layered visual features from the image to be processed using the pyramid module, and use each layer of visual features as text-encoded prompt information; at the same time, process the top layer of visual features separately to form another type of encoded information, and fuse it with the text information in the fusion stage.

[0015] A2. The feature vector dynamic allocation module processes the visual features of each layer to obtain feature vectors. Then, the obtained feature vectors are used by the key-value pair generation module to extract cue vectors containing key-value pairs.

[0016] A3. Denoise the top-level visual feature information and text encoding information, and align the encoding information of the two modalities of text and image by means of positive and negative samples;

[0017] A4. The aligned two modal information are fused through a multimodal fusion module to obtain the coding information used for prediction;

[0018] A5. Use the prediction module to predict entity relationships in the training set;

[0019] A6. Calculate the loss function and sum it proportionally to obtain the final loss value of this model. Train the multimodal relation extraction model by stochastic gradient descent.

[0020] B. Multimodal Relation Extraction Task:

[0021] The trained relation extraction model was used in a real-world multimodal relation extraction task to generate the most likely relation type for entity pairs in the text based on visual cues.

[0022] Further, in step A1, the extraction of hierarchical visual features using the pyramid module specifically includes:

[0023] The global image and its corresponding auxiliary image in the dataset are cropped to obtain the cropped global image and the auxiliary image I. Aux ={I A1 , ..., I Aj , ..., I An}; where n represents the number of auxiliary images, j∈[1,n];

[0024] Based on ResNet, the pyramid module is used to analyze the global image and the auxiliary image I. Aux Encode the image to generate the corresponding global image encoding sequence P. I and auxiliary image coding sequence P Aux ={P A1 , ..., P Aj , ..., P An};

[0025] Further, in step A2, the processing of each layer of visual features through the feature vector dynamic allocation module specifically includes:

[0026] The global image encoding sequence P obtained in A1 I and auxiliary image coding sequence P Aux The process involves segmentation, stacking, and reshaping to obtain the corresponding global cue vector F. I and auxiliary prompt vector list F Aux ={F A1 F Aj F An};

[0027] Calculate the weight score using the multi-head attention mechanism weight formula:

[0028]

[0029] Among them, the global cue vector F is used. I The global context weight W is calculated using the corresponding query Q and key K through a weight formula. I Using the global cue vector F I The corresponding query Q and the j-th auxiliary hint vector F Aj The corresponding key K is used to calculate the j-th auxiliary context weight W. A Softmax is the normalized activation function, dk This represents the parameter factor.

[0030] Calculate the weighted score using the following contextual formula:

[0031] T = W × V

[0032] Among them, global context weight W is used. I and global cue vector F I The corresponding value V is used to calculate the global context F using the context formula. I Using the j-th auxiliary context weight W A and the j-th auxiliary cue vector F Aj The corresponding value V is used to calculate the j-th auxiliary context T using the context formula. A ;

[0033] Add the global context to the j-th auxiliary context to obtain adaptively generated context information:

[0034] FC = F I +αT A

[0035] Where FC represents the j-th context information obtained, and α represents the weight parameter;

[0036] The module for each group F I and F Aj By performing the above operations h times, h context information FCs are obtained. All context information FCs are linearly integrated, where h represents the number of attention heads:

[0037] FCs=Concat(FC1, FC2,..., FC h W o

[0038] Where Concat is the concatenation function, W o It outputs the projection matrix; then it performs two normalization steps to ensure that the data distribution of each sample is roughly the same across different feature dimensions, and also uses the ReLU function to introduce non-linearity to obtain the processed information.

[0039] The feature vector dynamic allocation module processes the same pair of image encoding sequences four times to obtain the final list of contexts, which is the FCL. j ={FCs1, FCs2, FCs3, FCs4};

[0040] The global hint vector F I Each auxiliary cue vector is processed separately to obtain a context list, and all the obtained context lists are integrated to obtain the feature vector:

[0041] F = Stack(FCL1, ..., FCL) j FCL n )

[0042] Here, Stack is the stacking function; the final feature vector F is the final feature vector obtained by integrating the encoded information of each image pair.

[0043] Further, in step A2, the step of extracting a hint vector containing key-value pairs from the obtained feature vector through the key-value pair generation module specifically includes:

[0044] By calculating a gating value, the model can dynamically decide whether to activate specific prompts based on the characteristics of the input data, thereby controlling the degree of influence of these prompts on the final output. The specific formula for the gating value PG is as follows:

[0045]

[0046] Among them, f img This represents each encoding layer in the feature vector F, corresponding to n context lists FCL1, FCL2, ..., FCL n ;∑ seq f img Represents along f img Summing the third dimension, Gate idx Represents the idx-th threshold function; each encoding layer f in the feature vector F img The contributions to the final cue vector are all different. The weights of the contributions from each coding layer can be controlled by a gating unit. This involves connecting the i-th row of PG with f. img Multiply each element in the i-th column one by one, and then sum them up.

[0047] f val =Einsum(′b,bh->bh′,PG[:,i],f img [i])

[0048] Where Einsum represents the expression for PG[:,i] and f img [i] Perform element-wise multiplication and summation, and calculate f four times to obtain f. val1 f val2 f val3 f val4 Add them together to get the contribution weight:

[0049] F val =f val1 +f val2 +f val3 +f val4

[0050] For each coding layer f in the feature vector F img Perform the above operations on all of them, then add all the resulting contribution weights to the same list, and perform splitting, stacking, and reshaping to obtain a hint vector containing key-value pairs.

[0051] Further, in step A3, the noise reduction processing of the top-level visual feature information and text encoding information specifically includes:

[0052] Using the key-value pair cue vectors obtained in A2 as an aid, BERT is used to encode the text. Two special markers, [CLS] and [SEP], are inserted at the beginning and end of each extracted sentence, respectively. The processed sentence information is used as the text information seq.

[0053] The global image encoding sequence P obtained in A1 I and auxiliary image coding sequence P Aux The top layer of visual information is spliced ​​together and then processed through segmentation, stacking, and reshaping to obtain the visual information vis.

[0054] To allow the gradient to pass through, a reparameterization technique is used to transform random sampling into a deterministic function with added random noise. This allows the gradient to flow during the sampling operation while maintaining the ability to backpropagate the gradient during sampling. The latent variable calculation formula is as follows:

[0055] z = μ + e( l ogσ)⊙ε

[0056] Where z is the latent variable, μ is the mean, σ is the standard deviation, ⊙ represents the Hadamard product, and ε is random noise sampled from the standard normal distribution;

[0057] The latent variable z of the text is calculated using the mean μ and standard deviation σ of the text information through the latent variable calculation formula. txt The latent variable z of vision is calculated using the mean μ and standard deviation σ of visual information through the latent variable calculation formula. img .

[0058] Furthermore, in step A3, aligning the encoded information of the two modalities using positive and negative samples specifically includes:

[0059] Positive sample score calculation:

[0060] score pos =Sigmoid(discriminator([sample txt ;sample img ]))

[0061] Negative sample score calculation:

[0062] score neg =Sigmoid(discriminator([sample txt ;negsample img ]))

[0063] Among them, sample txt It is the latent variable z of the text txt The category-level text latent variables obtained after slicing in the second dimension, sample img It is composed of visual latent variable z img The category-level visual latent variable, negsample, obtained after slicing in the second dimension. img It is a sample img The negative samples obtained after rolling are used as a discriminator and an activation function.

[0064] Further, in step A4, the fusion of the aligned two modal information through a multimodal fusion module specifically includes:

[0065] First, a multimodal mutual encoder is used to initially fuse the text and visual information. The formula for calculating the mutual attention matrix is ​​as follows:

[0066]

[0067] in, It is a scaling factor. It is the latent variable z in the text. txt The bonds generated after linear layer processing It is the visual latent variable z img The keys generated after linear layer processing, T denote transpose. Then, visually-involved text features and text-involved visual features are calculated.

[0068] The formula for calculating visually-involved text features is as follows:

[0069]

[0070] The formula for calculating visual features involving text is as follows:

[0071]

[0072] in, It is the latent variable z in the text. txt The value generated after linear layer processing It is the visual latent variable z img The value generated after linear layer processing;

[0073] Finally, the fused textual and visual feature information is calculated:

[0074]

[0075] The F obtained here mul This is the result of the first fusion. The result here is processed through a linear layer and used for the next multi-head attention encoder. The output of the last multi-head attention encoder is then used for relation prediction.

[0076] Further, in step A5, the prediction module predicts the entity relationships in the training set, specifically including:

[0077] First, obtain the raw score y for each sample using a classifier. i Then for each sample's y i Apply the Softmax function and normalize it to a probability distribution:

[0078]

[0079] Among them, y i It contains c different categories, y i [c] represents the predicted value of sample i in category c. p i [c] represents the i-th sample y. i The probability of belonging to class c.

[0080] Furthermore, step A6 specifically includes:

[0081] A61. Calculate the divergence loss function:

[0082]

[0083] Where D represents the dimension of the latent variables, μ i and σ i These represent the mean and standard deviation, respectively; the divergence loss of the text is calculated using the mean and standard deviation of the text information through a divergence loss function. The divergence loss of vision is calculated using the mean and standard deviation of visual information via a divergence loss function.

[0084] A62. Calculate the loss function for positive and negative samples:

[0085] First, calculate the positive sample loss function, with the objective being a full-1 tensor, meaning that positive samples should be correctly identified as matches by the model. The formula is as follows:

[0086]

[0087] Then, the negative sample loss function is calculated, with the target being the all-zero tensor, meaning that negative samples should be identified by the model as incorrect matches. The formula is as follows:

[0088]

[0089] Among them, scores pos and scores neg Let num represent the model prediction scores for positive and negative samples, respectively. neg The number of negative samples is represented by ones, which represents a tensor matrix of all 1s, and zeros represents a tensor matrix of all 0s. CrossEntropy is the binary cross-entropy.

[0090] Positive and negative sample loss:

[0091]

[0092] The total loss can be obtained by adding the two losses together.

[0093] A63. Calculate the prediction loss function:

[0094]

[0095] Among them, y i It contains c different categories, N is the number of samples, and y i [c] represents the predicted value of sample i in category c;

[0096] A64. Combining all the loss functions mentioned above yields the final loss of the model. And minimize the final loss using stochastic gradient descent. The following formula is used to train a multimodal relation extraction model:

[0097]

[0098] Where β1, β2, and β3 are the weight parameters of the loss function. and These represent the divergence loss for text and visual data, respectively.

[0099] The advantages of this invention over the prior art are:

[0100] 1) This invention considers the contribution of different auxiliary images to the global image. By designing a feature vector dynamic allocation module, each pair of images can adaptively generate a cue vector containing key-value pairs, thereby avoiding the negative impact of some auxiliary images with low contribution value on model prediction.

[0101] 2) The modal imbalance problem caused during modal fusion was considered. A mutual attention fusion module was constructed based on the features of multimodal data to generate fusion features for textual and visual information. Finally, the two features were fused in a balanced manner, avoiding the modal loss problem that may be caused by ordinary encoders. Attached Figure Description

[0102] Figure 1 This is an overall flowchart of the multimodal relationship extraction method in this invention;

[0103] Figure 2 This is a schematic diagram illustrating pyramid feature extraction according to the present invention;

[0104] Figure 3 This is a schematic diagram illustrating the dynamic generation of feature vectors in this invention;

[0105] Figure 4 This is a schematic diagram illustrating the generation of key-value pair hint vectors in this invention;

[0106] Figure 5 This is a schematic diagram illustrating the principle of multimodal fusion in this invention.

[0107] Figure 6 This is a schematic diagram illustrating the process of multimodal relation extraction in this invention. Detailed Implementation

[0108] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0109] This invention aims to propose an adaptive multimodal relation extraction method based on a mutual attention mechanism, overcoming the problem that existing methods cannot effectively fuse textual and visual information, thus enabling it to better perform multimodal relation extraction tasks when faced with complex text-image pairs. Its overall implementation process is as follows: Figure 1 As shown, it includes the following steps:

[0110] Step S1: Extract the layered visual features of the image, use each layer of visual features to generate cue vectors, and use the top layer of visual features for subsequent denoising, alignment and modality fusion.

[0111] Step S2: Generate adaptive cue vectors from the hierarchical visual features using an encoder;

[0112] Step S3: Use the generated prompt vector as an aid to encode the text and extract the corresponding text information;

[0113] Step S4: Denoise the top-level visual features and encoded text information using reparameterization techniques, and align them using positive and negative samples.

[0114] Step S5: Fuse the two processed modal information;

[0115] Step S6: Calculate the divergence loss, positive and negative sample loss and prediction loss respectively, and combine all the losses to train the multimodal relation extraction model using stochastic gradient descent.

[0116] Step S7: When performing a specific multimodal relation extraction task, the above operations are performed on the text-image pair to finally obtain the relationship type between the two entities in the text.

[0117] Example:

[0118] The following uses a text-image pair consisting of text (Text) and image (Img) as an example to illustrate the solution of the present invention in detail. The specific implementation process is as follows:

[0119] Step S1: First, encode the image and extract its layered visual features. This includes the following steps:

[0120] Step S11: Image Processing. Image Img contains a global image and three auxiliary images. Each image is extracted using the YOLOv3 vision toolkit, and then resized to 224×224 to obtain a processed global image and three auxiliary images I. Aux ={I A1 I A2 I A3}; where I A1 For the first auxiliary image, I A2 For the second auxiliary image, I A3 This is the third auxiliary image.

[0121] Step S12: Extract layered visual features. For example... Figure 2 As shown, the global image obtained in step S11 and the auxiliary image I are compared. Aux Encoding is performed using a pyramid module. The global image I and the first auxiliary image I are... A1 Second auxiliary image I A2 Third auxiliary image I A3 Each image undergoes encoding operations via a ResNet-based visual encoder within a pyramid module, followed by convolution operations using convolutional kernels. Each image generates a sequence of four layers of encoded information of varying sizes, resulting in the global image encoding sequence P. I and auxiliary image coding sequence P Aux ={P A1P A2 P A3}

[0122] Step S2: Integrate the hierarchical visual features obtained in Step S1 and dynamically generate corresponding cue vectors. This specifically includes the following steps:

[0123] Step S21: Dynamically generate feature vectors. For example... Figure 3 As shown, in order to make the global image coding sequence P I and auxiliary image coding sequence P Aux The processed global cue vector F can be obtained by further calculations, including segmentation, stacking, and reshaping. I and auxiliary prompt vector list F Aux ={F A1 F A2 F A3}. Among them, F A1 F is the first auxiliary cue vector. A2 F is the second auxiliary cue vector. A3 This is the third auxiliary cue vector. And F I F A1 F A2 F A3 ∈R 20bsz×hz bsz represents the batch size, and hz represents the hidden layer size.

[0124] To calculate the contribution of each auxiliary image to the global image, the global cue vector F is used. I and the first auxiliary cue vector F A1 For example, to calculate the contribution of the first auxiliary image to the global image, the weight score is first calculated using the following weight formula:

[0125]

[0126] Among them, the global cue vector F is used. I The global context weight W is calculated using the corresponding query Q and key K through a weight formula. I ∈R 20bsz×20bsz Using the global cue vector F I The corresponding query Q and the first auxiliary hint vector F A1 The corresponding key K is calculated to obtain the first auxiliary context weight W through the weight formula. A1 ∈R 20bsz×20bsz Softmax is the normalized activation function, d k This represents the parameter factor, preventing large values ​​from saturating the Softmax function and thus resulting in a smoother weight distribution. The weight scores are then applied to the following contextual formula to obtain contextual hints:

[0127] T = W × V

[0128] Among them, global context weight W is used. I and global cue vector F I The corresponding value V is used to calculate the global context F using the context formula. I Using the first auxiliary context weight W A1 and the first auxiliary cue vector F A1 The corresponding value V is used to calculate the first auxiliary context T through the context formula. A1 To obtain adaptively generated context information, the global context T is... I and the first auxiliary context T A1 We sum the weighted information to obtain the first context information:

[0129] FC A1 =F I +αT A1

[0130] Among them, FC A1 ∈R 20bsz×m This represents the first context information obtained, α represents the weight parameter, and m represents the value obtained by dividing the hidden layer size hz by the number of attention heads h.

[0131] The module provides a set of global cue vectors F I and the first auxiliary cue vector F A1 By performing the above operations h times, h first context information FCs are obtained. A1 For subsequent processing, all output first context information is linearly integrated together:

[0132]

[0133] Where Concat is the concatenation function, W O It is the output projection matrix. FCs A1 This is the first encoded information obtained by integrating all the first context information. Then, two normalization steps are performed to ensure that the data distribution of each sample is roughly the same across different feature dimensions. Simultaneously, the ReLU function is used to introduce non-linearity to obtain the processed information. The feature vector dynamic allocation module assigns the same set of global cue vectors F... I The first auxiliary cue vector F A1 After performing the above process four times, the final first context list is obtained, and the result is:

[0134] Similarly, the global cue vector F I Second auxiliary cue vector F A2After performing the same operations as above, a second context list FCL2 is obtained; the global hint vector F is then... I and the third auxiliary cue vector F A3 The same operations described above are performed to obtain the third context list FCL3. The three context lists {FCL1, FCL2, FCL3} are then integrated to obtain the feature vector F.

[0135] F = Stack(FCL1, FCL2, FCL3)

[0136] Here, Stack is the stacking function. The final feature vector F is obtained by integrating the encoded information of each image pair.

[0137] Step S22: Generate a hint vector containing key-value pairs. For example... Figure 4 As shown, to determine the degree to which each final context in the feature vector F enhances the text input sequence, a gating value needs to be calculated. The model can dynamically decide whether to activate specific cue information based on the features of the input data, thereby controlling the degree of influence of these cue information on the final output. The specific formula for the gating value PG is as follows:

[0138]

[0139] Among them, f img This represents each coding layer in the feature vector F. In this embodiment, the feature vector F contains three coding layers, corresponding to three context lists FCL1, FCL2, FCL3, and ∑. seq f img Represents along f img Summing the third dimension, Gate idx Let f represent the idx-th threshold function; LeakyReLu is an activation function. Each encoding layer f in the feature vector F... img The contributions to the final cue vector are all different, and the weight of the contribution of each coding layer can be controlled by the gating unit.

[0140] Connect the i-th row of the gate value PG with f img Multiply each element in the i-th column one by one, and then sum them:

[0141] f val =Einsum(′b,bh->bh′,PG[:,i],f img [i])

[0142] Where Einsum represents the expression for PG[:,i] and f img [i] Perform element-wise multiplication and summation, and calculate f four times to obtain f. val1 f val2 fval3 f val4 Add them together to get the contribution weight:

[0143] F val =f val1 +f val2 +f val3 +f val4

[0144] For each coding layer f in the feature vector F img Perform the above operations on all cases, and then calculate all the resulting contribution weights {F}. val1 F va2 F va3 Add to the same list:

[0145] f′ p =F val1 ∪F va2 ∪F va3

[0146] f′ p The process involves splitting, stacking, and reshaping to obtain a cue vector f containing key-value pairs. p .

[0147] Step S3: The hint vector f obtained in step S2 p The text encoding information is obtained as a cue for text encoding, and then noise reduction and alignment are performed on the information from both the text and visual modalities. Specifically, the following steps are included:

[0148] Step S31: Obtain textual and visual information. The cue vector f obtained in step S22... p As a prefix, given the input text Text, the text is encoded using the BERT (Bidirectional Encoder Representations from Transformers) model. Two special tags, [CLS] and [SEP], are inserted at the beginning and end of each extracted sentence, respectively. The processed sentence information is used as the text information seq.

[0149] The global image encoding sequence P obtained in step S12 I and auxiliary image coding sequence P Aux ={P A1 P A2 P A3 The top layer of visual information is spliced ​​together and then processed through segmentation, stacking, and reshaping to obtain the visual information vis.

[0150] Step S32: Denoise the text and visual information. To allow the gradient to pass through, a reparameterization technique is used to transform random sampling into a deterministic function with random noise, thus allowing the gradient to flow during the sampling operation while maintaining the ability to backpropagate the gradient during sampling. The latent variable calculation formula is as follows:

[0151] z = μ + e (logσ) ⊙ε

[0152] Where z is the latent variable, μ is the mean, σ is the standard deviation, ⊙ represents the Hadamard product, and ε is random noise sampled from the standard normal distribution.

[0153] The latent variable z of the text information seq is calculated using the latent variable calculation formula based on the mean μ and standard deviation σ of the latent variable. txt The visual latent variable z is calculated using the mean μ and standard deviation σ of visual information vis through the latent variable calculation formula. img Reparameterization techniques allow for control over the distribution of latent variables by adjusting the mean and log-standard deviation, thereby optimizing model performance while maintaining distributional flexibility.

[0154] Step S33: Align the text and visual information. Since text and visual information are encoded by different encoders, their representations differ. To eliminate this modal difference, the encoded information of the two modalities is aligned using positive and negative samples. Specifically, this includes:

[0155] Positive sample score calculation:

[0156] score pos =Sigmoid(discriminator([sample txt ;sample img ]))

[0157] Negative sample score calculation:

[0158] score neg =Sigmoid(discriminator([sample txt ;negsample img ]))

[0159] Among them, sample txt It is the latent variable z of the text txt The category-level text latent variables obtained after the second-dimensional slicing, sample img It is composed of visual latent variable z img The category-level visual latent variable obtained after slicing the second dimension, negsample img It is a sampleimg After the negative samples are obtained through rolling, the discriminator is used, and the sigmoid function is used as the activation function. The output of the discriminator is activated using the sigmoid function, and the probability value between 0 and 1 is obtained as the sample score. The target value for positive samples is 1, and the target value for negative samples is 0. The cross-entropy loss function is used to calculate the loss between the sample score and the target value.

[0160] loss=CrossEntropy(score;target)

[0161] Here, score represents the sample score, target represents the target value, and CrossEntropy represents the binary cross-entropy. For example, the score of a positive sample is scorepos, and the corresponding target value is the all-one tensor one.

[0162] Step S4: Perform modal fusion on the processed text and visual information from step S3, while ensuring a balance between the two modalities. This specifically includes the following steps:

[0163] Step S41: Perform multimodal fusion. For example... Figure 5 As shown, in order to perform modal fusion and balance the information of the two modalities, a multimodal mutual encoder is first used to initially fuse the text and visual information. The formula for calculating the mutual attention matrix is ​​as follows:

[0164]

[0165] in, It is a scaling factor. The key is generated by processing text information through a linear layer. The key is generated by processing visual information through a linear layer, where T represents transpose. Then, text features involving vision and visual features involving text are calculated. The text and visual information here are derived from the latent text variable z obtained in step S32, respectively. txt and visual latent variable z img .

[0166] The formula for calculating visually-involved text features is as follows:

[0167]

[0168] The formula for calculating visual features involving text is as follows:

[0169]

[0170] in, It is the latent variable z in the text. txt The value generated after linear layer processing It is the visual latent variable z img The value generated after processing by a linear layer.

[0171] Finally, the fused textual and visual feature information is calculated:

[0172]

[0173] in, Visual features that indicate text involvement The F represents the text features involving visual input, which is obtained here. mul This is the result of the first fusion. Passing this result through a linear layer, we get the result out1.

[0174] Step S42: Perform two encoding operations using the encoder. To obtain more accurate fusion information, the result here is passed through a linear layer, processed, and then used for the next multi-head attention encoder. The output of the last multi-head attention encoder is then used for relation prediction, as shown in the following formula:

[0175] First coding:

[0176] out2 = CrossAttetion(z img ;out1;out1)

[0177] Second encoding:

[0178] final = CrossAttetion(z) txt ;out2;out2)

[0179] CrossAttention is an encoder designed based on the attention mechanism. Out2 represents the output of the first multi-head attention encoder, and final represents the output of the last multi-head attention encoder, which is the final fusion result. This result is used for relation prediction.

[0180] Step S5: Predict entity relationships in the training set using the prediction module. This specifically includes the following steps:

[0181] Step S51: Extract the entity hidden state. The final fusion result `final` obtained in step S42 is the final text feature obtained after the text-to-image cross-modal attention mechanism. To obtain the logical value for category prediction, these features are usually passed to a classifier, typically a linear layer.

[0182] For each sample i in the final fusion result final, extract the head entity h using the head index. i Extract the tail entity t using the tail index. i Then connect them to form the entity hidden state ei :

[0183] e i =[h i ;t i ]

[0184] The hidden state e of each entity i Pass it to the classifier to get the logical value of the class prediction:

[0185] y i =Classifier(e i )

[0186] Where Classifier is the classifier, y i This is the logical value of the obtained category prediction, also known as the category prediction score or raw score.

[0187] Step S52: Predict entity relationships. First, obtain the raw score y for each sample using a classifier. i Then for each sample's y i Apply the Softmax function and normalize it to a probability distribution:

[0188]

[0189] Among them, y i It contains c different categories, y i [c] represents the predicted value of sample i in category c. p i [c] represents the i-th sample y. i The probability of belonging to class c.

[0190] Steps S6 and S5 predict the relationships between entity pairs. In this step, the divergence loss, positive and negative sample loss, and prediction loss are calculated respectively, and the model is trained using stochastic gradient descent to update the parameter matrix. Specifically, this includes the following steps:

[0191] Step S61: Calculate the divergence loss function. Divergence loss is typically used to measure the difference between two probability distributions. It provides a way for the model to quantify this difference, helping the model learn to estimate the target probability distribution more accurately, thus enabling the optimization process to move in the direction of reducing this difference. The specific formula for the divergence loss function is as follows:

[0192]

[0193] Where D represents the dimension of the latent variables, μ i and σ i These represent the mean and standard deviation, respectively. The divergence loss of the text is calculated using the mean and standard deviation of the text information through a divergence loss function. The divergence loss of vision is calculated using the mean and standard deviation of visual information via a divergence loss function.

[0194] Step S62: Calculate the positive and negative sample loss function. The positive and negative sample loss function provides the model with a learning direction, guiding the model to adjust parameters to minimize the loss, thereby better fitting the training data. In step S33, the text latent variable z... txt and visual latent variable z img After processing, we obtain the category-level text latent variable sample. txt and category-level visual latent variable samples img And use it to calculate the positive sample scores. pos and negative sample scores neg The target value for positive samples is 1, and the target value for negative samples is 0. The cross-entropy loss function is used to calculate the loss between the sample score and the target value.

[0195] First, calculate the positive sample loss function, with the objective being a full-1 tensor, meaning that positive samples should be correctly identified as matches by the model. The formula is as follows:

[0196]

[0197] Here, ones represents a tensor matrix consisting entirely of 1s.

[0198] Then, the negative sample loss function is calculated, with the target being the all-zero tensor, meaning that negative samples should be identified by the model as incorrect matches. The formula is as follows:

[0199]

[0200] Among them, scores pos and scores neg Let num represent the model prediction scores for positive and negative samples, respectively. neg The number of negative samples is represented by , and zeros represents a tensor matrix consisting entirely of zeros.

[0201] Positive and negative sample loss:

[0202]

[0203] The positive and negative sample losses can be obtained by adding the positive sample loss and the negative sample loss together.

[0204] Step S63: Calculate the prediction loss function. A batch contains N samples, each with c possible classes. i ∈R CLet be the logistic value of the class prediction for the i-th sample. The cross-entropy loss function can be further expressed as:

[0205]

[0206] Among them, y i It contains c different categories.

[0207] Step S64: After obtaining the divergence loss, positive and negative sample loss, and prediction loss in steps S61, S62, and S63 respectively, this step combines all loss functions to perform joint training and obtain the final loss of the model. The formula is as follows:

[0208]

[0209] Where β1, β2, and β3 are the weight parameters of the loss function. and Let the divergence loss for text and vision be represented respectively, and the final loss be minimized using stochastic gradient descent. To train a multimodal relation extraction model for better relation prediction.

[0210] Step S7: After completing model training in step S6, the model can be used for practical multimodal relation extraction tasks. For example, for a text-image pair "Text" and "Img", it is necessary to predict the type of relationship between two entities in "Text". Figure 6 As shown, the first step is to extract feature information from the text and image, perform denoising and alignment processing, and then fuse the processed modal information. Then, similar to steps S51 and S52, the logical value of the category prediction is obtained, and the original score y of each sample is obtained through a classifier. i Then for each sample's y i Applying the Softmax function and normalizing it to a probability distribution yields the probability p. i [c]. Sort the probabilities and use the prediction with the highest probability as the final predicted relation type.

[0211] It is understood that the present invention has been described through some embodiments, and those skilled in the art will recognize that various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the invention. Furthermore, under the teachings of the present invention, these features and embodiments can be modified to adapt to specific situations and materials without departing from the spirit and scope of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the protection scope of the present invention.

Claims

1. An adaptive multimodal relation extraction method based on mutual attention mechanism, characterized in that, Includes the following steps: A. Training a multimodal relation extraction model: A1. Extract layered visual features from the image to be processed using the pyramid module, and use each layer of visual features as text-encoded prompt information; at the same time, process the top layer of visual features separately to form another type of encoded information, and fuse it with the text information in the fusion stage. A2. The feature vector dynamic allocation module processes the visual features of each layer to obtain feature vectors; then, the obtained feature vectors are used by the key-value pair generation module to extract the cue vectors containing key-value pairs. A3. Denoise the top-level visual feature information and text encoding information, and align the encoding information of the two modalities of text and image by means of positive and negative samples; A4. The aligned two modal information are fused through a multimodal fusion module to obtain the coding information used for prediction; A5. Use the prediction module to predict entity relationships in the training set; A6. Calculate the loss function and sum it proportionally to obtain the final loss value of this model. Train the multimodal relation extraction model by stochastic gradient descent. B. Multimodal Relation Extraction Task: The trained relation extraction model was used in a real multimodal relation extraction task to generate the most likely relation type for entity pairs in the text based on visual cues. In step A2, the processing of each layer of visual features through the feature vector dynamic allocation module specifically includes: The global image encoding sequence obtained in A1 and auxiliary image coding sequence The process involves segmentation, stacking, and reshaping to obtain the corresponding global cue vector. and auxiliary prompt vector list ; Calculate the weight score using the multi-head attention mechanism weight formula: Among them, global hint vectors are used. Corresponding query s and keys The global context weight is calculated using the weight formula. Use global hint vectors Corresponding query and the Auxiliary cue vectors Corresponding key Calculation yields the first Auxiliary context weights , It is a normalized activation function. This represents the parameter factor. Calculate the weighted score using the following contextual formula: Among them, global context weights are used. and global hint vector corresponding value The global context is obtained through context formula calculation. , using the Auxiliary context weights and the Auxiliary cue vectors corresponding value The first is obtained by calculating using the context formula. Auxiliary context ; Add the global context to the first In the auxiliary context, thus obtaining the adaptively generated context information: in, The representative obtained the first Contextual information, Represents the weighting parameter; The module is for each group and pass The above operations yielded Context information All context information Linear integration, The number of attention heads: in, It's a concatenation function. It outputs the projection matrix; then it performs two normalization steps to ensure that the data distribution of each sample is roughly the same across different feature dimensions, and also uses the ReLU function to introduce non-linearity to obtain the processed information. The feature vector dynamic allocation module processes the same pair of image encoding sequences four times to obtain a final list of contexts. ; global hint vector Each auxiliary cue vector is processed separately to obtain a context list, and all the obtained context lists are integrated to obtain the feature vector: in, It is a stacking function; the final eigenvector is obtained. It is the final feature vector obtained by integrating the encoded information of each image pair; In step A2, the step of extracting a hint vector containing key-value pairs from the obtained feature vector through the key-value pair generation module specifically includes: By calculating a gating value, the model dynamically determines whether to activate specific prompts based on the characteristics of the input data, thereby controlling the degree of influence of these prompts on the final output. The specific formula is as follows: in, Representing the eigenvector Each coding layer in the code corresponds to Zhang Context List ; Representative along Summing is performed on the third dimension. Indicates the first One threshold function; eigenvector Each coding layer in The contributions to the final cue vector are all different. The weights of the contributions from each coding layer can be controlled through a gating unit. The Action and The Multiply each element of the column one by one, and then sum them up. in, Indicates to and Perform element-wise multiplication and then sum the results, calculating four times to obtain the final result. Add them together to get the contribution weight: For eigenvectors Each coding layer in Perform the above operations on all of them, then add all the resulting contribution weights to the same list, and perform splitting, stacking, and reshaping to obtain a hint vector containing key-value pairs.

2. The adaptive multimodal relation extraction method based on mutual attention mechanism as described in claim 1, characterized in that, In step A1, the extraction of hierarchical visual features using the pyramid module specifically includes: The global image and its corresponding auxiliary image in the dataset are cropped to obtain the cropped global image. and auxiliary images ;in, Indicates the number of auxiliary images. ; Based on ResNet, the pyramid module is used to process the global image. and auxiliary images Encode the image to generate the corresponding global image encoding sequence. and auxiliary image coding sequence .

3. The adaptive multimodal relation extraction method based on mutual attention mechanism as described in claim 1, characterized in that, In step A3, the noise reduction processing of the top-level visual feature information and text encoding information specifically includes: Using the key-value pair cue vectors obtained from A2 as an aid, BERT is used to encode the text. Two special markers, [CLS] and [SEP], are inserted at the beginning and end of each extracted sentence, respectively. The processed sentence information is used as the text information. ; The global image encoding sequence obtained in A1 and auxiliary image coding sequence The top layer of visual information is stitched together, and then processed through segmentation, stacking, and reshaping to obtain the final visual information. ; To allow the gradient to pass through, a reparameterization technique is used to transform random sampling into a deterministic function with added random noise. This allows the gradient to flow during the sampling operation while maintaining the ability to backpropagate the gradient during sampling. The latent variable calculation formula is as follows: in, As latent variables, The mean, Standard deviation It represents the Hadamah accumulation. It is random noise sampled from a standard normal distribution; The mean of the text information and standard deviation Calculate the latent variables of the text Using the mean of visual information and standard deviation Calculate the latent variables of vision using the latent variable calculation formula. .

4. The adaptive multimodal relation extraction method based on mutual attention mechanism as described in claim 3, characterized in that, In step A3, aligning the encoded information of the two modalities using positive and negative samples specifically includes: Positive sample score calculation: Negative sample score calculation: in, It is a latent variable of text The category-level text latent variables obtained after slicing in the second dimension, It is composed of visual latent variables The category-level visual latent variables obtained after slicing in the second dimension, yes Negative samples obtained through rolling It is a discriminator; It is an activation function.

5. The adaptive multimodal relation extraction method based on mutual attention mechanism as described in claim 1, characterized in that, In step A4, the fusion of the aligned two modal information through a multimodal fusion module specifically includes: First, a multimodal mutual encoder is used to initially fuse the text and visual information. The formula for calculating the mutual attention matrix is ​​as follows: in, It is a scaling factor. Textual latent variables The bonds generated after linear layer processing Visual latent variables The bonds generated after linear layer processing The transpose is indicated; then, the visually involved text features and the visually involved text features are calculated. The formula for calculating visually-involved text features is as follows: The formula for calculating visual features involving text is as follows: in, Textual latent variables The value generated after linear layer processing Visual latent variables Values ​​generated after linear layer processing; Finally, the fused textual and visual feature information is calculated: The result here This is the result of the first fusion. The result here is processed through a linear layer and used for the next multi-head attention encoder. The output of the last multi-head attention encoder is then used for relation prediction.

6. The adaptive multimodal relation extraction method based on mutual attention mechanism as described in claim 1, characterized in that, In step A5, the prediction module predicts the entity relationships in the training set, specifically including: First, obtain the raw score for each sample using a classifier. Then for each sample Apply the Softmax function and normalize it to a probability distribution: in, It includes Different categories, Indicates sample In category The predicted value; That is, the first Sample Belongs to the The probability of a class.

7. The adaptive multimodal relation extraction method based on mutual attention mechanism as described in any one of claims 1-6, characterized in that, Step A6 specifically includes: A61. Calculate the divergence loss function: in, The dimension of the latent variable. and These represent the mean and standard deviation, respectively; the divergence loss of the text is calculated using the mean and standard deviation of the text information through a divergence loss function. The divergence loss of vision is calculated using the mean and standard deviation of visual information through a divergence loss function. ; A62. Calculate the loss function for positive and negative samples: First, calculate the positive sample loss function, with the objective being a full-1 tensor, meaning that positive samples should be correctly identified as matches by the model. The formula is as follows: Then, the negative sample loss function is calculated, with the target being the all-zero tensor, meaning that negative samples should be identified by the model as incorrect matches. The formula is as follows: in, and These represent the model prediction scores for positive and negative samples, respectively. Indicates the number of negative samples. Represents a tensor matrix consisting of all 1s. Represents a tensor matrix consisting entirely of zeros. It is a binary cross-entropy; Positive and negative sample loss: The total loss can be obtained by adding the two losses together. A63. Calculate the prediction loss function: in, It includes Different categories, For the number of samples, Indicates sample In category The predicted value; A64. Combining all the loss functions mentioned above yields the final loss of the model. And minimize the final loss using stochastic gradient descent. The following formula is used to train a multimodal relation extraction model: in, , and These are the weight parameters of the loss function. and These represent the divergence loss for text and visual data, respectively.

Images