Self-adaptive multi-modal relation extraction method based on mutual attention mechanism

Abstract

The invention discloses a self-adaptive multi-modal relation extraction method based on a mutual attention mechanism, and relates to the field of multi-modal relation extraction. Firstly, hierarchical feature information of an image is extracted through a pyramid module, and then a self-adaptive feature vector is generated by using the information; denoising the extracted text information and the uppermost visual information, and aligning the text information and the uppermost visual information; fusing the two types of processed modal information through a multi-modal mutual attention fusion module; calculating a loss function, and training the model by minimizing loss through a stochastic gradient descent method; when a multi-modal relation extraction task is executed, after a text-image pair is obtained, an original score of each sample is obtained through a classifier and is normalized into probability distribution to obtain a probability, and finally the multi-modal relation extraction task is completed according to a probability sorting result. According to the invention, the negative influence of the auxiliary image with low contribution value on model prediction can be avoided, and the modal loss problem possibly caused by a common encoder can be avoided.

Description

Technical Field

[0001] The present invention relates to the technical field of relationship extraction, and in particular to an adaptive multimodal relationship extraction method based on a mutual attention mechanism. Background Art

[0002] The goal of relation extraction (RE) is to identify the semantic relationship between two given entities in text information, and it plays a vital role in tasks such as question answering and knowledge base retrieval. Most existing text-based RE methods can be roughly divided into sequence-based models and dependency-based models. These models can only extract information from text, and their prediction performance decreases significantly when the text information lacks sufficient context. Therefore, using visual information to assist the extraction of entity pairs in text has become the main focus of current research, namely multimodal relation extraction (MRE). MRE helps the model identify entity pairs in text through image information, thereby more effectively improving the prediction accuracy.

[0003] For the extraction of visual information, existing methods have introduced an adaptive co-attention network, which extends the combination of bidirectional long short-term memory network and conditional random field, takes global image information as clues, and introduces an adaptive co-attention mechanism. There are also methods that propose a new neural network model that uses visual objects in the image as fine-grained image representations, which can retain more local information in the image. However, these methods focus only on the extraction of visual information, but do not consider how to fuse two different modal information.

[0004] The existing ideas for solving this problem can be roughly divided into two types:

[0005] 1. Extract visual information as visual prefix to help extract text information;

[0006] This method extracts visual information from the image and uses the extracted visual information as a hint for text entity extraction for prefix fusion.

[0007] 2. Extract the information of the two modalities separately and use the attention encoder to perform modal fusion;

[0008] This method can make the most of visual information to complete text information, but it requires an effective visual information extraction method.

[0009] The MRE dataset contains the global image and auxiliary images obtained by cropping the whole image. However, for each text-image pair, not all auxiliary images contained in the image can necessarily improve the prediction performance. Existing methods ignore the above problems, and the attention-based encoding model used in the fusion stage also leads to the problem of modal information loss. Summary of the invention

[0010] The purpose of the present invention is to propose an adaptive multimodal relationship extraction method based on a mutual attention mechanism, so as to better complete the task of multimodal relationship extraction when facing complex text-image pairs. This is to solve the technical problems that the prior art cannot effectively fuse text and visual information, and the attention-based encoding model used in the fusion stage of the prior art will also cause the loss of modal information.

[0011] The technical solution adopted by the present invention to solve the above technical problems is:

[0012] An adaptive multimodal relation extraction method based on mutual attention mechanism includes the following steps:

[0013] A. Training multimodal relationship extraction model:

[0014] A1. Use the pyramid module to extract hierarchical visual features from the image to be processed, and use each layer of visual features as prompt information for text encoding; at the same time, process the top layer of visual features separately to form another type of encoding information, and fuse it with the text information in the fusion stage;

[0015] A2. Process each layer of visual features through the feature vector dynamic allocation module to obtain a feature vector. Then, the obtained feature vector is used to extract the prompt vector containing the key-value pair through the key-value pair generation module;

[0016] A3: De-noise the top-level visual feature information and text encoding information, and align the encoding information of the two modalities of text and image through positive and negative samples;

[0017] A4, fusing the aligned two modal information through a multimodal mutual fusion module to obtain coding information for prediction;

[0018] A5. Use the prediction module to predict the entity relationships in the training set;

[0019] A6. Calculate the loss function and sum it proportionally to get the final loss value of the model, and train the multimodal relationship extraction model by stochastic gradient descent.

[0020] B. Multimodal relationship extraction task:

[0021] The trained relation extraction model is used in the actual multimodal relation extraction task to generate the most likely relation type for entity pairs in the text based on visual cues.

[0022] Furthermore, in step A1, extracting hierarchical visual features using a pyramid module specifically includes:

[0023] The global image and the corresponding auxiliary image in the data set are cropped to obtain the cropped global image and the auxiliary image I Aux = {I A1 , ..., I Aj , ..., I An}; where n represents the number of auxiliary images, j∈[1,n];

[0024] Based on ResNet, a pyramid module is used to transform the global image and auxiliary image I Aux Encode and generate the corresponding global image coding sequence P I and auxiliary picture coding sequence P Aux = {P A1 , ..., P Aj , ..., P An};

[0025] Furthermore, in step A2, processing each layer of visual features by the feature vector dynamic allocation module specifically includes:

[0026] For the global image coding sequence P obtained in A1 I and auxiliary picture coding sequence P Aux Perform segmentation, stacking and reshaping to obtain the corresponding global prompt vector F I and auxiliary hint vector list F Aux ={F A1 , ..., F Aj , ..., F An};

[0027] The weight score is calculated using the multi-head attention mechanism weight formula:

[0028]

[0029] Among them, the global hint vector F is used I The corresponding query Q and key K are calculated by the weight formula to obtain the global context weight W I , using the global hint vector F I The corresponding query Q and the j-th auxiliary hint vector F Aj The corresponding key K is used to calculate the jth auxiliary context weight W A , Softmax is the normalized activation function, dk represents the parameter factor,

[0030] The weight score is calculated through the following contextual formula to calculate the contextual hint:

[0031] T=W×V

[0032] Among them, the global context weight W is used I and the global hint vector F I The corresponding value V is calculated by the context formula to obtain the global context F I , using the jth auxiliary context weight W A and the jth auxiliary cue vector F Aj The corresponding value V is calculated by the context formula to obtain the jth auxiliary context T A ;

[0033] Add the global context to the jth auxiliary context to obtain the adaptively generated context information:

[0034] FC=F I +αT A

[0035] Among them, FC represents the jth context information obtained, and α represents the weight parameter;

[0036] Module for each group F I and F Aj Through h times of the above operation, h context information FC is obtained. All context information FC are linearly integrated, and h represents the number of attention heads:

[0037] FCs=Concat(FC1, FC2,..., FC h )W o

[0038] Among them, Concat is the concatenation function, W o is the output projection matrix; then two normalization steps are performed to ensure that the data distribution of each sample is roughly the same in different feature dimensions, and the ReLU function is used to introduce nonlinearity to obtain the processed information;

[0039] The feature vector dynamic allocation module processes the same pair of image coding sequences four times to obtain a list of the final context after processing, the result of which is FCL j ={FCs1, FCs2, FCs3, FCs4};

[0040] The global hint vector F I The context lists are processed separately with all auxiliary hint vectors, and all the context lists are integrated to get the feature vector:

[0041] F=Stack(FCL1,...,FCL j , ..., FCL n )

[0042] Among them, Stack is a stacking function; the final feature vector F is the final feature vector obtained by integrating the encoding information of each image pair.

[0043] Furthermore, in step A2, extracting a prompt vector containing a key-value pair from the obtained feature vector through a key-value pair generation module specifically includes:

[0044] By calculating a gate value, the model can dynamically decide whether to activate specific prompt information based on the characteristics of the input data, thereby controlling the degree of influence of these prompt information on the final output. The specific formula of the gate value PG is as follows:

[0045]

[0046] Among them, f img Represents each encoding layer in the feature vector F, corresponding to n context lists FCL1, FCL2, ..., FCL n ;∑ seq f img Represents along f img The third dimension of Gate idx represents the idxth threshold function; each coding layer f in the feature vector F img The contribution to the final hint vector is different. The weight of the contribution of each coding layer can be controlled by the gating unit, which connects the i-th row of PG with f img Multiply the elements of the i-th column element by element and then sum them;

[0047] f val =Einsum(′b,bh->bh′,PG[:,i],f img [i])

[0048] Among them, Einsum represents the sum of PG[:, i] and f img [i] Perform element-by-element multiplication and then sum, and calculate four times to get f val1 , f val2 , f val3 , f val4 , and add them together to get the contribution weight:

[0049] F val =f val1 +f val2 +f val3 +f val4

[0050] For each encoding layer f in the feature vector F img The above operations are performed for all, and then all the resulting contribution weights are added to the same list, split, stacked, and reshaped to obtain a prompt vector containing key-value pairs.

[0051] Furthermore, in step A3, the denoising process of the top-level visual feature information and text encoding information specifically includes:

[0052] Using the key-value pair hint vector obtained in A2 as an aid, use BERT to encode the text, insert two special tags [CLS] and [SEP] at the beginning and end of each extracted sentence, and use the processed sentence information as the text information seq;

[0053] The global image coding sequence P obtained in A1 I and auxiliary picture coding sequence P Aux The top-level visual information is spliced, and then segmented, stacked, and reshaped to obtain the visual information vis;

[0054] In order to make the gradient passable, the random sampling is converted into a deterministic function plus random noise through the reparameterization trick, allowing the gradient to flow in the sampling operation while maintaining the ability to backpropagate the gradient during the sampling process. The latent variable calculation formula is as follows:

[0055] z=μ+e( l ogσ)⊙ε

[0056] Where z is the latent variable, μ is the mean, σ is the standard deviation, ⊙ represents the Hadamard product, and ε is the random noise sampled from the standard normal distribution;

[0057] The latent variable z of the text is calculated using the mean μ and standard deviation σ of the text information through the latent variable calculation formula txt , use the mean μ and standard deviation σ of the visual information to calculate the visual latent variable z through the latent variable calculation formula img .

[0058] Furthermore, in step A3, aligning the encoded information of the two modes by means of positive and negative samples specifically includes:

[0059] Positive sample score calculation:

[0060] score pos =Sigmoid(discriminator([sample txt ;sample img ]))

[0061] Negative sample score calculation:

[0062] score neg =Sigmoid(discriminator([sample txt ; negsample img ]))

[0063] Among them, sample txt is the textual latent variable z txt The category-level text latent variable obtained after slicing the second dimension, sample img is the visual latent variable z img Category-level visual latent variables obtained after slicing along the second dimension, negsample img is sample img The negative sample obtained after rolling, discriminator is the discriminator; Sigmoid is the activation function.

[0064] Furthermore, in step A4, the aligned two modal information are fused through a multimodal mutual fusion module, specifically including:

[0065] First, the text and visual information are initially fused through a multimodal mutual encoder, and the formula for calculating the mutual attention matrix is ​​as follows:

[0066]

[0067] in, is the scaling factor, is the textual latent variable z txt The key generated by the linear layer, is the visual latent variable z img The key generated by the linear layer processing, T represents the transpose. Then, the text features participated by the visual and the visual features participated by the text are calculated;

[0068] The formula for calculating the text features of visual participation is as follows:

[0069]

[0070] The formula for calculating the visual features of text participation is as follows:

[0071]

[0072] in, is the textual latent variable z txt The value generated by the linear layer is is the visual latent variable z img The value generated by the linear layer;

[0073] Finally, the fused text and visual feature information is calculated:

[0074]

[0075] Here we get F mul This is the result of the first fusion. The result here is passed through a linear layer and processed for the next multi-head attention encoder, and the output of the last multi-head attention encoder is used for relationship prediction.

[0076] Furthermore, in step A5, the prediction module is used to predict the entity relationship in the training set, specifically including:

[0077] First, the original score y of each sample is obtained through the classifier i , then for each sample y i Apply the Softmax function and normalize it to a probability distribution:

[0078]

[0079] Among them, y i contains c different categories, y i [c] represents the predicted value of sample i in category c. i [c] is the i-th sample y i The probability of belonging to class c.

[0080] Furthermore, step A6 specifically includes:

[0081] A61. Calculate the divergence loss function:

[0082]

[0083] Where D represents the dimension of the latent variable, μ i and σ i Represent the mean and standard deviation respectively; the divergence loss of the text is calculated by the divergence loss function using the mean and standard deviation of the text information The visual divergence loss is calculated using the mean and standard deviation of the visual information through the divergence loss function

[0084] A62. Calculate the loss function of positive and negative samples:

[0085] First, the positive sample loss function is calculated. The target is a full 1 tensor, which means that the positive sample should be recognized as a correct match by the model. The formula is as follows:

[0086]

[0087] Then calculate the negative sample loss function, the target is a full 0 tensor, which means that the negative sample should be recognized by the model as an incorrect match. The formula is as follows:

[0088]

[0089] Among them, scores pos and scores neg Represents the model prediction scores of positive samples and negative samples respectively, num neg Represents the number of negative samples, ones represents a tensor matrix of all 1s, zeros represents a tensor matrix of all 0s, and CrossEntropy is the binary cross entropy;

[0090] Positive and negative sample loss:

[0091]

[0092] Add the two obtained losses together to get the total loss;

[0093] A63. Calculate the prediction loss function:

[0094]

[0095] Among them, y i contains c different categories, N is the number of samples, y i [c] represents the predicted value of sample i in category c;

[0096] A64. Combine all the above loss functions to obtain the final loss of the model And minimize the final loss by stochastic gradient descent To train the multimodal relationship extraction model, the formula is as follows:

[0097]

[0098] Among them, β1, α2 and β3 are the weight parameters of the loss function, and denote the divergence loss of text and vision respectively.

[0099] The beneficial effects of the present invention relative to the prior art are:

[0100] 1) The present invention takes into account the contribution of different auxiliary images to the global image. By designing a feature vector dynamic allocation module, each group of image pairs can adaptively generate a prompt vector containing key-value pairs, thereby avoiding the negative impact of some auxiliary images with lower contribution value on model prediction.

[0101] 2) The modal imbalance problem caused by modal fusion is taken into consideration. A mutual attention fusion module is constructed based on the multi-modal features, so that the text information and visual information produce their own fusion features. Finally, the two features are fused in a balanced manner, avoiding the modal loss problem that may be caused by ordinary encoders. BRIEF DESCRIPTION OF THE DRAWINGS

[0102] Figure 1 It is an overall flow chart of the multimodal relationship extraction method in the present invention;

[0103] Figure 2 A schematic diagram of pyramid feature extraction performed in the present invention;

[0104] Figure 3 A schematic diagram of the dynamic generation of feature vectors in the present invention;

[0105] Figure 4 A schematic diagram of generating a key-value pair prompt vector according to the present invention;

[0106] Figure 5 This is a schematic diagram of the principle of multimodal fusion of the present invention;

[0107] Figure 6 Schematic diagram of the process of multimodal relationship extraction in the present invention. DETAILED DESCRIPTION

[0108] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0109] The present invention aims to propose an adaptive multimodal relationship extraction method based on mutual attention mechanism, which can make up for the problem that existing methods cannot effectively integrate text and visual information, so that it can better complete the task of multimodal relationship extraction when facing complex text-image pairs. The overall implementation process is as follows Figure 1 As shown, the following steps are included:

[0110] Step S1, extracting hierarchical visual features of the image, using each layer of visual features to generate a prompt vector, and using the top layer of visual features for subsequent denoising alignment and modality fusion;

[0111] Step S2, generating an adaptive prompt vector by using the hierarchical visual features through an encoder;

[0112] Step S3, using the generated hint vector as an aid to encode the text and extract the corresponding text information;

[0113] Step S4, denoising the top-level visual features and the encoded text information by using a reparameterization technique, and aligning them using positive and negative samples;

[0114] Step S5, fusing the processed two modal information;

[0115] Step S6, respectively calculating the divergence loss, positive and negative sample loss, and prediction loss, combining all losses and training the multimodal relationship extraction model through the stochastic gradient descent method;

[0116] Step S7: When executing a specific multimodal relationship extraction task, the above operation is performed on the text-image pair to finally obtain the relationship type between the two entities in the text.

[0117] Example:

[0118] The following takes a text-image pair consisting of a text Text and an image Img as an example to explain the solution of the present invention in detail. The specific implementation process is as follows:

[0119] Step S1: first encode the image and extract the hierarchical visual features of the image. Specifically, the following steps are included:

[0120] Step S11, process the image. The image Img contains a global image and three auxiliary images. Each image is extracted using the YOLOv3 visual basic toolkit and rescaled to a size of 224×224 to obtain a processed global image and three auxiliary images I Aux = {I A1 , I A2 , I A3}; Among them, I A1 is the first auxiliary image, I A2 is the second auxiliary image, I A3 is the third auxiliary image.

[0121] Step S12: extracting hierarchical visual features. Figure 2 As shown, the global image obtained in step S11 and the auxiliary image I Aux Use the pyramid module for encoding. A1 , second auxiliary image I A2 , third auxiliary image I A3 The pyramid module is used to encode the image using the ResNet-based visual encoder. The convolution kernel is used to perform convolution operations. Each image will generate a sequence consisting of four layers of encoding information of different sizes, and the global image encoding sequence P is obtained. I and auxiliary picture coding sequence P Aux = {P A1, P A2 , P A3}.

[0122] Step S2: integrate the hierarchical visual features obtained in step S1 and dynamically generate corresponding prompt vectors. Specifically, the following steps are included:

[0123] Step S21: Dynamically generate feature vectors. Figure 3 As shown, in order to make the global image coding sequence P I and auxiliary picture coding sequence P Aux Through the following calculations, it can be segmented, stacked and reshaped to obtain the processed global prompt vector F I and auxiliary hint vector list F Aux ={F A1 , F A2 , F A3}. Among them, F A1 is the first auxiliary cue vector, F A2 is the second auxiliary hint vector, F A3 is the third auxiliary hint vector. And F I , F A1 , F A2 , F A3 ∈R 20bsz×hz , bsz represents the batch size, and hz represents the hidden layer size.

[0124] In order to calculate the contribution of each auxiliary image to the global image, the global prompt vector F I and the first auxiliary hint vector F A1 Take α as an example to calculate the contribution of the first auxiliary image to the global image. First, the weight score is calculated by the following weight formula:

[0125]

[0126] Among them, the global hint vector F is used I The corresponding query Q and key K are calculated by the weight formula to obtain the global context weight W I ∈R 20bsz×20bsz , using the global hint vector F I The corresponding query Q and the first auxiliary hint vector F A1 The corresponding key K is calculated by the weight formula to obtain the first auxiliary context weight W A1 ∈R 20bsz×20bsz , Softmax is the normalized activation function, d k represents the parameter factor, which prevents large values ​​from causing saturation of the Softmax function, thereby making the weight distribution smoother. The weight score is then applied to the following context formula to obtain the context hint:

[0127] T=W×V

[0128] Among them, the global context weight W is used I and the global hint vector F I The corresponding value V is calculated by the context formula to obtain the global context F I , using the first auxiliary context weight W A1 and the first auxiliary hint vector F A1 The corresponding value V is calculated by the context formula to obtain the first auxiliary context T A1 In order to obtain the context information for adaptive generation, the global context T I and the first auxiliary context T A1 Weighted addition is performed to obtain the first context information:

[0129] FC A1 =F I +αT A1

[0130] Among them, FC A1 ∈R 20bsz×m Represents the first context information obtained, α represents the weight parameter, and m represents the value obtained by dividing the hidden layer size hz by the number of attention heads h.

[0131] The module is trained on a set of global hint vectors F I and the first auxiliary hint vector F A1 Through h times of the above operation, h first context information FC is obtained. A1 , for subsequent processing, all output first context information is linearly integrated together:

[0132]

[0133] Among them, Concat is the concatenation function, W O is the output projection matrix. FCs A1 That is, the first encoded information obtained by integrating all the first context information. Then two normalization steps are performed to ensure that the data distribution of each sample is roughly the same in different feature dimensions. At the same time, the ReLU function is used to introduce nonlinearity to obtain the processed information. The feature vector dynamic allocation module allocates the same set of global prompt vectors F I , the first auxiliary hint vector F A1 Perform the above process four times to get the final first context list, the result is:

[0134] Similarly, the global hint vector F I and the second auxiliary hint vector F A2After the same operation as above, the second context list FCL2 is obtained; the global prompt vector F I and the third auxiliary hint vector F A3 The same operation as above is performed to obtain the third context list FCL3. The three context lists {FCL1, FCL2, FCL3} are integrated to obtain the feature vector F

[0135] F=Stack(FCL1,FCL2,FCL3)

[0136] Stack is a stacking function. The final feature vector F is the final feature vector obtained by integrating the encoding information of each image pair.

[0137] Step S22: Generate a prompt vector containing key-value pairs. Figure 4 As shown, in order to determine the degree of enhancement of the text input sequence by each final context in the feature vector F, a gate value needs to be calculated. The model can dynamically decide whether to activate specific prompt information based on the characteristics of the input data, thereby controlling the degree of influence of these prompt information on the final output. The specific formula of the gate value PG is as follows:

[0138]

[0139] Among them, f img represents each coding layer in the feature vector F. In this embodiment, the feature vector F contains three coding layers, corresponding to three context lists FCL1, FCL2, FCL3, ∑ seq f img Represents along f img The third dimension of Gate idx represents the idxth threshold function, LeakyReLu is an activation function. Each encoding layer f in the feature vector F img The contributions to the final prompt vector are different, and the weight of the contribution of each encoding layer can be controlled by the gating unit.

[0140] The i-th row of the gated value PG is compared with f img Multiply the elements of the i-th column of by element, and then sum them:

[0141] f val =Einsum(′b,bh->bh′,PG[:,i],f img [i])

[0142] Among them, Einsum represents the sum of PG[:, i] and f img [i] Perform element-by-element multiplication and then sum, and calculate four times to get f val1 , f val2 , fval3 , f val4 , and add them together to get the contribution weight:

[0143] F val =f val1 +f val2 +f val3 +f val4

[0144] For each encoding layer f in the feature vector F img The above operation is performed, and then all the contribution weights obtained are val1 , F va2 , F va3} added to the same list:

[0145] f′ p =F val1 ∪F va2 ∪F va3

[0146] f′ p Split, stack and reshape to obtain a hint vector f containing key-value pairs p .

[0147] Step S3: The prompt vector f obtained in step S2 is p The text encoding information is obtained as a hint for text encoding, and then the information of the two modalities of text and vision is subjected to denoising and alignment. The specific steps include:

[0148] Step S31, obtain text and visual information. p As a prefix, given the input text Text, the text is encoded using the BERT (Bidirectional Encoder Representations from Transformers) model, two special tags [CLS] and [SEP] are inserted at the beginning and end of each extracted sentence, respectively, and the processed sentence information is used as the text information seq.

[0149] The global image coding sequence P obtained in step S12 I and auxiliary picture coding sequence P Aux = {P A1 , P A2 , P A3 The top-level visual information is spliced, and then segmented, stacked, and reshaped to obtain the visual information vis.

[0150] Step S32: Denoise the text and visual information. In order to make the gradient passable, the random sampling is converted into a deterministic function plus random noise through the reparameterization technique, thereby allowing the gradient to flow in the sampling operation while maintaining the ability to back propagate the gradient during the sampling process. The latent variable calculation formula is as follows:

[0151] z=μ+e (logσ) ⊙ε

[0152] Where z is the latent variable, μ is the mean, σ is the standard deviation, ⊙ represents the Hadamard product, and ε is the random noise sampled from the standard normal distribution.

[0153] Use the mean μ and standard deviation σ of the text information seq to calculate the text latent variable z through the latent variable calculation formula txt , use the mean μ and standard deviation σ of the visual information vis to calculate the visual latent variable z through the latent variable calculation formula img The reparameterization technique allows controlling the distribution of the latent variable by adjusting the mean and log standard deviation, thereby optimizing the performance of the model while maintaining distribution flexibility.

[0154] Step S33: aligning text and visual information. Since text and visual information are encoded by different encoders, there are differences in the representation of these two modal information. In order to eliminate the modal difference, the encoded information of the two modalities is aligned by positive and negative samples, which specifically includes:

[0155] Positive sample score calculation:

[0156] score pos =Sigmoid(discriminator([sample txt ;sample img ]))

[0157] Negative sample score calculation:

[0158] score neg =Sigmoid(discriminator([sample txt ; negsample img ]))

[0159] Among them, sample txt is the textual latent variable z txt The category-level text latent variable obtained after slicing the second dimension, sample img is the visual latent variable z img The category-level visual latent variable obtained after slicing the second dimension, negsample img is sampleimg The negative sample obtained by rolling, discriminator is the discriminator, Sigmoid is the activation function, the Sigmoid function is used to activate the output of the discriminator, and a probability value between 0 and 1 is obtained as the sample score. The target value of the positive sample is 1, and the target value of the negative sample is 0. The binary cross entropy CrossEntropy loss function is used to calculate the loss between the sample score and the target value:

[0160] loss=CrossEntropy(score;target)

[0161] Among them, score represents the sample score, target represents the target value, and CrossEntropy represents the binary cross entropy. For example, the score of a positive sample is scorepos, and the corresponding target value is the all-1 tensor one.

[0162] Step S4: perform modal fusion on the text information and visual information processed in step S3, and ensure the balance of the two modalities. Specifically, the following steps are included:

[0163] Step S41: Perform multi-modal fusion. Figure 5 As shown in the figure, in order to perform modal fusion and balance the information of the two modalities, the text and visual information are first fused through a multimodal mutual encoder, and the formula for calculating the mutual attention matrix is ​​as follows:

[0164]

[0165] in, is the scaling factor, It is the key generated by the linear layer processing of text information. is the key generated by the linear layer processing of visual information, and T represents transposition. Then, the text features participated by the visual and the visual features participated by the text are calculated. The text information and visual information here come from the text latent variable z obtained in step S32 respectively. txt and the visual latent variable z img .

[0166] The formula for calculating the text features of visual participation is as follows:

[0167]

[0168] The formula for calculating the visual features of text participation is as follows:

[0169]

[0170] in, is the textual latent variable z txt The value generated by the linear layer is is the visual latent variable z img The values ​​generated by the linear layer.

[0171] Finally, the fused text and visual feature information is calculated:

[0172]

[0173] in, Visual features that indicate textual involvement, Indicates the text features of visual involvement, where F mul This is the result of the first fusion. Pass the result here through the linear layer to get the result out1.

[0174] Step S42, encode twice with the encoder. In order to obtain more accurate fusion information, the result here is passed through the linear layer, processed and used for the next multi-head attention encoder, and the output of the last multi-head attention encoder is used for relationship prediction. The formula is as follows:

[0175] First time coding:

[0176] out2 = CrossAttetion(z img ;out1;out1)

[0177] Second encoding:

[0178] final=CrossAttetion(z txt ;out2;out2)

[0179] Among them, CrossAttetion is an encoder designed according to the attention mechanism, out2 represents the output of the first multi-head attention encoder, and final represents the output of the last multi-head attention encoder, which is the final fusion result, and this result is used for relationship prediction.

[0180] Step S5: Use the prediction module to predict the entity relationships in the training set. Specifically, the following steps are included:

[0181] Step S51, extracting the entity hidden state. The final fusion result obtained in step S42 is the final text feature obtained after the cross-modal attention mechanism from text to image. In order to obtain the logical value of the category prediction, these features are usually passed to a classifier, usually a linear layer.

[0182] For each sample i in the final fusion result, use the head index to extract the head entity h i , use the tail index to extract the tail entity t i , and then concatenate them to form the entity hidden state ei :

[0183] e i =[h i ;t i ]

[0184] The hidden state e of each entity i Pass it to the classifier to get the logical value of the category prediction:

[0185] y i =Classifier(e i )

[0186] Among them, Classifier is the classifier, y i It is the logical value of the obtained category prediction, which can also be called the score or raw score of the category prediction.

[0187] Step S52: Predict entity relationships. First, obtain the original score y of each sample through the classifier. i , then for each sample y i Apply the Softmax function and normalize it to a probability distribution:

[0188]

[0189] Among them, y i contains c different categories, y i [c] represents the predicted value of sample i in category c. i [c] is the i-th sample y i The probability of belonging to class c.

[0190] Step S6 and step S5 realize the relationship prediction between entity pairs. In this step, the divergence loss, positive and negative sample loss and prediction loss are calculated respectively, and the training is performed through the stochastic gradient descent method to update the parameter matrix in the model. Specifically, the following steps are included:

[0191] Step S61, calculate the divergence loss function. Divergence loss is usually used to measure the difference between two probability distributions. It provides a way for the model to quantify the difference between the two distributions, which can help the model learn to estimate the target probability distribution more accurately, so that the optimization process can proceed in the direction of reducing this difference. The specific formula of the divergence loss function is as follows:

[0192]

[0193] Where D represents the dimension of the latent variable, μ i and σ i Represents the mean and standard deviation respectively. The divergence loss of the text is calculated by the divergence loss function using the mean and standard deviation of the text information The visual divergence loss is calculated using the mean and standard deviation of the visual information through the divergence loss function

[0194] Step S62: Calculate the positive and negative sample loss functions. The positive and negative sample loss functions provide a learning direction for the model, guiding the model to adjust parameters to minimize the loss, so as to better fit the training data. txt and the visual latent variable z img Processing is performed to obtain the category-level text latent variable sample txt and category-level visual latent variables sample img , and use it to calculate the positive sample scores pos and negative sample scores neg The target value of the positive sample is 1, the target value of the negative sample is 0, and the binary cross entropy loss function is used to calculate the loss between the sample score and the target value.

[0195] First, the positive sample loss function is calculated. The target is a full 1 tensor, which means that the positive sample should be recognized as a correct match by the model. The formula is as follows:

[0196]

[0197] Among them, ones represents a tensor matrix of all 1s.

[0198] Then calculate the negative sample loss function, the target is a full 0 tensor, which means that the negative sample should be recognized by the model as an incorrect match. The formula is as follows:

[0199]

[0200] Among them, scores pos and scores neg Represents the model prediction scores of positive samples and negative samples respectively, num neg Represents the number of negative samples, and zeros represents a tensor matrix of all 0s.

[0201] Positive and negative sample loss:

[0202]

[0203] Adding the positive sample loss and the negative sample loss together gives the positive and negative sample loss.

[0204] Step S63: Calculate the prediction loss function. A batch contains N samples, each sample has c possible categories. i ∈R Cis the logical value of the category prediction of the i-th sample, and the cross entropy loss function can be further expressed as:

[0205]

[0206] Among them, y i There are c different categories included.

[0207] Step S64: After obtaining the divergence loss, positive and negative sample loss, and prediction loss in steps S61, S62, and S63, this step combines all loss functions to achieve joint training and obtain the final loss of the model. The formula is as follows:

[0208]

[0209] Among them, β1, β2 and β3 are the weight parameters of the loss function, and Represent the divergence loss of text and vision respectively, and minimize the final loss by stochastic gradient descent method To train multimodal relationship extraction models for better relationship prediction.

[0210] Step S7: After completing the model training in step S6, the model can be used for actual multimodal relationship extraction tasks. For example, for a text-image pair Text and Img, it is necessary to predict the relationship type between two entities in Text. Figure 6 As shown, first, it is necessary to extract the feature information of text and image, perform denoising and alignment, and fuse the processed modal information. Then, similar to steps S51 and S52, the logical value of the category prediction is obtained, and the original score y of each sample is obtained through the classifier. i , then for each sample y i Apply the Softmax function and normalize it to a probability distribution to get the probability p i [c]. Sort the probabilities and take the prediction result with the highest probability as the final predicted relationship type.

[0211] It is to be understood that the present invention is described by some embodiments, and it is known to those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the scope of protection of the present invention.

Claims

1. An adaptive multimodal relation extraction method based on mutual attention mechanism, It is characterized in that The following steps are involved: A. Training multimodal relationship extraction model: A1. Use the pyramid module to extract hierarchical visual features from the image to be processed, and use each layer of visual features as prompt information for text encoding; at the same time, process the top layer of visual features separately to form another type of encoding information, and fuse it with the text information in the fusion stage; A2. Process each layer of visual features through a feature vector dynamic allocation module to obtain a feature vector; then, extract a prompt vector containing a key-value pair from the obtained feature vector through a key-value pair generation module; A3: De-noise the top-level visual feature information and text encoding information, and align the encoding information of the two modalities of text and image through positive and negative samples; A4, fusing the aligned two modal information through a multimodal mutual fusion module to obtain coding information for prediction; A5. Use the prediction module to predict the entity relationships in the training set; A6. Calculate the loss function and sum it proportionally to get the final loss value of the model, and train the multimodal relationship extraction model by stochastic gradient descent. B. Multimodal relationship extraction task: The trained relation extraction model is used in the actual multimodal relation extraction task to generate the most likely relation type for entity pairs in the text based on visual cues.

2. The adaptive multimodal relationship extraction method based on mutual attention mechanism as claimed in claim 1, It is characterized in that In step A1, extracting hierarchical visual features using a pyramid module specifically includes: The global image and the corresponding auxiliary image in the data set are cropped to obtain the cropped global image I and auxiliary image I Aux = {I A1 ,…,I Aj ,...,I An }; where n represents the number of auxiliary images, j∈[1,n]; Based on ResNet, a pyramid module is used to transform the global image I and the auxiliary image I Aux Encode and generate the corresponding global image coding sequence P I and auxiliary picture coding sequence P Aux = {P A1 ,…,P Aj ,...,P An }.

3. The adaptive multimodal relationship extraction method based on mutual attention mechanism as claimed in claim 1, It is characterized in that In step A2, the visual features of each layer are processed by the feature vector dynamic allocation module, specifically including: For the global image coding sequence P obtained in A1 I and auxiliary picture coding sequence P Aux Perform segmentation, stacking and reshaping to obtain the corresponding global prompt vector F I and auxiliary hint vector list F Aux ={F A1 ,…,F Aj ,...,F An }; The weight score is calculated using the multi-head attention mechanism weight formula: Among them, the global hint vector F is used I The corresponding query Q and key K are calculated by the weight formula to obtain the global context weight W I , using the global hint vector F I The corresponding query Q and the j-th auxiliary hint vector F Aj The corresponding key K is used to calculate the jth auxiliary context weight W A , Softmax is the normalized activation function, d k represents the parameter factor, The weight score is calculated through the following contextual formula to calculate the contextual hint: T=W×V Among them, the global context weight W is used I and the global hint vector F I The corresponding value V is calculated by the context formula to obtain the global context T I , using the jth auxiliary context weight W A and the jth auxiliary cue vector F Aj The corresponding value V is calculated by the context formula to obtain the jth auxiliary context T A ; Add the global context to the jth auxiliary context to obtain the adaptively generated context information: FC=T I +αT A Among them, FC represents the jth context information obtained, and α represents the weight parameter; Module for each group F I and F Aj Through h times of the above operation, h context information FC is obtained. All context information FC are linearly integrated, and h represents the number of attention heads: FCs=Concat(FC 1 ,FC 2 ,...,FC h )W O Among them, Concat is the concatenation function, W O is the output projection matrix; then two normalization steps are performed to ensure that the data distribution of each sample is roughly the same in different feature dimensions, and the ReLU function is used to introduce nonlinearity to obtain the processed information; The feature vector dynamic allocation module processes the same pair of image coding sequences four times to obtain a list of the final context after processing, the result of which is FCL j ={FCs 1 , FCs 2 , FCs 3 , FCs 4 }, combine all the lists obtained: F=Stack(FCL 1 ,FCL 2 ,...,FCL n ) Among them, Stack is a stacking function; the final feature vector F is the final feature vector obtained by integrating the encoding information of each image pair.

4. The adaptive multimodal relationship extraction method based on mutual attention mechanism as claimed in claim 3, It is characterized in that In step A2, extracting a prompt vector containing a key-value pair from the obtained feature vector through a key-value pair generation module specifically includes: By calculating a gate value, the model dynamically decides whether to activate specific prompt information based on the characteristics of the input data, thereby controlling the degree of influence of these prompt information on the final output. The specific formula of the gate value PG is as follows: Among them, f img Represents each encoding layer in the feature vector F, corresponding to n context lists FCL 1 , FCL 2 ,...,FCL n ;∑ seq f img Represents along f img The third dimension of Gate idx represents the idxth threshold function; each coding layer f in the feature vector F img The contribution to the final hint vector is different. The weight of the contribution of each coding layer can be controlled by the gating unit, which connects the i-th row of PG with f img Multiply the elements of the i-th column element by element and then sum them; f val =Einsum(′b,bh->bh′,PG[:,i],f img [i]) Among them, Einsum represents the sum of PG[:,i] and f img [i] Perform element-by-element multiplication and then sum, and calculate four times to get f val1 , f val2 , f val3 , f val4 , and add them together to get the contribution weight: F val =f val1 +f val2 +f val3 +f val4 For each encoding layer f in the feature vector F img The above operations are performed, and then all the resulting contribution weights are added to the same list, split, stacked, and reshaped to obtain a prompt vector containing key-value pairs.

5. The adaptive multimodal relationship extraction method based on mutual attention mechanism as claimed in claim 1, It is characterized in that In step A3, the top-level visual feature information and text encoding information are subjected to noise reduction processing, specifically including: Using the key-value pair hint vector obtained in A2 as an aid, use BERT to encode the text, insert two special tags [CLS] and [SEP] at the beginning and end of each extracted sentence, and use the processed sentence information as the text information seq; The global image coding sequence P obtained in A1 I and auxiliary picture coding sequence P Aux The top-level visual information is spliced, and then segmented, stacked, and reshaped to obtain the visual information vis; In order to make the gradient pass, the random sampling is transformed into a deterministic function plus random noise through the reparameterization trick, allowing the gradient to flow in the sampling operation while maintaining the ability to backpropagate the gradient during the sampling process. The latent variable calculation formula is as follows: from=μ+e (logσ) ⊙ε Where z is the latent variable, μ is the mean, σ is the standard deviation, ⊙ represents the Hadamard product, and ε is the random noise sampled from the standard normal distribution; The latent variable z of the text is calculated using the mean μ and standard deviation σ of the text information through the latent variable calculation formula txt , use the mean μ and standard deviation σ of the visual information to calculate the visual latent variable z through the latent variable calculation formula img .

6. The adaptive multimodal relationship extraction method based on mutual attention mechanism as claimed in claim 5, It is characterized in that In step A3, the aligning of the encoded information of the two modes by means of positive and negative samples specifically includes: Positive sample score calculation: score pos =Sigmoid(discriminator([sample txt ;sample img ]) Negative sample score calculation: score neg =Sigmoid(discriminator([sample txt ;negsample img ])) Among them, sample txt is the textual latent variable z txt The category-level text latent variable obtained after slicing the second dimension, sample img is the visual latent variable z img Category-level visual latent variables obtained after slicing along the second dimension, negsample img is sample img The negative sample obtained after rolling, discriminator is the discriminator; Sigmoid is the activation function.

7. The adaptive multimodal relationship extraction method based on mutual attention mechanism as claimed in claim 1, It is characterized in that In step A4, the aligned two modal information are fused through a multimodal mutual fusion module, specifically including: First, the text and visual information are initially fused through a multimodal mutual encoder, and the formula for calculating the mutual attention matrix is ​​as follows: in, is the scaling factor, is the textual latent variable z txt The key generated by the linear layer, is the visual latent variable z img The key generated by the linear layer processing, T represents the transpose; then, the text features participated by the visual and the visual features participated by the text are calculated; The formula for calculating the text features of visual participation is as follows: The formula for calculating the visual features of text participation is as follows: in, is the textual latent variable z txt The value generated by the linear layer is is the visual latent variable z img The value generated by the linear layer; Finally, the fused text and visual feature information is calculated: Here we get F mul This is the result of the first fusion. The result here is passed through a linear layer and processed for the next multi-head attention encoder, and the output of the last multi-head attention encoder is used for relationship prediction.

8. The adaptive multimodal relationship extraction method based on mutual attention mechanism as claimed in claim 1, It is characterized in that In step A5, the prediction module is used to predict the entity relationships in the training set, specifically including: First, the original score y of each sample is obtained through the classifier i , then for each sample y i Apply the Softmax function and normalize it to a probability distribution: Among them, y i contains c different categories, y i [c] represents the predicted value of sample i in category c; p i [c] is the i-th sample y i The probability of belonging to class c.

9. The adaptive multimodal relationship extraction method based on mutual attention mechanism according to any one of claims 1 to 8, It is characterized in that Step A6 specifically includes: A61. Calculate the divergence loss function: Where D represents the dimension of the latent variable, μ i and σ i Represent the mean and standard deviation respectively; the divergence loss of the text is calculated by the divergence loss function using the mean and standard deviation of the text information The visual divergence loss is calculated using the mean and standard deviation of the visual information through the divergence loss function A62. Calculate the loss function of positive and negative samples: First, the positive sample loss function is calculated. The target is a full 1 tensor, which means that the positive sample should be recognized as a correct match by the model. The formula is as follows: Then calculate the negative sample loss function, the target is a full 0 tensor, which means that the negative sample should be recognized by the model as an incorrect match. The formula is as follows: Among them, scores pos and scores neg Represents the model prediction scores of positive samples and negative samples respectively, num neg Represents the number of negative samples, ones represents a tensor matrix of all 1s, zeros represents a tensor matrix of all 0s, and CrossEntropy is the binary cross entropy; Positive and negative sample loss: Add the two obtained losses together to get the total loss; A63. Calculate the prediction loss function: Among them, y i contains c different categories, N is the number of samples, y i [c] represents the predicted value of sample i in category c; A64. Combine all the above loss functions to obtain the final loss of the model And minimize the final loss by stochastic gradient descent To train the multimodal relationship extraction model, the formula is as follows: Among them, β 1 , β 2 and β 3 is the weight parameter of the loss function, and denote the divergence loss of text and vision respectively.

Images