Multi-modal relation extraction method and device under non-paired data and electronic equipment

Abstract

The invention discloses a multi-modal relation extraction method, and particularly relates to a multi-modal relation extraction method and device under non-paired data and electronic equipment. The method comprises the following steps: performing feature extraction on image modal data to obtain a plurality of different levels of visual features, and performing feature extraction on text modal data to obtain final layer text features; fusing the visual features with the final layer text features based on a cross attention mechanism to obtain hierarchical visual features corresponding to the visual features; carrying out fusion processing on the hierarchical visual features and the final layer text features to obtain multi-modal features corresponding to the hierarchical visual features; aggregating the plurality of hierarchical visual features, the plurality of multi-modal features and the final layer text feature to obtain a multi-modal fusion feature; and inputting the multi-modal fusion features into a prediction module for relation extraction to obtain a relation extraction result. According to the method, the accuracy of the multi-modal relation extraction result of the non-paired data can be improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of multimodal relationship extraction methods, and specifically relates to a multimodal relationship extraction method, device and electronic equipment under unpaired data. Background Art

[0002] As social media evolves dynamically in form and concept, the exponential growth of user data highlights the importance of multimodal knowledge graphs. By integrating data from various modalities such as text, images, and audio, these graphs provide powerful support for rich information representation and intelligent analysis. In this context, multimodal relationship extraction has become an important subtask, which aims to identify and extract meaningful semantic relationships from heterogeneous data, thereby facilitating the construction of more comprehensive and precise knowledge networks. Effective execution of this task can not only improve the accuracy and completeness of knowledge graphs, but also enhance their ability to intelligently process complex information networks. With the advancement of deep learning, multimodal relationship extraction has achieved good results on structured data and has achieved remarkable success in fields such as data mining, visual question answering, and information retrieval.

[0003] However, despite these significant advances, almost all multimodal relation extraction works focus on improving model accuracy under perfect data configuration, assuming that the input text and image pairs match perfectly. This assumption has obvious limitations in practical applications: on the one hand, the visual and textual data provided by users on social media often inherently contain mismatches (for example, images expressing personal emotions are inconsistent with the semantics of the text); on the other hand, due to factors such as inaccurate data collection, images paired with text may be of low quality or fail to provide relevant information. Therefore, the above methods commonly used in the prior art have poor accuracy in extracting multimodal relations from non-matching multimodal data. Summary of the invention

[0004] The technical problem to be solved by the present invention is that the above-mentioned method commonly used in the prior art has poor accuracy when extracting multimodal relationships from unmatched multimodal data. In order to solve the above-mentioned problem, the present invention provides a method, device and electronic device for extracting multimodal relationships under unpaired data.

[0005] The content of the present invention includes: In a first aspect, an embodiment of the present invention provides a method for extracting multimodal relationships under unpaired data, comprising: Perform feature extraction on image modality data to obtain visual features at multiple different levels, and perform feature extraction on text modality data to obtain final-level text features; For each visual feature of the multiple different levels of visual features, the visual feature is fused with the final layer text feature based on a cross attention mechanism to obtain a hierarchical visual feature corresponding to the visual feature; For each of the hierarchical visual features, the hierarchical visual features are fused with the final layer text features to obtain a multimodal feature corresponding to the hierarchical visual feature; Aggregating the plurality of the hierarchical visual features, the plurality of the multimodal features and the final layer text features to obtain a multimodal fusion feature; The multimodal fusion features are input into a prediction module to perform relationship extraction to obtain a relationship extraction result.

[0006] Optionally, the feature extraction of the image modality data to obtain visual features at multiple different levels includes: After preprocessing the image modal data to obtain visual features at the 0th level, the visual features at the 0th level are input into a plurality of image encoding modules connected in sequence to obtain visual features at a plurality of different levels, wherein each of the image encoding modules is used to output the visual features at one level.

[0007] Optionally, the fusing the visual features with the final-layer text features based on a cross-attention mechanism to obtain a hierarchical visual feature corresponding to the visual feature includes: Taking the visual features as key-value pairs and the final layer text features as queries, a plurality of attention heads are calculated; The hierarchical visual feature corresponding to the visual feature is calculated based on the multiple attention heads.

[0008] Optionally, the mth attention head for: ; ; in, is the first of the multiple visual features at different levels Hierarchical visual features, , is the number of image coding modules, is the final layer text feature, Used to represent the Softmax function, is the attention score, is the weight matrix used to generate the value vector, is the weight matrix used to generate the query vector, is the weight matrix used to generate the key vector, M is the number of attention heads, is the input feature dimension.

[0009] Optionally, the aggregating the plurality of the hierarchical visual features, the plurality of the multimodal features and the final layer text features to obtain a multimodal fusion feature includes: Calculating an aggregation weight based on a plurality of the hierarchical visual features and the final layer text features; The plurality of hierarchical visual features, the plurality of multimodal features and the final layer text features are aggregated based on the aggregation weights to obtain the multimodal fusion features.

[0010] Optionally, the aggregation weight for: ; in, is the pre-trained projection matrix, is the number of visual features, is the final layer text feature, , For the The hierarchical visual features, , Used to represent the Softmax function.

[0011] Optionally, the prediction module includes a prediction classification head, and the step of inputting the multimodal fusion features into the prediction module for relationship extraction to obtain a relationship extraction result includes: Inputting the multimodal fusion features into the prediction classification head for classification prediction to obtain a predicted multivariate logical value; A probability prediction result is determined based on the predicted multivariate logical value, and the probability prediction result is used to characterize the relationship extraction result.

[0012] In a second aspect, an embodiment of the present invention further provides a multimodal relationship extraction device under unpaired data, comprising: The feature extraction module is used to extract features from image modality data to obtain visual features at multiple different levels, and to extract features from text modality data to obtain final-level text features; An attention fusion module, for fusing the visual features of each level of the multiple different levels of visual features with the final level text features based on a cross attention mechanism to obtain a hierarchical visual feature corresponding to the visual feature; A multi-level fusion module, for fusing each of the hierarchical visual features with the final-level text feature to obtain a multimodal feature corresponding to the hierarchical visual feature; An aggregation module, used for aggregating the plurality of hierarchical visual features, the plurality of multimodal features and the final layer text features to obtain a multimodal fusion feature; The relationship extraction module is used to input the multimodal fusion features into the prediction module to perform relationship extraction and obtain a relationship extraction result.

[0013] In a third aspect, an embodiment of the present invention provides an electronic device, comprising: a memory, a processor, and a program stored in the memory and executable on the processor; the processor is configured to read the program in the memory to implement the steps in the multimodal relationship extraction method under unpaired data as described in the first aspect.

[0014] In a fourth aspect, an embodiment of the present invention provides a readable storage medium for storing a program, which, when executed by a processor, implements the steps in the multimodal relationship extraction method under unpaired data as described in the first aspect.

[0015] The beneficial effect of the present invention is that the embodiment of the present invention provides a multimodal relationship extraction method under unpaired data, performs feature extraction on image modal data to obtain multiple visual features at different levels, and performs feature extraction on text modal data to obtain final-level text features; based on the cross-attention mechanism, the visual features are fused with the final-level text features to obtain hierarchical visual features corresponding to the visual features; the hierarchical visual features are fused with the final-level text features to obtain multimodal features corresponding to the hierarchical visual features; multiple hierarchical visual features, multiple multimodal features and final-level text features are aggregated to obtain multimodal fusion features, and the multimodal fusion features are input into the prediction module for relationship extraction to obtain relationship extraction results. Through the above method, the fusion of multi-level visual features and text features is realized, ensuring the balanced feature representation between visual and text modalities, thereby improving the accuracy of the extraction results of multimodal relationship extraction, making it more suitable for diverse and dynamic conditions in open world environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Attached Figure 1 A flowchart of a method for extracting multimodal relationships from unpaired data provided by an embodiment of the present invention; Attached Figure 2 A schematic diagram of an implementation framework of a method for extracting multimodal relationships from unpaired data provided by an embodiment of the present invention; Attached Figure 3 A schematic diagram of a multimodal relationship extraction device under unpaired data provided by an embodiment of the present invention; Attached Figure 4 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0017] In the embodiments of the present application, the term "and / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. In the embodiments of the present application, the term "multiple" refers to two or more, and other quantifiers are similar. The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first" and "second" are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more.

[0018] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0020] See also Figure 1 , Figure 1 The present invention provides a flow chart of a method for extracting multimodal relationships from unpaired data. The method specifically includes the following steps: Step 101 , extracting features from image modal data to obtain visual features at multiple different levels, and extracting features from text modal data to obtain final-level text features.

[0021] Step 102: for each level of visual features in the multiple different levels of visual features, the visual features are fused with the final level text features based on a cross-attention mechanism to obtain a hierarchical visual feature corresponding to the visual feature.

[0022] Step 103: For each of the hierarchical visual features, the hierarchical visual feature is fused with the final layer text feature to obtain a multimodal feature corresponding to the hierarchical visual feature.

[0023] Step 104 , aggregating the plurality of the hierarchical visual features, the plurality of the multimodal features and the final layer text features to obtain a multimodal fusion feature.

[0024] Step 105: input the multimodal fusion features into a prediction module to perform relationship extraction to obtain a relationship extraction result.

[0025] In step 101, different encoders are used to extract features from data of different modalities. In specific implementation, the specific structures of the text encoder and the image encoder can be set and adjusted according to actual needs, and are not limited here.

[0026] The embodiment of the present invention also provides a multimodal relationship extraction model under unpaired data, which can be used to implement the various processes of the multimodal relationship extraction method under unpaired data. In some embodiments, the method further includes: The multimodal relationship extraction model under unpaired data is iteratively trained based on the training data set. The multimodal relationship extraction model under unpaired data can be used to implement the above steps 101 to 105. The multimodal relationship extraction model under unpaired data includes: A text encoder, wherein the text encoder is used to extract features from text modality data to obtain final-layer text features; An image encoder, wherein the image encoder is used to extract features from the image modality data to obtain visual features at multiple different levels; A cross attention module, wherein the cross attention module is used to fuse the visual features with the final layer text features based on a cross attention mechanism to obtain a hierarchical visual feature corresponding to the visual feature; A fusion layer, wherein the fusion layer is used to fuse the hierarchical visual features with the final layer text features to obtain multimodal features corresponding to the hierarchical visual features; An aggregation layer, the aggregation layer is used to aggregate the plurality of the hierarchical visual features, the plurality of the multimodal features and the final layer text features to obtain a multimodal fusion feature; A prediction module is used to perform relationship extraction on the multimodal fusion features to obtain a relationship extraction result.

[0027] Optionally, as a specific embodiment, a text encoder constructed based on a bidirectional encoder representation transformer (Bidirectional Encoder Representations from Transformers, BERT) is used to encode text modality data. Figure 2In this embodiment, the specific process of extracting features from text modal data to obtain the final layer text features is as follows: The text modality data to be processed includes sentences When processing text modality data, we first use the BERTWordpiece word analyzer to analyze the sentence Tokenize and add special tokens (such as ) to indicate the start and end of the subject entity and object entity. In some embodiments, by adding special tokens at the beginning of the sequence , and pad the end of the sequence to the maximum sequence length using special tokens , to align the lemma sequence with the BERT encoding process. The above process can be expressed as follows: ; ; in, is the tokenized subsequence of the subject entity, is the tokenized subsequence of the object entity, Used to represent the processing operation of BERT. BERT converts each word into Processed into context vector representation, Indicates The output embedding matrix of layer BERT, for Length, Text features The dimension of is the maximum number of BERT layers used by the text encoder. In this embodiment, text features at different levels can be obtained, and the text features output by the last layer are Determined as the final layer text feature.

[0028] Optionally, in some embodiments, the feature extraction of the image modality data to obtain multiple visual features at different levels includes: After preprocessing the image modal data to obtain visual features at the 0th level, the visual features at the 0th level are input into a plurality of image encoding modules connected in sequence to obtain visual features at a plurality of different levels, wherein each of the image encoding modules is used to output the visual features at one level.

[0029] This embodiment uses multiple sequentially connected ViT modules to encode image modality data. ViT is a visual model based on the Transformer architecture, which is mainly used for computer vision tasks (such as image classification, target detection, etc.). In this embodiment, the specific process of extracting features from image modality data to obtain visual features at multiple different levels is as follows: See also Figure 2 , when processing image modality data, the input image modality data It is first resized to a specific dimension and then split into non-overlapping two-dimensional image patches, denoted as Similarly, in some embodiments, special words Added to sequence to indicate image modality data Each patch is then mapped into a one-dimensional feature to fit the input dimension of ViT, thereby obtaining the visual feature at level 0 As the initial input, the visual features at level 0 pass The pre-trained ViT module (i.e., image encoding module) is processed Different levels of visual features, the above process can be described as follows: ; in, Indicates The output embedding matrix of layer ViT is is the maximum number of ViT layers used by the image encoder, that is, the number of image coding modules, for Length, is the characteristic dimension of the visual feature, It is used to represent the Vision Transformer encoder. In this embodiment, the maximum number of layers of ViT is the image coding module (i.e. Figure 2 The number of ViT modules in the .

[0030] After obtaining the visual features of different levels and the final level text features, in step 102 of this embodiment, the visual features from different levels and the final level text features are combined based on the cross attention mechanism. Optionally, in some embodiments, step 102 includes: Taking the visual features as key-value pairs and the final layer text features as queries, a plurality of attention heads are calculated; The hierarchical visual feature corresponding to the visual feature is calculated based on the multiple attention heads.

[0031] Specifically, this embodiment includes multiple cross-attention modules, such as Figure 2 As shown, for the For the cross-attention module corresponding to the layered visual features, the cross-attention module uses the final layer text features As a query, use Visual features extracted by layers As key-value pairs, the cross-attention module can learn visual representation information at different levels.

[0032] Optionally, in some embodiments, the mth attention head for: ; ; ; In the above formula, It is the first of multiple different levels of visual features. Hierarchical visual features, For the Hierarchical visual features correspond to hierarchical visual features, , is the number of image coding modules, is the final layer text feature, Used to represent the Softmax function, is the attention score, is the weight matrix used to generate the value vector, is the weight matrix used to generate the query vector, is the weight matrix used to generate the key vector, is the output transformation matrix, M is the number of attention heads, is the feature dimension, Used to represent a regularization layer.

[0033] In step 103, for each hierarchical visual feature, the hierarchical visual feature is fused with the final layer text feature to obtain a multimodal feature corresponding to the hierarchical visual feature. Through the above operation, for each hierarchical visual feature, its corresponding multimodal feature can be calculated.

[0034] Optionally, in some embodiments, Hierarchical visual features corresponding to hierarchical visual features Corresponding multimodal features for: ; in, , represents the linear projection matrix, is the activation function used in BERT, is the internal bias term, For the Layer external bias term, For the The fully connected layer in the layer.

[0035] Through the above method, we can get A collection of hierarchical visual features and A collection of multimodal features : ; .

[0036] In order to capture more data caused by the mismatch between different modal data contents In this embodiment, we hope to obtain a multimodal feature that integrates multi-level information rather than a multimodal feature based on text features. Therefore, it is necessary to aggregate the feature representations of different views. In the prior art, traditional static aggregation methods use a static weight to obtain a relatively stable weight representation. Such methods basically assume that the quality or importance of features at different levels is relatively stable in all samples. However, in the extraction of multimodal relationships in the open world, this assumption is not feasible.

[0037] Optionally, in some embodiments, step 104 includes: Calculating an aggregation weight based on a plurality of the hierarchical visual features and the final layer text features; The plurality of hierarchical visual features, the plurality of multimodal features and the final layer text features are aggregated based on the aggregation weights to obtain the multimodal fusion features.

[0038] In this embodiment, the aggregation weight is adaptively calculated based on the information of each level and the contextual relevance of the text to obtain the final multimodal fusion feature. for: ; in, is the pre-trained projection matrix, , is the number of visual features, is the final layer text feature, is a collection of hierarchical visual features. , For the The hierarchical visual features, , The operation used to characterize the Softmax function.

[0039] It should be understood that It is a trainable projection matrix used to calculate the weights of different multimodal representations when they are aggregated. Since text modality data is dominant in the multimodal relationship extraction task, the final layer of text modality data is specially introduced when fusing multimodal features to ensure that the model can use enough key information.

[0040] Based on the aggregate weight Hierarchical visual features, The multimodal features are aggregated with the final layer text features to obtain the multimodal fusion features. The specific process of representation can be described as follows: ; in, Used to represent the text feature aggregation results. Used to indicate the The hierarchical visual feature aggregation results are The result used to represent the first dimension of the aggregation result. The first Dimensional results.

[0041] In the process of model training, in order to solve the inherent cross-modal semantic gap, a contrastive learning loss is set in this embodiment, so as to integrate contrastive learning into the model training process. The operating principle of contrastive learning is to bring semantically similar samples closer in the latent space and push dissimilar samples farther away. This process enhances the model's ability to identify and align semantically equivalent concepts across different types of data. The specific implementation is as follows: For each training batch of data , is the size of the batch data. After inputting the model, the output multimodal fusion feature set is . According to the contrastive learning loss obtained by F for: ; ; in, is the indicative function, when 1 if yes, 0 otherwise. is the multimodal fusion feature corresponding to the i-th sample in the batch data, is the multimodal fusion feature corresponding to the jth sample in the batch data.

[0042] To achieve credible multimodal relation extraction, uncertainty needs to be modeled based on the matching of the two modal data, which often exists at multiple scales. This variability poses a challenge in achieving a balanced representation of relations across modalities. Existing techniques mainly focus on the visual prominence of entities, which may lead to an overemphasis on entities with larger spatial domains and an underestimation of entities with smaller spatial domains, which may lead to biased interpretations and unbalanced understanding of entity relations, thereby distorting the uncertainty of multimodal data.

[0043] The prior art usually only performs fusion encoding on high-level features of vision and text (i.e., the final features output by the encoder). This fusion encoding method is difficult to fully reflect the matching relationship between visual features and text features in different views. In addition, the prior art finally uses the fused text features as the features of the input classification head, overemphasizing the use of text information. These designs make the baseline unbalanced in capturing information from different modalities and utilizing information from different views in the multimodal relationship extraction task. Unlike the prior art, in this embodiment, the balance of multimodal data information between different modalities is fully considered to construct a robust and balanced multimodal feature to ensure that entities with different spatial ranges are accurately modeled and their relationships are clearly depicted. This robust feature representation is crucial for modeling the uncertainty of heteroscedastic data because it enables the model to accurately interpret and integrate data from different modalities and reflect the matching of data from different modalities.

[0044] In step 105, relationship extraction is performed based on the multimodal fusion features to obtain a relationship extraction result. Optionally, in some embodiments, the prediction module includes a prediction classification head, and step 105 includes: Inputting the multimodal fusion features into the prediction classification head for classification prediction to obtain a predicted multivariate logical value; A probability prediction result is determined based on the predicted multivariate logical value, and the probability prediction result is used to characterize the relationship extraction result.

[0045] Specifically, the multimodal fusion features Input the prediction classification head to obtain the predicted multivariate logistic value In this embodiment, in order to fully explore the reliable information in the data and model the heteroscedastic uncertainty, the prediction score Gaussian distribution modeling is performed. Each dimension of is modified from a deterministic element to a distribution to improve the robustness of the prediction results. The prediction of a sample can be expressed as follows: .

[0046] Predicting multivariate logistic values Perform Softmax operation to obtain probability prediction results , and use it as the output result during reasoning to obtain the relationship extraction result: .

[0047] in, For with its parameters The trainable MLP, i.e. the prediction classification head, is another with its parameters A trainable MLP of , called a variance predictor, is a diagonal matrix, and the logarithm of the square difference of each diagonal value corresponds to An element of . Vector The variance is The Gaussian noise interference is used to operate the interfered vector through the Softmax function to obtain the final output.

[0048] Specifically, the multimodal fusion features Input prediction classification head Make a prediction and get the prediction result : ; In order to model the uncertainty of the input heteroscedastic data, the multimodal fusion features are input into the variance predictor Uncertainty in the data obtained : .

[0049] In this embodiment, heteroscedastic uncertainty modeling of multimodal fusion features not only provides a probabilistic understanding of data variability, but also enhances the model's ability to make wise decisions under uncertainty. This approach ensures that the model can more effectively identify and prioritize the most relevant and reliable information, significantly improving its ability to handle unpaired or mismatched data scenarios.

[0050] According to the above content, the expected log-likelihood loss function of the model can be written as: ; in, Represents the data The corresponding relationship category label, Representation data The corresponding relationship is . Ideally, we would like to obtain the analytical result of the above function through the integral function of Gaussian distribution. However, there is no method to achieve this. Therefore, in this embodiment, Monte Carlo integration is used to approximate the target, and the unary terms are sampled through the Softmax function. This operation only requires one calculation (passing the input to the model to obtain logits), and the remaining calculations only involve sampling from logits, which only accounts for a small part of the network calculation, so it does not significantly increase the test time of the model. During the training of the model, the following numerically stable random loss can be obtained in the above manner: : ; ; In the above formula, Indicates the number of sampling times, represents the t-th sampling variable, represents category c, Represents the sampling result of the sample corresponding to category c, represents the output corresponding to category c, Represents the identity matrix.

[0051] In order to prevent the model from experiencing distribution collapse (i.e. approaches 0), in this embodiment, a regularization loss is set : .

[0052] in, Used to indicate Divergence calculation. It should be understood that if Figure 2 As shown, by is the mean, The prediction result obtained by sampling the normal distribution of the variance is used to calculate the uncertainty. As a specific embodiment, during the model training process, the uncertainty loss constructed to model the heteroscedastic uncertainty is Including regularization loss and random loss : ; in, is the weight of the regularization loss.

[0053] Optionally, in some embodiments, the method further comprises: Based on the training data set, the multimodal relationship extraction model under unpaired data is iteratively trained. The loss value of iterative training Including contrastive learning loss , uncertainty loss and prediction task loss , loss value for: ; in, for The corresponding preset weights, for Corresponding preset weights. Contrasting learning loss and uncertainty loss The specific calculation method of can be found in the above content, which will not be repeated here. The following is an explanation of the prediction task loss.

[0054] After obtaining the multimodal fusion features, the multimodal fusion features are input into the prediction classification head Get the prediction results : ; in, To predict the logits output by the classification head, The output of the softmax function can be used as a probability prediction result : ; We can further get the prediction task loss for: ; in, is the relation category label corresponding to the input data, Used to represent the softmax function, Used to represent the cross entropy loss function.

[0055] In this embodiment, multi-view contrastive learning is combined to capture the matching information between different modal data, a self-gated fusion mechanism is used to avoid modal bias and balance the use of information between different views, and uncertainty modeling is used to enhance the model's ability to handle noise between modal data, thereby improving its generalization performance in the open world.

[0056] The following is an experiment to illustrate the beneficial effects of the multimodal relationship extraction method under unpaired data provided by the embodiment of the present invention. In order to evaluate the performance of the method on the trusted multimodal relationship extraction task, the experiment uses the most widely used multimodal relationship extraction dataset (MNRE). Each sample in MNRE includes the text content of a post captured from Twitter and its corresponding image. Given an image, the visual localization toolkit is used to obtain the image with the highest In order to obtain a test dataset that meets the requirements of the credible multimodal relationship extraction task, the The test set of the MNRE dataset is randomly selected for image and text shuffling, retaining the correct text and relationship and replacing the original matching image. .

[0057] The following are the baseline methods used in this experiment: (1) IFAformer: Aligns cross-modal features of text and images by leveraging an attention mechanism based on textual and visual prefixes.

[0058] (2) HVPNeT: Leveraging multi-scale visual features extracted from 4 ResNet blocks to create informative multimodal representations, HVPNet extracts fixed hierarchical visual representations for all text tokens and focuses on cross-level aggregation.

[0059] (3) MKGformer: proposed a hybrid Transformer to integrate visual and textual representations.

[0060] (4) HVFormer: adopts a MoE paradigm to align multimodal features of different scales.

[0061] We use ViTB / 32 as the backbone of the image encoder and pre-trained BERT-base as the text encoder for experiments. The two models have similar architectures, with a hidden dimension of 768, 12 attention heads and 12 layers. At the same time, considering the computational overhead and the small difference in the features encoded in adjacent layers of ViT, in this experiment, only the visual features of the 0th, 6th, and 12th layers of ViT are processed. The experiment uses AdamW to optimize the model with an initial learning rate of , train for 20 epochs.

[0062] Comparison of performance indicators of models trained on paired and unpaired data, the performance of models trained under different pairing conditions, especially under 0% and 50% mismatch ratios, is compared to evaluate their robustness in handling paired and unpaired data scenarios. Performance indicators include accuracy (ACC), precision (P), recall (R), and F1 score (F1), which are measured under 0% and 50% matching test conditions. The specific experimental results are shown in Table 1: Table 1 Experimental results under mismatch ratios of 0% and 50%

[0063] By comparing each method on the MNRE dataset, there are different proportions When there are mismatched samples, the accuracy (Acc), precision (P), recall (R) and F1 score (F1) on the matching samples (match), mismatched samples (mismatch) and all samples (total) are reported. The results of the method provided in this embodiment on the clean test set are also reported. Table 2 summarizes the experimental results of the baseline method and the method proposed in this example.

[0064] According to the experimental results, the method provided by this embodiment is superior to all baseline methods and demonstrates its overall superior performance. Specifically, compared with the baseline method, the method proposed in this embodiment is superior in both the task of extracting credible multimodal relations and the task of extracting credible multimodal relations. Still in the multimodal relationship extraction task , there are relatively stable improvements in accuracy, recall, precision and F1 score, especially on mismatched samples. This shows that the data uncertainty modeling we proposed can improve the robustness of the model to noise caused by content mismatch between data of different modalities.

[0065] Table 2 Performance comparison of extracting credible multimodal relations on the MNRE dataset

[0066] See also Figure 3 The embodiment of the present invention further provides a multimodal relationship extraction device 300 under unpaired data, comprising: The feature extraction module 301 is used to extract features from the image modality data to obtain visual features at multiple different levels, and to extract features from the text modality data to obtain final-level text features; An attention fusion module 302 is used to fuse the visual features of each level of the multiple different levels of visual features with the final level text features based on a cross attention mechanism to obtain a hierarchical visual feature corresponding to the visual feature; A multi-level fusion module 303 is used to fuse each of the hierarchical visual features with the final-level text feature to obtain a multimodal feature corresponding to the hierarchical visual feature; Aggregation module 304, used for aggregating the plurality of hierarchical visual features, the plurality of multimodal features and the final layer text features to obtain a multimodal fusion feature; The relationship extraction module 305 is used to input the multimodal fusion features into the prediction module to perform relationship extraction and obtain a relationship extraction result.

[0067] Optionally, the feature extraction module 301 includes: An image encoding unit is used to preprocess the image modal data to obtain visual features of the 0th level, and then input the visual features of the 0th level into multiple image encoding modules connected in sequence to obtain visual features of the multiple different levels, wherein each of the image encoding modules is used to output the visual features of one level.

[0068] Optionally, the multi-level fusion module 303 includes: A first computing unit, configured to use the visual feature as a key-value pair and the final layer text feature as a query to calculate a plurality of attention heads; The second computing unit is used to calculate the hierarchical visual feature corresponding to the visual feature based on the multiple attention heads.

[0069] Optionally, the mth attention head for: ; ; in, is the first of the multiple visual features at different levels Hierarchical visual features, , is the number of image coding modules, is the final layer text feature, Used to represent the Softmax function, is the attention score, is the weight matrix used to generate the value vector, is the weight matrix used to generate the query vector, is the weight matrix used to generate the key vector, M is the number of attention heads, is the input feature dimension.

[0070] Optionally, the aggregation module 304 includes: A third calculation unit, configured to calculate an aggregation weight based on the plurality of the hierarchical visual features and the final layer text features; An aggregation unit is used to aggregate the plurality of hierarchical visual features, the plurality of multimodal features and the final layer text feature based on the aggregation weight to obtain the multimodal fusion feature.

[0071] Optionally, the aggregation weight for: ; in, is the pre-trained projection matrix, is the number of visual features, is the final layer text feature, , For the The hierarchical visual features, , Used to represent the Softmax function.

[0072] Optionally, the prediction module includes a prediction classification head, and the relationship extraction module 305 includes: An input unit, used for inputting the multimodal fusion feature into the prediction classification head for classification prediction to obtain a predicted multivariate logical value; A determination unit is used to determine a probability prediction result based on the predicted multivariate logical value, and the probability prediction result is used to characterize the relationship extraction result.

[0073] The multimodal relationship extraction device 300 under unpaired data provided in the embodiment of the present application can execute the above method embodiment, and its implementation principle and technical effect are similar, which will not be repeated in this embodiment.

[0074] It should be noted that the division of units in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional unit in each embodiment of the present application may be integrated into a processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0075] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a processor-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.

[0076] like Figure 4As shown, an embodiment of the present application provides an electronic device 400, including: a memory 402, a processor 401, and a program stored in the memory 402 and executable on the processor 401; the processor 401 is used to read the program in the memory 402 to implement the steps in the multimodal relationship extraction method under unpaired data as described above.

[0077] The embodiment of the present application also provides a readable storage medium, on which a program is stored. When the program is executed by a processor, each process of the embodiment of the multimodal relationship extraction method under unpaired data is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here. Among them, the readable storage medium can be any available medium or data storage device that can be accessed by the processor, including but not limited to magnetic storage (such as floppy disk, hard disk, magnetic tape, magneto-optical disk (MO), etc.), optical storage (such as compact disk (CD), digital video disc (DVD), Blu-ray Disc (BD), high-definition versatile disc (HVD), etc.), and semiconductor storage (such as read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read only memory (EEPROM), non-volatile memory (NAND FLASH), solid-state drive (Solid State Disk or Solid State Drive, SSD)), etc.

[0078] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0079] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD), and includes a number of instructions for a terminal (which can be a mobile phone, computer, server, air conditioner, or network equipment, etc.) to execute the methods described in each embodiment of the present application.

[0080] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A multimodal relationship extraction method under unpaired data, characterized in that: include: Perform feature extraction on image modality data to obtain visual features at multiple different levels, and perform feature extraction on text modality data to obtain final-level text features; For each visual feature of the multiple different levels of visual features, the visual feature is fused with the final layer text feature based on a cross attention mechanism to obtain a hierarchical visual feature corresponding to the visual feature; For each of the hierarchical visual features, the hierarchical visual features are fused with the final layer text features to obtain a multimodal feature corresponding to the hierarchical visual feature; Aggregating the plurality of the hierarchical visual features, the plurality of the multimodal features and the final layer text features to obtain a multimodal fusion feature; The multimodal fusion features are input into a prediction module to perform relationship extraction to obtain a relationship extraction result.

2. The method according to claim 1, characterized in that: The feature extraction of the image modality data is performed to obtain visual features at multiple different levels, including: After preprocessing the image modal data to obtain visual features at the 0th level, the visual features at the 0th level are input into a plurality of image encoding modules connected in sequence to obtain visual features at a plurality of different levels, wherein each of the image encoding modules is used to output the visual features at one level.

3. The method according to claim 1, characterized in that: The method of fusing the visual features with the final-layer text features based on the cross-attention mechanism to obtain hierarchical visual features corresponding to the visual features includes: Taking the visual features as key-value pairs and the final layer text features as queries, a plurality of attention heads are calculated; The hierarchical visual feature corresponding to the visual feature is calculated based on the multiple attention heads.

4. The method according to claim 3, characterized in that: The mth attention head for: ； ； in, is the first of the multiple visual features at different levels Hierarchical visual features, , is the number of image coding modules, is the final layer text feature, Used to represent the Softmax function, is the attention score, is the weight matrix used to generate the value vector, is the weight matrix used to generate the query vector, is the weight matrix used to generate the key vector, M is the number of attention heads, is the input feature dimension.

5. The method according to claim 1, characterized in that: The step of aggregating the plurality of hierarchical visual features, the plurality of multimodal features and the final layer text features to obtain a multimodal fusion feature includes: Calculating an aggregation weight based on a plurality of the hierarchical visual features and the final layer text features; The plurality of hierarchical visual features, the plurality of multimodal features and the final layer text features are aggregated based on the aggregation weights to obtain the multimodal fusion features.

6. The method according to claim 5, characterized in that: The aggregation weight for: ； in, is the pre-trained projection matrix, is the number of visual features, is the final layer text feature, , For the The hierarchical visual features, , Used to represent the Softmax function.

7. The method according to claim 1, characterized in that: The prediction module includes a prediction classification head, and the multimodal fusion feature is input into the prediction module for relationship extraction to obtain a relationship extraction result, including: Inputting the multimodal fusion features into the prediction classification head for classification prediction to obtain a predicted multivariate logical value; A probability prediction result is determined based on the predicted multivariate logical value, and the probability prediction result is used to characterize the relationship extraction result.

8. A multimodal relationship extraction device under unpaired data, characterized in that: include: The feature extraction module is used to extract features from image modality data to obtain visual features at multiple different levels, and to extract features from text modality data to obtain final-level text features; An attention fusion module, for fusing the visual features of each level of the multiple different levels of visual features with the final level text features based on a cross attention mechanism to obtain a hierarchical visual feature corresponding to the visual feature; A multi-level fusion module, for fusing each of the hierarchical visual features with the final-level text feature to obtain a multimodal feature corresponding to the hierarchical visual feature; An aggregation module, used for aggregating the plurality of hierarchical visual features, the plurality of multimodal features and the final layer text features to obtain a multimodal fusion feature; The relationship extraction module is used to input the multimodal fusion features into the prediction module to perform relationship extraction and obtain a relationship extraction result.

9. An electronic device, comprising: A memory, a processor, and a program stored in the memory and executable on the processor; wherein the processor is used to read the program in the memory to implement the steps of the multimodal relationship extraction method under unpaired data as described in any one of claims 1 to 7.

10. A readable storage medium for storing a program, characterized in that: When the program is executed by a processor, the steps of the multimodal relationship extraction method under unpaired data are implemented as described in any one of claims 1 to 7.

Images