Interactive multimodal fusion method and device for link prediction

Abstract

The present invention provides an interactive multimodal fusion method and device for link prediction. The method obtains a knowledge graph to be predicted, including a structured modality, a visual modality and a text modality; extracts high-level feature information of the structured modality, the visual modality and the text modality respectively, and obtains a structured modality embedding representation, a visual modality embedding representation and a text modality embedding representation; aggregates the structured modality embedding representation, the visual modality embedding representation and the text modality embedding representation, and obtains a multimodal embedding representation through bilinear matrix calculation; predicts a target entity through each different representation respectively, and obtains a structured prediction result, a visual prediction result, a text prediction result and a multimodal prediction result; adjusts the weight of each prediction result to obtain a target prediction result, and can effectively retain the proprietary knowledge in each modality through multimodal fusion to achieve complementary link prediction, thereby greatly improving the accuracy of link prediction.

Description

Technical Field

[0001] The present invention relates to the field of knowledge graph technology, and in particular to an interactive multimodal fusion method and device for link prediction. Background Art

[0002] In the field of knowledge graphs, since knowledge graphs contain rich knowledge, they are very important for many practical applications, such as question answering, dialogue systems, and recommendation systems. However, due to the complexity, diversity, and volatility of knowledge, knowledge graphs are usually incomplete. To solve this problem, link prediction is proposed to predict potential missing triplets. Traditional link prediction models, including transformation-based methods and neural network methods, are affected by the problem of structured bias between triplets. Therefore, some recent studies have used multimodal information for link prediction. Most of them map all modal information into a unified space to obtain commonalities and then perform link prediction.

[0003] However, this method cannot retain the unique information in each modality, resulting in relatively low accuracy of link prediction. Summary of the invention

[0004] The present invention provides an interactive multimodal fusion method and device for link prediction, which are used to solve the defect of low link prediction accuracy in the prior art, and effectively improve the effect of connection prediction by retaining the unique information of each modality for fusion.

[0005] The present invention provides an interactive multimodal fusion method for link prediction, comprising:

[0006] Obtaining a knowledge graph to be predicted, wherein the knowledge graph to be predicted includes a structured modality, a visualization modality, and a text modality;

[0007] Extracting high-level feature information of the structured modality, the visual modality, and the text modality respectively to obtain a structured modality embedding representation, a visual modality embedding representation, and a text modality embedding representation;

[0008] Aggregating the structured modality embedding representation, the visualization modality embedding representation, and the text modality embedding representation, and obtaining a multimodal embedding representation through bilinear matrix calculation;

[0009] Predicting a target entity through the structured modal embedding representation, the visualization modal embedding representation, the text modal embedding representation, and the multimodal embedding representation, respectively, to obtain a structured prediction result, a visualization prediction result, a text prediction result, and a multimodal prediction result;

[0010] The weights of the structured prediction result, the visualization prediction result, the text prediction result, and the multimodal prediction result are adjusted to obtain a target prediction result.

[0011] According to an interactive multimodal fusion method for link prediction provided by the present invention, the high-level feature information of the structured modality, the visualization modality and the text modality are extracted respectively to obtain the structured modality embedding representation, the visualization modality embedding representation and the text modality embedding representation, including:

[0012] The high-level feature information of the structured modality is extracted through the structured encoder to obtain a structured embedding representation;

[0013] The high-level feature information of the visualization modality is extracted through the visualization encoder to obtain the visualization embedding representation;

[0014] The high-level feature information of the text modality is extracted through the text encoder to obtain the text embedding representation.

[0015] According to an interactive multimodal fusion method for link prediction provided by the present invention, the structured modality embedding representation, the visualization modality embedding representation and the text modality embedding representation are aggregated to obtain a multimodal embedding representation by bilinear matrix calculation, including:

[0016] Converting the structured modality, the visualization modality, and the text modality into tensors;

[0017] Decomposing the tensor by a bilinear matrix to obtain a core tensor;

[0018] The structured modality embedding representation, the visualization modality embedding representation and the text modality embedding representation are fused with the core tensor to obtain a multimodal embedding representation.

[0019] According to an interactive multimodal fusion method for link prediction provided by the present invention, the target entity is predicted by the structured modal embedding representation, the visual modal embedding representation, the text modal embedding representation and the multimodal embedding representation respectively to obtain a structured prediction result, a visual prediction result, a text prediction result and a multimodal prediction result, including:

[0020] Providing a context transformation matrix for the structured modality embedding representation, the visualization modality embedding representation, the text modality embedding representation, and the multimodal embedding representation through relationships;

[0021] The similarity of candidate entities is predicted by multiplying the context transformation matrix with the structured modality embedding representation, the visualization modality embedding representation, the text modality embedding representation and the multimodal embedding representation, thereby obtaining a structured prediction result, a visualization prediction result, a text prediction result and a multimodal prediction result.

[0022] According to an interactive multimodal fusion method for link prediction provided by the present invention, the weights of the structured prediction result, the visual prediction result, the text prediction result and the multimodal prediction result are adjusted to obtain a target prediction result, including:

[0023] Based on the joint loss function, respectively determining the weights of the structured prediction result, the visualization prediction result, the text prediction result, and the multimodal prediction result;

[0024] Based on the preset weight adjustment rules, the weights are adjusted to obtain the target prediction results.

[0025] The present invention also provides an interactive multi-modal fusion device for link prediction, comprising:

[0026] An acquisition module is used to acquire a knowledge graph to be predicted, wherein the knowledge graph to be predicted includes a structured modality, a visual modality, and a text modality;

[0027] A modality encoder module, used to extract high-level feature information of the structured modality, the visual modality and the textual modality respectively, to obtain a structured modality embedding representation, a visual modality embedding representation and a textual modality embedding representation;

[0028] A multimodal fusion module, used for aggregating the structured modality embedding representation, the visualization modality embedding representation and the text modality embedding representation, and obtaining a multimodal embedding representation through bilinear matrix calculation;

[0029] A contextual relationship module, used to predict the target entity through the structured modal embedding representation, the visual modal embedding representation, the text modal embedding representation and the multimodal embedding representation, respectively, to obtain a structured prediction result, a visual prediction result, a text prediction result and a multimodal prediction result;

[0030] The decision fusion module is used to adjust the weights of the structured prediction result, the visualization prediction result, the text prediction result and the multimodal prediction result to obtain a target prediction result.

[0031] According to an interactive multimodal fusion device for link prediction provided by the present invention, the modal encoder module is specifically used for:

[0032] The high-level feature information of the structured modality is extracted through the structured encoder to obtain a structured embedding representation;

[0033] The high-level feature information of the visualization modality is extracted through the visualization encoder to obtain the visualization embedding representation;

[0034] The high-level feature information of the text modality is extracted through the text encoder to obtain the text embedding representation.

[0035] According to an interactive multimodal fusion device for link prediction provided by the present invention, the multimodal fusion module is specifically used for:

[0036] Converting the structured modality, the visualization modality, and the text modality into tensors;

[0037] Decomposing the tensor by a bilinear matrix to obtain a core tensor;

[0038] The structured modality embedding representation, the visualization modality embedding representation and the text modality embedding representation are fused with the core tensor to obtain a multimodal embedding representation.

[0039] According to an interactive multimodal fusion device for link prediction provided by the present invention, the contextual relationship module is specifically used for:

[0040] Providing a context transformation matrix for the structured modality embedding representation, the visualization modality embedding representation, the text modality embedding representation, and the multimodal embedding representation through relationships;

[0041] The similarity of candidate entities is predicted by multiplying the context transformation matrix with the structured modality embedding representation, the visualization modality embedding representation, the text modality embedding representation and the multimodal embedding representation, thereby obtaining a structured prediction result, a visualization prediction result, a text prediction result and a multimodal prediction result.

[0042] According to an interactive multimodal fusion device for link prediction provided by the present invention, the decision fusion module is specifically used for:

[0043] Based on the joint loss function, respectively determining the weights of the structured prediction result, the visualization prediction result, the text prediction result, and the multimodal prediction result;

[0044] Based on the preset weight adjustment rules, the weights are adjusted to obtain the target prediction results.

[0045] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any of the above-mentioned interactive multimodal fusion methods for link prediction are implemented.

[0046] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-described interactive multimodal fusion methods for link prediction.

[0047] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of any of the above-mentioned interactive multimodal fusion methods for link prediction are implemented.

[0048] The present invention provides an interactive multimodal fusion method and device for link prediction. The method obtains a knowledge graph to be predicted, wherein the knowledge graph to be predicted includes a structured modality, a visual modality and a text modality; extracts high-level feature information of the structured modality, the visual modality and the text modality respectively to obtain a structured modality embedding representation, a visual modality embedding representation and a text modality embedding representation; aggregates the structured modality embedding representation, the visual modality embedding representation and the text modality embedding representation to obtain a multimodal embedding representation through bilinear matrix calculation; predicts a target entity through the structured modality embedding representation, the visual modality embedding representation, the text modality embedding representation and the multimodal embedding representation respectively to obtain a structured prediction result, a visual prediction result, a text prediction result and a multimodal prediction result; adjusts the weights of the structured prediction result, the visual prediction result, the text prediction result and the multimodal prediction result to obtain a target prediction result. Through multimodal fusion, the proprietary knowledge in each modality can be effectively retained to achieve complementary link prediction, thereby greatly improving the accuracy of link prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0050] Figure 1 It is a flowchart of the interactive multimodal fusion method for link prediction provided by the present invention;

[0051] Figure 2It is a structural schematic diagram of an interactive multimodal fusion device for link prediction provided by the present invention;

[0052] Figure 3 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] Combine the following Figure 1-Figure 3 An interactive multimodal fusion method and device for link prediction of the present invention are described.

[0055] Figure 1 It is a flowchart of the interactive multimodal fusion method for link prediction provided by the present invention.

[0056] like Figure 1 As shown, an interactive multimodal fusion method for link prediction provided by an embodiment of the present invention includes the following steps:

[0057] 101. Obtain the knowledge graph to be predicted, which includes structured modality, visualization modality and text modality.

[0058] The knowledge graph that needs to be predicted is defined as the knowledge graph to be predicted, where the knowledge graph to be predicted is a whole knowledge graph, and then the missing parts in the knowledge graph are predicted by the present invention. The knowledge graph to be predicted includes multiple modes, including structured mode, visualization mode and text mode.

[0059] 102. The high-level feature information of the structured modality, the visual modality and the textual modality is extracted respectively to obtain the structured modality embedding representation, the visual modality embedding representation and the textual modality embedding representation.

[0060] In a specific implementation process, a modality-specific encoder can be used to input the original features of each modality, such as descriptive text, images, etc., and high-level feature information can be extracted through a pre-trained encoder model, such as BERT, VGG, etc., to obtain the encoded embedded representation of each modality for subsequent prediction. The modality-specific encoder refers to a tool used to extract structural, visual and text features.

[0061] Specifically, the high-level feature information of the structured modality is extracted through a structured encoder to obtain a structured embedding representation;

[0062] The structured encoder is a graph attention model using the TransE energy function, and its loss function is as follows:

[0063]

[0064] Among them, γ is the boundary hyperparameter, Indicated in The set of negative sample triplets extracted from , h represents the head entity, r represents the relationship, t represents the tail entity, such as (Beijing, is the capital, China), and E represents the TransE energy function, which is specifically as shown in formula (2):

[0065]

[0066] Then, the high-level feature information of the visualization modality is extracted through the visualization encoder to obtain the visualization embedding representation, which usually uses the VGG16 model pre-trained on ImageNet. The hidden layer output of the last layer before the output is used as the visualization feature, which is a 4096-dimensional vector.

[0067] Then, the high-level feature information of the text modality is extracted through the text encoder to obtain the text embedding representation. The BERT model is usually used to obtain a vector with 768 dimensions.

[0068] 103. Aggregate the structured modal embedding representation, visualization modal embedding representation and text modal embedding representation, and obtain the multimodal embedding representation through bilinear matrix calculation.

[0069] Multimodal fusion refers to the use of bilinear Tucker decomposition plus contrastive learning to model the interaction between different modalities. Specifically, it can be to transform the structured modality, visualization modality and text modality into tensors, and decompose a tensor into a core tensor multiplied by a matrix (decomposing a mode into 4 factors) based on bilinear matrix and Tucker decomposition, as shown in formula (3):

[0070]

[0071] in, is the transformation matrix, It is a small core tensor. The s, v, t in the subscripts represent the structured mode, the visualization mode, and the text mode, respectively. d represents the overall mapping. c represents the core tensor. The variable d represents the original dimension and t represents the mapping dimension. D represents the overall mapping dimension.

[0072] Then, the entity embedding representation, i.e., the structured modal embedding representation, the visualization modal embedding representation, and the text modal embedding representation, are mapped to the low-dimensional space respectively, and then the structured modal embedding representation, the visualization modal embedding representation, and the text modal embedding representation are fused with the core tensor to obtain the multimodal embedding representation, as shown in formula (4):

[0073]

[0074] in, is an implicit representation, Decomposed into transformation matrices for each mode k.

[0075] For each sampled entity, its contrast loss function is shown in formula (5):

[0076]

[0077] Wherein, M = {(s,v), (s,t), (v,t)} represents the set of all modality pairs, s, v, t in the subscript and superscript represent the structured modality, the visualization modality and the text modality respectively, m represents the fusion modality, and the calculation formula of d is as shown in formula (6):

[0078]

[0079] 104. The target entity is predicted by structured modal embedding representation, visual modal embedding representation, textual modal embedding representation and multimodal embedding representation respectively, and structured prediction results, visual prediction results, textual prediction results and multimodal prediction results are obtained.

[0080] The essence of prediction is to use the embedding representation of the entity and the embedding representation of the relationship as input, and obtain the context-related embedding representation through bilinear matrix calculation for predicting the target entity. Specifically, the context transformation matrix can be provided for the structured modal embedding representation, visual modal embedding representation, text modal embedding representation and multimodal embedding representation through the relationship; then the context transformation matrix is ​​multiplied with the structured modal embedding representation, visual modal embedding representation, text modal embedding representation and multimodal embedding representation respectively to obtain the structured modal embedding representation, visual modal embedding representation, text modal embedding representation and multimodal embedding representation with context information, and then multiplied with the embedding representation of all other candidate entities to predict the similarity between the candidate entities and obtain the structured prediction results, visual prediction results, text prediction results and multimodal prediction results. The calculation of the loss function L in the prediction process is as shown in formula (7)(8)(9):

[0081]

[0082]

[0083]

[0084] Where e is the initial entity embedding, is the contextual entity embedding taking into account the relationship, W is the weight matrix, r and b are bias vectors. e represents the embedding representation of the entity, r represents the relationship, the subscript represents the weight matrix W related to r, and when used as a variable, it represents the relationship embedding representation vector, b is the bias vector, cosine is the cosine similarity calculation, sigma represents the sigmoid activation function, and log represents the logarithmic calculation.

[0085] 105. Adjust the weights of the structured prediction results, the visualization prediction results, the text prediction results, and the multimodal prediction results to obtain the target prediction results.

[0086] After obtaining the corresponding prediction scores under different modalities, the prediction results of each modality can be combined to obtain the final prediction result. The input is the probability vector representing the similarity score output by the contextual relationship model of multiple modalities, and the final probability vector is obtained by weighting. Specifically, the weights of the structured prediction results, visualization prediction results, text prediction results and multimodal prediction results can be determined based on the joint loss function. Under the multimodal setting, a certain modality is assigned a different contextual relationship model, and then its own results are used for training. In the training, the joint loss function is as follows:

[0087]

[0088] in, represents the binary cross entropy loss of modality k, γk is the learned weight parameter (initialized to 1), k represents s, v, t, m, s represents structured modality, v represents visualization modality, t represents text modality, and m represents multimodality. It is the contrastive learning loss in multimodal fusion.

[0089] Then, in the inference stage, different similarity scores are assigned different weights to balance the contributions of different modalities. That is, based on the preset weight adjustment rules, the weights are adjusted to obtain the target prediction result, as shown in formula (11):

[0090]

[0091] Among them, γk is the weight of modality k, k represents s, v, t, m, s represents structured modality, v represents visualization modality, t represents text modality, and m represents multimodality.

[0092] Therefore, the final prediction is made by comprehensively considering the structured modality, visualization modality, text modality and multi-modality to obtain the target prediction result, that is, the entire link prediction is completed, and the existence of the edge between the two nodes in the knowledge graph is further verified.

[0093] The present embodiment provides an interactive multimodal fusion method for link prediction, which obtains a knowledge graph to be predicted, wherein the knowledge graph to be predicted includes a structured modality, a visual modality, and a text modality; extracts high-level feature information of the structured modality, the visual modality, and the text modality respectively to obtain a structured modality embedding representation, a visual modality embedding representation, and a text modality embedding representation; aggregates the structured modality embedding representation, the visual modality embedding representation, and the text modality embedding representation to obtain a multimodal embedding representation through bilinear matrix calculation; predicts a target entity through the structured modality embedding representation, the visual modality embedding representation, the text modality embedding representation, and the multimodal embedding representation respectively to obtain a structured prediction result, a visual prediction result, a text prediction result, and a multimodal prediction result; adjusts the weights of the structured prediction result, the visual prediction result, the text prediction result, and the multimodal prediction result to obtain a target prediction result. Through multimodal fusion, the proprietary knowledge in each modality can be effectively retained to achieve complementary link prediction, thereby greatly improving the accuracy of link prediction.

[0094] Based on the same general inventive concept, the present application also protects an interactive multimodal fusion device for link prediction. The interactive multimodal fusion device for link prediction provided by the present invention is described below. The interactive multimodal fusion device for link prediction described below and the interactive multimodal fusion method for link prediction described above can be referenced to each other.

[0095] Figure 2 It is a structural schematic diagram of an interactive multimodal fusion device for link prediction provided by the present invention.

[0096] like Figure 2 As shown, an interactive multimodal fusion device for link prediction provided by an embodiment of the present invention includes:

[0097] An acquisition module 21 is used to acquire a knowledge graph to be predicted, wherein the knowledge graph to be predicted includes a structured modality, a visual modality, and a text modality;

[0098] A modality encoder module 22, used to extract high-level feature information of the structured modality, the visual modality and the textual modality respectively, to obtain a structured modality embedding representation, a visual modality embedding representation and a textual modality embedding representation;

[0099] A multimodal fusion module 23, used for aggregating the structured modal embedding representation, the visualization modal embedding representation and the text modal embedding representation, and obtaining a multimodal embedding representation through bilinear matrix calculation;

[0100] A contextual relationship module 24, used to predict the target entity through the structured modal embedding representation, the visual modal embedding representation, the text modal embedding representation and the multimodal embedding representation, and obtain a structured prediction result, a visual prediction result, a text prediction result and a multimodal prediction result;

[0101] The decision fusion module 25 is used to adjust the weights of the structured prediction result, the visualization prediction result, the text prediction result and the multimodal prediction result to obtain a target prediction result.

[0102] The present embodiment provides an interactive multimodal fusion device for link prediction, which obtains a knowledge graph to be predicted, wherein the knowledge graph to be predicted includes a structured modality, a visual modality, and a text modality; extracts high-level feature information of the structured modality, the visual modality, and the text modality respectively to obtain a structured modality embedding representation, a visual modality embedding representation, and a text modality embedding representation; aggregates the structured modality embedding representation, the visual modality embedding representation, and the text modality embedding representation to obtain a multimodal embedding representation through bilinear matrix calculation; predicts a target entity through the structured modality embedding representation, the visual modality embedding representation, the text modality embedding representation, and the multimodal embedding representation respectively to obtain a structured prediction result, a visual prediction result, a text prediction result, and a multimodal prediction result; adjusts the weights of the structured prediction result, the visual prediction result, the text prediction result, and the multimodal prediction result to obtain a target prediction result. Through multimodal fusion, the proprietary knowledge in each modality can be effectively retained to achieve complementary link prediction, thereby greatly improving the accuracy of link prediction.

[0103] Furthermore, the modal encoder module in this embodiment is specifically used for:

[0104] The high-level feature information of the structured modality is extracted through the structured encoder to obtain a structured embedding representation;

[0105] The high-level feature information of the visualization modality is extracted through the visualization encoder to obtain the visualization embedding representation;

[0106] The high-level feature information of the text modality is extracted through the text encoder to obtain the text embedding representation.

[0107] Furthermore, the multimodal fusion module in this embodiment is specifically used for:

[0108] Converting the structured modality, the visualization modality, and the text modality into tensors;

[0109] Decomposing the tensor by a bilinear matrix to obtain a core tensor;

[0110] The structured modality embedding representation, the visualization modality embedding representation and the text modality embedding representation are fused with the core tensor to obtain a multimodal embedding representation.

[0111] Furthermore, the context relationship module in this embodiment is specifically used for:

[0112] Providing a context transformation matrix for the structured modality embedding representation, the visualization modality embedding representation, the text modality embedding representation, and the multimodal embedding representation through relationships;

[0113] The similarity of candidate entities is predicted by multiplying the context transformation matrix with the structured modality embedding representation, the visualization modality embedding representation, the text modality embedding representation and the multimodal embedding representation, thereby obtaining a structured prediction result, a visualization prediction result, a text prediction result and a multimodal prediction result.

[0114] Furthermore, the decision fusion module in this embodiment is specifically used for:

[0115] Based on the joint loss function, respectively determining the weights of the structured prediction result, the visualization prediction result, the text prediction result, and the multimodal prediction result;

[0116] Based on the preset weight adjustment rules, the weights are adjusted to obtain the target prediction results.

[0117] Figure 3 It is a structural schematic diagram of the electronic device provided by the present invention.

[0118] like Figure 3As shown, the electronic device may include: a processor (processor) 310, a communication interface (Communications Interface) 320, a memory (memory) 330 and a communication bus 340, wherein the processor 310, the communication interface 320, and the memory 330 communicate with each other through the communication bus 340. The processor 310 can call the logic instructions in the memory 330 to execute an interactive multimodal fusion method for link prediction, which includes: obtaining a knowledge graph to be predicted, wherein the knowledge graph to be predicted includes a structured modality, a visual modality, and a text modality; respectively extracting high-level feature information of the structured modality, the visual modality, and the text modality to obtain a structured modality embedding representation, a visual modality embedding representation, and a text modality embedding representation; aggregating the structured modality embedding representation, the visual modality embedding representation, and the text modality embedding representation, and obtaining a multimodal embedding representation through bilinear matrix calculation; respectively predicting a target entity through the structured modality embedding representation, the visual modality embedding representation, the text modality embedding representation, and the multimodal embedding representation to obtain a structured prediction result, a visual prediction result, a text prediction result, and a multimodal prediction result; adjusting the weights of the structured prediction result, the visual prediction result, the text prediction result, and the multimodal prediction result to obtain a target prediction result.

[0119] In addition, the logic instructions in the above-mentioned memory 330 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0120] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the interactive multimodal fusion method for link prediction provided by the above methods, which includes: obtaining a knowledge graph to be predicted, wherein the knowledge graph to be predicted includes a structured modality, a visual modality, and a textual modality; respectively extracting high-level feature information of the structured modality, the visual modality, and the textual modality to obtain a structured modality embedding representation and a visual modality embedding representation. and text modal embedding representation; aggregating the structured modal embedding representation, the visual modal embedding representation and the text modal embedding representation to obtain a multimodal embedding representation through bilinear matrix calculation; predicting a target entity through the structured modal embedding representation, the visual modal embedding representation, the text modal embedding representation and the multimodal embedding representation respectively to obtain a structured prediction result, a visual prediction result, a text prediction result and a multimodal prediction result; adjusting the weights of the structured prediction result, the visual prediction result, the text prediction result and the multimodal prediction result to obtain a target prediction result.

[0121] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the interactive multimodal fusion method for link prediction provided by the above-mentioned methods, the method comprising: obtaining a knowledge graph to be predicted, the knowledge graph to be predicted comprising a structured modality, a visual modality and a text modality; extracting high-level feature information of the structured modality, the visual modality and the text modality respectively to obtain a structured modality embedding representation, a visual modality embedding representation and a text modality embedding representation; aggregating the structured modality embedding representation, the visual modality embedding representation and the text modality embedding representation to obtain a multimodal embedding representation through bilinear matrix calculation; predicting a target entity through the structured modality embedding representation, the visual modality embedding representation, the text modality embedding representation and the multimodal embedding representation respectively to obtain a structured prediction result, a visual prediction result, a text prediction result and a multimodal prediction result; adjusting the weights of the structured prediction result, the visual prediction result, the text prediction result and the multimodal prediction result to obtain a target prediction result.

[0122] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.

[0123] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0124] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An interactive multimodal fusion method for link prediction, characterized in that: include: Obtaining a knowledge graph to be predicted, wherein the knowledge graph to be predicted includes a structured modality, a visualization modality, and a text modality; Extract high-level feature information of the structured modality, the visualization modality and the text modality respectively, and obtain structured modality embedding representation, visualization modality embedding representation and text modality embedding representation; wherein the original features of each modality in the structured modality, the visualization modality and the text modality include at least: descriptive text, image; the high-level feature information includes at least: BERT features and VGG features; Aggregating the structured modality embedding representation, the visualization modality embedding representation, and the text modality embedding representation, and obtaining a multimodal embedding representation through bilinear matrix calculation; Predicting a target entity through the structured modal embedding representation, the visualization modal embedding representation, the text modal embedding representation, and the multimodal embedding representation, respectively, to obtain a structured prediction result, a visualization prediction result, a text prediction result, and a multimodal prediction result; Specifically including: providing a context transformation matrix for the structured modality embedding representation, the visualization modality embedding representation, the text modality embedding representation and the multimodal embedding representation through relationships; Predicting the similarity of candidate entities by multiplying the context transformation matrix with the structured modality embedding representation, the visualization modality embedding representation, the text modality embedding representation, and the multimodal embedding representation, respectively, to obtain a structured prediction result, a visualization prediction result, a text prediction result, and a multimodal prediction result; The weights of the structured prediction result, the visualization prediction result, the text prediction result, and the multimodal prediction result are adjusted to obtain a target prediction result.

2. The interactive multimodal fusion method for link prediction according to claim 1, characterized in that: The extracting high-level feature information of the structured modality, the visual modality and the text modality respectively to obtain a structured modality embedding representation, a visual modality embedding representation and a text modality embedding representation includes: The high-level feature information of the structured modality is extracted through the structured encoder to obtain a structured embedding representation; The high-level feature information of the visualization modality is extracted through the visualization encoder to obtain the visualization embedding representation; The high-level feature information of the text modality is extracted through the text encoder to obtain the text embedding representation.

3. The interactive multimodal fusion method for link prediction according to claim 1, characterized in that: The aggregating the structured modality embedding representation, the visualization modality embedding representation, and the text modality embedding representation to obtain a multimodal embedding representation by bilinear matrix calculation includes: Converting the structured modality, the visualization modality, and the text modality into tensors; Decomposing the tensor by a bilinear matrix to obtain a core tensor; The structured modality embedding representation, the visualization modality embedding representation and the text modality embedding representation are fused with the core tensor to obtain a multimodal embedding representation.

4. The interactive multimodal fusion method for link prediction according to claim 1, characterized in that: The adjusting the weights of the structured prediction result, the visualization prediction result, the text prediction result, and the multimodal prediction result to obtain a target prediction result includes: Based on the joint loss function, respectively determining the weights of the structured prediction result, the visualization prediction result, the text prediction result, and the multimodal prediction result; Based on the preset weight adjustment rules, the weights are adjusted to obtain the target prediction results.

5. An interactive multimodal fusion device for link prediction, characterized in that: include: An acquisition module, used to acquire a knowledge graph to be predicted, wherein the knowledge graph to be predicted includes a structured modality, a visual modality, and a text modality; A modality encoder module is used to extract high-level feature information of the structured modality, the visualization modality and the text modality respectively, and obtain a structured modality embedding representation, a visualization modality embedding representation and a text modality embedding representation; wherein the original features of each modality in the structured modality, the visualization modality and the text modality include at least: descriptive text, image; the high-level feature information includes at least: BERT features and VGG features; A multimodal fusion module, used for aggregating the structured modality embedding representation, the visualization modality embedding representation and the text modality embedding representation, and obtaining a multimodal embedding representation through bilinear matrix calculation; A contextual relationship module, used to predict the target entity through the structured modal embedding representation, the visual modal embedding representation, the text modal embedding representation and the multimodal embedding representation, respectively, to obtain a structured prediction result, a visual prediction result, a text prediction result and a multimodal prediction result; The contextual relationship module is specifically used to: provide a context transformation matrix for the structured modal embedding representation, the visual modal embedding representation, the text modal embedding representation and the multimodal embedding representation through relationships; predict the similarity of candidate entities by multiplying the context transformation matrix with the structured modal embedding representation, the visual modal embedding representation, the text modal embedding representation and the multimodal embedding representation, respectively, to obtain a structured prediction result, a visual prediction result, a text prediction result and a multimodal prediction result; The decision fusion module is used to adjust the weights of the structured prediction result, the visualization prediction result, the text prediction result and the multimodal prediction result to obtain a target prediction result.

6. The interactive multimodal fusion device for link prediction according to claim 5, characterized in that: The modal encoder module is specifically used for: The high-level feature information of the structured modality is extracted through the structured encoder to obtain a structured embedding representation; The high-level feature information of the visualization modality is extracted through the visualization encoder to obtain the visualization embedding representation; The high-level feature information of the text modality is extracted through the text encoder to obtain the text embedding representation.

7. The interactive multimodal fusion device for link prediction according to claim 5, characterized in that: The multimodal fusion module is specifically used for: Converting the structured modality, the visualization modality, and the text modality into tensors; Decomposing the tensor by a bilinear matrix to obtain a core tensor; The structured modality embedding representation, the visualization modality embedding representation and the text modality embedding representation are fused with the core tensor to obtain a multimodal embedding representation.

8. The interactive multimodal fusion device for link prediction according to claim 5, characterized in that: The decision fusion module is specifically used for: Based on the joint loss function, respectively determining the weights of the structured prediction result, the visualization prediction result, the text prediction result, and the multimodal prediction result; Based on the preset weight adjustment rules, the weights are adjusted to obtain the target prediction results.

Images