Method and device for completing multi-modal knowledge graph, and electronic device

Abstract

This disclosure provides a method, apparatus, and electronic device for completing multimodal knowledge graphs, relating to the field of data processing technology, particularly deep learning. The specific implementation scheme is as follows: Feature extraction is performed on the multimodal knowledge graph to obtain the first feature vector of each modality; each modality is grouped pairwise, and semantic information is extracted and replaced from the first feature vectors of the two modalities within any group to obtain the second feature vectors of each of the two modalities in that group; the second feature vectors corresponding to each group are fused to obtain a multimodal joint feature vector; based on the multimodal joint feature vector, the multimodal knowledge graph is completed to obtain the target multimodal knowledge graph. In this embodiment, common features between multimodalities can be mined, and redundancy can be reduced through information exchange. Important semantic information can be jointly learned across modalities, fully utilizing multimodal information and improving the performance of knowledge graph completion.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and in particular to the field of deep learning and other related technologies. Background Technology

[0002] In related technologies, structural information in knowledge graphs is generally used to complete knowledge graphs, or entity representations are enhanced based on text / image information to complete knowledge graphs. However, this completion method is insufficient in mining multimodal information and may lead to the problem of missing relationships, resulting in low accuracy in completing knowledge graphs. Therefore, how to make full use of multimodal information to improve the accuracy of knowledge graph completion has become one of the important research directions. Summary of the Invention

[0003] This disclosure provides a method, apparatus, and electronic device for completing a multimodal knowledge graph.

[0004] According to one aspect of this disclosure, a method for completing a multimodal knowledge graph is provided, comprising:

[0005] Feature extraction is performed on the multimodal knowledge graph to obtain the first feature vector of each modality;

[0006] Each modality is grouped into pairs, and semantic information is extracted and replaced from the first feature vectors of the two modalities in any group to obtain the second feature vectors of each of the two modalities in any group.

[0007] The second feature vectors corresponding to each group are fused to obtain the multimodal joint feature vector;

[0008] Based on the multimodal joint feature vector, the multimodal knowledge graph is completed to obtain the target multimodal knowledge graph.

[0009] In this embodiment, semantic information extraction and replacement of the first feature vectors of two modalities within any group can uncover common features between molecular structure information and textual description information. This allows for information exchange to reduce redundancy and enables cross-modal joint learning of important semantic information. Fusing the second feature vectors corresponding to each group reduces the differences between modalities. Based on the multimodal joint feature vectors, the multimodal knowledge graph is completed, fully utilizing multimodal information and improving the accuracy of knowledge graph completion.

[0010] According to another aspect of this disclosure, a training method for a multimodal knowledge graph completion model is provided, comprising:

[0011] Obtain the multimodal knowledge graph of the samples and their corresponding sample triples;

[0012] The multimodal knowledge graph completion model is trained based on the sample multimodal knowledge graph to obtain the completed multimodal knowledge graph of the sample multimodal knowledge graph, as well as the predicted triples of the completed multimodal knowledge graph;

[0013] The multimodal knowledge graph completion model is adjusted based on the sample triples and the predicted triples, and then the training continues using the next sample multimodal knowledge graph until the training ends and the target multimodal knowledge graph completion model is obtained; wherein, any triple includes a head entity, a tail entity, and the relationship between the head entity and the tail entity.

[0014] In this embodiment, a multimodal knowledge graph is completed based on a multimodal joint feature vector, making full use of multimodal information. The multimodal knowledge graph completion model is trained based on positive and negative samples, which improves the accuracy of knowledge graph completion and solves the problems of not being able to adapt to molecular structure information and not being able to explicitly utilize the features shared between modalities. This can greatly improve the performance of multimodal knowledge graph completion.

[0015] According to another aspect of this disclosure, a method for completing a multimodal knowledge graph is provided, comprising:

[0016] The multimodal knowledge graph to be completed is input into the trained multimodal knowledge graph completion model to obtain the completed target multimodal knowledge graph.

[0017] The multimodal knowledge graph completion model is trained using the training method described above.

[0018] According to another aspect of this disclosure, a multimodal knowledge graph completion device is provided, comprising:

[0019] The first acquisition module is used to extract features from the multimodal knowledge graph and obtain the first feature vector of each modality;

[0020] The second acquisition module is used to group each modality into pairs, and extract and replace the semantic information of the first feature vectors of the two modalities in any group to obtain the second feature vectors of each of the two modalities in any group.

[0021] The third acquisition module is used to fuse the second feature vectors corresponding to each group to obtain the multimodal joint feature vector;

[0022] The completion acquisition module is used to complete the multimodal knowledge graph based on the multimodal joint feature vector and obtain the target multimodal knowledge graph.

[0023] According to another aspect of this disclosure, a training apparatus for a multimodal knowledge graph completion model is provided, comprising:

[0024] The first acquisition module is used to acquire the multimodal knowledge graph of the samples and their corresponding sample triples;

[0025] The second acquisition module is used to train the multimodal knowledge graph completion model based on the sample multimodal knowledge graph in order to obtain the completed multimodal knowledge graph of the sample multimodal knowledge graph and the predicted triples of the completed multimodal knowledge graph.

[0026] The third acquisition module is used to adjust the multimodal knowledge graph completion model based on the sample triples and the predicted triples, and return to continue training using the next sample multimodal knowledge graph until the training ends and the target multimodal knowledge graph completion model is obtained; wherein, any triple includes a head entity, a tail entity, and the relationship between the head entity and the tail entity.

[0027] According to another aspect of this disclosure, a multimodal knowledge graph completion device is provided, comprising:

[0028] The completion module is used to input the multimodal knowledge graph to be completed into the trained multimodal knowledge graph completion model to obtain the completed target multimodal knowledge graph; wherein, the multimodal knowledge graph completion model is trained using the training device of the multimodal knowledge graph completion model described above.

[0029] According to another aspect of this disclosure, an electronic device is provided, comprising:

[0030] At least one processor; and

[0031] A memory that is communicatively connected to at least one processor; wherein,

[0032] The memory stores instructions that can be executed by at least one processor, which enables the at least one processor to execute the multimodal knowledge graph completion method or the training method of the multimodal knowledge graph completion model disclosed herein.

[0033] According to another aspect of this disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute a method for completing a multimodal knowledge graph or a method for training a multimodal knowledge graph completion model according to this disclosure.

[0034] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of a method for completing a multimodal knowledge graph according to this disclosure or the steps of a method for training a multimodal knowledge graph completion model.

[0035] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0036] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0037] Figure 1 This is a flowchart of a method for completing a multimodal knowledge graph according to an embodiment of this disclosure;

[0038] Figure 2 This is a flowchart of a method for completing a multimodal knowledge graph according to an embodiment of this disclosure;

[0039] Figure 3 This is a schematic diagram of a method for completing a multimodal knowledge graph according to an embodiment of this disclosure;

[0040] Figure 4 This is a schematic diagram of a method for completing a multimodal knowledge graph according to an embodiment of this disclosure;

[0041] Figure 5 This is a schematic diagram of a method for completing a multimodal knowledge graph according to an embodiment of this disclosure;

[0042] Figure 6 This is a flowchart of a method for completing a multimodal knowledge graph according to an embodiment of this disclosure;

[0043] Figure 7 This is a schematic diagram of a method for completing a multimodal knowledge graph according to an embodiment of this disclosure;

[0044] Figure 8 This is a flowchart of a training method for a multimodal knowledge graph completion model according to an embodiment of this disclosure;

[0045] Figure 9 This is a schematic diagram of the structure of a multimodal knowledge graph completion model according to an embodiment of this disclosure;

[0046] Figure 10 This is a structural diagram of a multimodal knowledge graph completion device according to an embodiment of the present disclosure;

[0047] Figure 11 This is a structural diagram of a training device for a multimodal knowledge graph completion model according to an embodiment of the present disclosure;

[0048] Figure 12 This disclosure provides a structural diagram of a multimodal knowledge graph completion device according to one embodiment;

[0049] Figure 13This disclosure provides a schematic block diagram of an electronic device according to one embodiment. Detailed Implementation

[0050] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0051] The following is a brief description of the technical field involved in the solution disclosed herein:

[0052] Data processing: Data is a form of expression of facts, concepts, or instructions, which can be processed manually or automatically. After being interpreted and given meaning, data becomes information. Data processing involves the collection, storage, retrieval, processing, transformation, and transmission of data. The fundamental purpose of data processing is to extract and derive valuable and meaningful data from large amounts of potentially chaotic and incomprehensible data. Data processing is a fundamental component of systems engineering and automatic control. Data processing permeates all areas of social production and social life. The development of data processing technology and the breadth and depth of its applications have profoundly influenced the progress of human society.

[0053] Deep learning (DL) is a new research direction in the field of machine learning (ML), bringing it closer to its original goal—artificial intelligence (AI). Deep learning learns the inherent patterns and hierarchical representations of sample data. The information gained during this learning process greatly aids in interpreting data such as text, images, and sound. Its ultimate goal is to enable machines to possess analytical and learning capabilities like humans, capable of recognizing data such as text, images, and sound. Deep learning is a complex machine learning algorithm that has achieved results in speech and image recognition far exceeding previous related technologies. Deep learning has also made significant progress in search technology, data mining, machine learning, machine translation, natural language processing, multimedia learning, speech recognition, recommendation and personalization technologies, and other related fields. Deep learning enables machines to mimic human activities such as sight, hearing, and thought, solving many complex pattern recognition problems and significantly advancing artificial intelligence-related technologies.

[0054] Figure 1 This is a flowchart of a multimodal knowledge graph completion method according to an embodiment of this disclosure, such as... Figure 1As shown, the method includes the following steps:

[0055] S101, extract features from the multimodal knowledge graph to obtain the first feature vector of each modality.

[0056] Multimodal biometrics refers to the integration or fusion of two or more biometric technologies, leveraging their unique advantages and combining them with data fusion technology to make the authentication and identification process more accurate and secure. Optionally, in the embodiments of this application, the modality may include text information, molecular information, structural information, etc.

[0057] Knowledge graphs are a series of different graphics that display the development process and structural relationships of knowledge. By using visualized graphs, we can vividly show the core structure, development history, cutting-edge fields, and overall knowledge architecture of a discipline, thus achieving the goal of multidisciplinary integration and providing practical and valuable references for disciplinary research.

[0058] In this embodiment of the disclosure, the multimodal knowledge graph can be a multimodal biological knowledge graph (BKG). The multimodal biological knowledge graph has the ability to model complex biological systems in a structured manner to support various tasks related to drugs, diseases and genes.

[0059] In this embodiment of the disclosure, in order to mine multimodal information of multimodal knowledge graphs, feature extraction can be performed on the multimodal knowledge graph. Optionally, a bidirectional encoder representation model (BERT) can be used to extract text information from the multimodal knowledge graph to obtain a text embedding vector as a first feature vector of a modality. Molecular information of the multimodal knowledge graph can be extracted using pre-trained graph neural networks to obtain a molecular embedding vector as a first feature vector of a modality. The structural information of the multimodal knowledge graph can be randomly initialized and updated during subsequent gradient backpropagation to obtain a structural embedding vector as a first feature vector of a modality.

[0060] S102, group each modality into pairs, and extract and replace the semantic information of the first feature vectors of the two modalities in any group to obtain the second feature vectors of each of the two modalities in any group.

[0061] In this embodiment of the disclosure, each modality is grouped into pairs. In some implementations, the text embedding vector and the molecular embedding vector can be used as the first feature vectors of two modalities within a group. The semantic information of the text embedding vector and the molecular embedding vector is extracted and replaced to obtain the second feature vectors of each of the two modalities.

[0062] Alternatively, the molecular embedding vector and the structural embedding vector can be used as the first feature vectors of two modalities within a set, and the semantic information of the molecular embedding vector and the structural embedding vector can be extracted and replaced to obtain the second feature vectors of each of the two modalities.

[0063] Alternatively, the structural embedding vector and the text embedding vector can be used as the first feature vectors of two modalities within a set, and semantic information can be extracted and replaced from the structural embedding vector and the text embedding vector to obtain the second feature vectors of each of the two modalities.

[0064] In this embodiment, semantic information can be extracted based on a collaborative attention mechanism to facilitate the extraction of common features among multiple modalities and the joint learning of important semantic information across modalities. After extracting semantic information from the first feature vectors of two modalities within any group, the extracted semantic information can be replaced to further reduce the differences between modalities. For example, redundant semantic information has little impact on the completion of the multimodal knowledge graph, so it can be replaced to facilitate the fusion of the second feature vectors of the two modalities in the next step.

[0065] S103, fuse the second feature vectors corresponding to each group to obtain the multimodal joint feature vector.

[0066] In some implementations, a bilinear function can be used to fuse the second feature vectors corresponding to each group to obtain a multimodal joint feature vector. To reduce computational cost and improve the efficiency of multimodal knowledge graph completion, a low-rank bilinear function can be used to fuse the second feature vectors corresponding to each group to obtain a multimodal joint feature vector.

[0067] S104, based on the multimodal joint feature vector, completes the multimodal knowledge graph to obtain the target multimodal knowledge graph.

[0068] In this embodiment of the disclosure, based on the multimodal joint feature vector, the tail entity corresponding to the head entity is connected and predicted. By judging the score of the triple, it is determined whether there is a relationship between the head entity and the predicted tail entity, and then the multimodal knowledge graph is completed to obtain the target multimodal knowledge graph.

[0069] In this embodiment, semantic information extraction and replacement of the first feature vectors of two modalities within any group can uncover common features between molecular structure information and textual description information. This allows for information exchange to reduce redundancy and enables cross-modal joint learning of important semantic information. Fusing the second feature vectors corresponding to each group reduces the differences between modalities. Based on the multimodal joint feature vectors, the multimodal knowledge graph is completed, fully utilizing multimodal information and improving the accuracy of knowledge graph completion.

[0070] Figure 2 This is a flowchart of a multimodal knowledge graph completion method according to an embodiment of this disclosure, such as... Figure 2 As shown, semantic information extraction and replacement are performed on the first feature vectors of two modalities within any group to obtain the second feature vectors of each of the two modalities in any group, including the following steps:

[0071] S201, perform semantic information extraction through collaborative attention on the first feature vectors of two modalities within any group to obtain the candidate feature vectors of each of the two modalities.

[0072] like Figure 3 As shown, to facilitate the extraction of common features among multiple modalities, this embodiment constructs a co-attention operator (CA). Taking a first feature vector as input, it applies scaling transformation and multi-head attention mechanisms to project the first feature vector into a new space, obtaining candidate feature vectors. It has a concise structure and can jointly learn important semantic information across modalities. The specific process is as follows:

[0073] Multiply the first eigenvectors of each of the two modes within any given set by their respective weight parameters to obtain the weighted eigenvectors of the two modes. Then, multiply the weighted eigenvectors of the two modes by their outer product to obtain the affinity matrix. Optionally, the affinity matrix can be obtained using the following formula:

[0074] A=σ(QW Q )·σ(D T W D )

[0075] Where A represents the affinity matrix, and d1 and d2 represent the dimensions characterized by the first eigenvectors of the two modalities within any set, representing a unified notation for the dimension of the data. σ represents the sigmoid function, used to map variables to the range 0 and 1. Q and D represent the first eigenvectors of the two modalities within any set, respectively. and Let Q and D represent the weight parameters of Q and D respectively. These weight parameters are trainable, and T represents the transpose of the matrix.

[0076] The affinity matrix is ​​scaled row-wise and column-wise to obtain the first and second matrices. Alternatively, the first and second matrices can be obtained using the following formulas:

[0077]

[0078]

[0079] Among them, A Q A D Let represent the first matrix and the second matrix, respectively. `softmax(...)` is the normalization exponential function used to map variables to values ​​in the (0,1) range, with the sum of the mapped values ​​being 1. `dim` represents the dimension in which the normalization exponential function operates. It should be noted that the first and second matrices have the same shape as the affinity matrix; the sum of each row in the first matrix is ​​1, and the sum of each column in the second matrix is ​​1.

[0080] Based on the first eigenvectors, the first matrix, and the second matrix of the two modalities, candidate eigenvectors for each modality are obtained. Alternatively, the following formula can be used to obtain the candidate eigenvectors for each modality:

[0081] Q ca =Q T ·A Q

[0082] D ca =A D ·D

[0083] Among them, Q ca D ca These represent the candidate feature vectors for each of the two modalities.

[0084] It should be noted that in multimodal biological knowledge graphs, information containing the same semantics may exist in different modalities, which is a common phenomenon. For example, the names of phenolic compounds usually have a suffix "-phine", and the names of piperazine-derived compounds have a suffix "-azine" (the name is a textual description of the compound). In the embodiments of this disclosure, the CA operator can be used to extract common semantic information between multimodalities.

[0085] For example, in an embodiment of this disclosure, the text is embedded into the vector h. t and molecular embedding vector h m Inputting the CA operator yields a set of candidate feature vectors. Embed the molecular vector h m and structure embedding vector h s Inputting the CA operator yields a set of candidate feature vectors. embed the structure vector h s and text embedding vector h t Inputting the CA operator yields a set of candidate feature vectors. in, This represents the trainable weights. d f d represents the dimension of the fused embedded representation. m d represents the dimension of the molecular embedding vector representation. t d represents the dimension of the text embedding vector representation. s The dimension of the structure embedding vector is represented by CA(...), and CA(...) represents the CA operator described in the above embodiments.

[0086] In this embodiment of the disclosure, the CA operator can be used to obtain information on the mutual reinforcement between the two modalities from explicit and implicit common semantic information.

[0087] S202: Select the target feature elements that need to be replaced with semantic information from the candidate feature vectors of the two modalities.

[0088] like Figure 4 As shown, different feature elements in the candidate feature vector have different attention weights. The smaller the attention weight, the less information the feature element provides. The information provided by a small attention weight is redundant and has little impact on the final result. Therefore, in this embodiment of the disclosure, after obtaining the candidate feature vector, semantic information replacement is performed on the candidate feature vector.

[0089] In this embodiment of the disclosure, the semantic information of each feature element in the candidate feature vector is obtained, and the feature element whose semantic information is less than a preset replacement factor is selected as the target feature element.

[0090] S203, determine the position of the target feature element in the candidate feature vector, perform semantic information replacement based on the position, and obtain the second feature vector.

[0091] For any given set, based on the position of the target feature element in one of the candidate feature vectors within the set, obtain the feature element to be replaced at the position in another candidate feature vector within the set, and replace the semantic information of the target feature element with the semantic information of the feature element to be replaced. For example... Figure 4 As shown, optionally, the second feature vector can be obtained using the following formula:

[0092] index = where(ln(Q) ca )<θ), Q ca [index] = D ca [index]

[0093] index = where(ln(D) ca)<θ), D ca [index] = Q ca [index]

[0094] In this context, the `index` function returns a value or a reference to a value in a table or range; `θ` represents a preset replacement factor; the magnitude of the semantic information feature of a feature element reflects the magnitude of its attention weight; `ln(...)` represents a layer normalization function, which facilitates the extraction of attention weights for feature elements in candidate feature vectors; and the `where(...)` function returns the index values ​​of all feature elements that meet the specified conditions. In this embodiment, the `where(...)` function returns the index values ​​of feature elements whose semantic information is less than the preset replacement factor. `θ` can take values ​​such as -1, -0.5, and 0.

[0095] For example, in an embodiment of this disclosure, After semantic information replacement, a second feature vector h is obtained. x1 ,h y1 ,Will After semantic information replacement, a second feature vector h is obtained. x2 ,h y2 ,Will After semantic information replacement, the second feature vector is obtained as h. x3 ,h y3 In other words, Wherein, EX() represents the semantic information replacement operation in the above embodiment, and i represents the i-th group. In this embodiment, i can be 1, 2, or 3.

[0096] In this embodiment, semantic information extraction and replacement of the first feature vectors of two modalities within any group can uncover common features between molecular structure information and textual description information. This allows for information exchange to reduce redundancy and enables cross-modal learning of important semantic information. This reduces the differences between modalities, facilitating the subsequent fusion of the second feature vectors corresponding to each group and improving the accuracy of knowledge graph completion.

[0097] like Figure 5 As shown, in some implementations, to enhance the diversity of the extracted semantic features, semantic information can be extracted and replaced multiple times in parallel on the first feature vectors of two modalities within any group, obtaining multiple candidate second feature vectors for the two modalities. Then, these multiple candidate second feature vectors are concatenated to obtain the second feature vectors for each of the two modalities. Optionally, the second feature vector can be obtained using the following formula: in, This represents a set of candidate second feature vectors obtained from the first semantic information extraction and replacement process. This represents a set of candidate second feature vectors obtained from the second semantic information extraction and replacement process. W represents a set of candidate second feature vectors obtained from the m-th semantic information extraction and replacement process. xi W yi represents the trainable weight parameters, and & represents concatenating vectors in a given dimension.

[0098] In this embodiment, the multimodal joint feature vector is obtained by fusing the second feature vector corresponding to each group using a low-rank bilinear function. The second feature vector is multiplied by the matrix of a preset linear mapping and normalized to obtain the third feature vector. Element-wise multiplication is performed between the third feature vectors of two modalities to obtain the fourth feature vector corresponding to any group. Element-wise multiplication is then performed on the fourth feature vector of each group to obtain the multimodal joint feature vector. Optionally, the multimodal joint feature vector can be obtained using the following formula:

[0099]

[0100] Among them, h f Represents the joint feature vector of multimodal features. ψ represents element-wise multiplication of two vectors, while ψ represents element-wise multiplication of a sequence of vectors. xi T yi Let b represent the matrix of the predefined linear mapping, and let U represent the error. xi T yi b and b are both trainable parameters.

[0101] In this embodiment, semantic information extraction and replacement of the first feature vectors of two modalities within any group can uncover common features between molecular structure information and textual description information. This allows for information exchange to reduce redundancy and cross-modal learning of important semantic information. Fusion of the second feature vectors corresponding to each group reduces the differences between modalities, fully utilizes multimodal information, and improves the accuracy of knowledge graph completion.

[0102] Figure 6 This is a flowchart of a multimodal knowledge graph completion method according to an embodiment of this disclosure, such as... Figure 6 As shown, based on the multimodal joint feature vector, the multimodal knowledge graph is completed to obtain the target multimodal knowledge graph, including the following steps:

[0103] S601, obtain the head entity embedding vector and relation embedding vector of the multimodal knowledge graph.

[0104] S602 predicts the tail entity corresponding to the head entity of the multimodal knowledge graph based on the multimodal joint feature vector, head entity embedding vector, and relation embedding vector.

[0105] Optionally, the multimodal joint feature vector, head entity embedding vector, and relation embedding vector can be superimposed to generate a multimodal joint feature map. This multimodal joint feature map is a multi-channel feature map. The multimodal joint feature map is then fed into a convolutional layer to obtain the target feature map. After flattening the target feature map, it is fed into a fully connected layer to obtain the tail entity embedding vector and the entity embedding vectors of the candidate entities in the multimodal knowledge graph. The tail entity embedding vector is then multiplied by the entity embedding vectors to calculate their similarity. The entities are then sorted according to their similarity to determine the tail entity corresponding to the head entity from the candidate entities.

[0106] Optionally, candidate entities can be all entities in a multimodal knowledge graph.

[0107] S603, based on the tail entity corresponding to the head entity, completes the multimodal knowledge graph to obtain the target multimodal knowledge graph.

[0108] The details of step S603 can be found in the relevant descriptions of the above embodiments, and will not be repeated here.

[0109] In this embodiment, multimodal information is fully utilized to improve the efficiency and accuracy of knowledge graph completion, and the problems of being unable to adapt to molecular structure information and unable to explicitly utilize the shared features between modes are solved, which can greatly improve the performance of multimodal knowledge graph completion.

[0110] Figure 7 This is a schematic diagram of a multimodal knowledge graph completion method according to an embodiment of this disclosure, such as... Figure 7 As shown in this embodiment, feature extraction is performed on the multimodal knowledge graph to obtain the molecular embedding vector h. m , structural embedding vector h s and text embedding vector h t , will h m h s and h t The modalities are paired, and multiple CA operators are input for semantic information extraction. The outputs of the CA operators are then semantically replaced and concatenated to form the second feature vectors for each modality. A low-rank bilinear function is used to fuse the second feature vectors corresponding to each pair to obtain the multimodal joint feature vector h. f And combined with the head entity embedding vector e h A multimodal joint feature map is generated from the relation embedding vector r. This multimodal joint feature map is then input into a convolutional layer and a fully connected layer to predict the tail entity e corresponding to the head entity. t .

[0111] In this embodiment, common features between molecular structure information and text description information can be mined, and redundancy can be reduced through information exchange. Important semantic information can be jointly learned across modalities, reducing the differences between modalities. Based on the multimodal joint feature vector, the multimodal knowledge graph can be completed, making full use of multimodal information and improving the accuracy of knowledge graph completion.

[0112] Figure 8 This is a flowchart of a training method for a multimodal knowledge graph completion model according to an embodiment of this disclosure, such as... Figure 8 As shown, the method includes the following steps:

[0113] S801, obtain the multimodal knowledge graph of the samples and their corresponding sample triples.

[0114] In this embodiment of the disclosure, any triple includes a head entity, a tail entity, and the relationship between the head entity and the tail entity. That is, the sample triple corresponding to the sample multimodal knowledge graph includes a sample head entity, a sample tail entity, and the relationship between the sample head entity and the sample tail entity.

[0115] S802 trains a multimodal knowledge graph completion model based on the sample multimodal knowledge graph to obtain the completed multimodal knowledge graph of the sample multimodal knowledge graph, and the predicted triples of the completed multimodal knowledge graph.

[0116] The multimodal knowledge graph completion model in this embodiment can be found in the relevant content of the multimodal knowledge graph completion method in the above embodiments, and will not be repeated here.

[0117] It should be noted that the multimodal knowledge graph completion model in this embodiment can complete the sample multimodal knowledge graph according to the multimodal knowledge graph completion method in the above embodiment, output the completed multimodal knowledge graph of the sample multimodal knowledge graph, and then obtain the predicted triples based on the completed multimodal knowledge graph.

[0118] In some implementations, the sample multimodal knowledge graph and sample triples are used to form positive samples. The multimodal knowledge graph completion model is then trained based on these positive samples to obtain the completed multimodal knowledge graph of the sample multimodal knowledge graph.

[0119] In some implementations, based on the sample triples of the sample multimodal knowledge graph, the inverse triples of the sample multimodal knowledge graph are obtained as another sample triple of the sample multimodal knowledge graph. The sample multimodal knowledge graph and the sample triples are combined to form positive samples, and the sample multimodal knowledge graph and the inverse triples are combined to form negative samples. Based on the positive and negative samples, the multimodal knowledge graph completion model is trained to obtain the completed multimodal knowledge graph of the sample multimodal knowledge graph.

[0120] S803: Adjust the multimodal knowledge graph completion model based on the sample triples and the predicted triples, and return to continue training using the next sample multimodal knowledge graph until the training ends and the target multimodal knowledge graph completion model is obtained.

[0121] In some implementations, sample triples and predicted triples are matched, the multimodal knowledge graph completion model is adjusted based on the matching results, and the process returns to continue training using the next sample multimodal knowledge graph until the training ends and the target multimodal knowledge graph completion model is obtained.

[0122] In some implementations, sample tail entities are obtained from sample triples, and predicted tail entities are obtained from predicted triples. The loss between the sample tail entities and the predicted tail entities is calculated. The multimodal knowledge graph completion model is adjusted based on this loss, and training continues using the next sample multimodal knowledge graph until training is complete and the target multimodal knowledge graph completion model is obtained. Optionally, a negative log-likelihood loss function can be used to obtain the loss between the sample tail entities and the predicted tail entities. Then, the parameters in the multimodal knowledge graph completion model are trained by minimizing the Bernoulli negative log-likelihood loss function, as shown below:

[0123]

[0124] Where n is the number of negative samples, q∈ n It is a true label, p∈ n This involves applying the sigmoid function to the predicted probability generated by Ψ.

[0125] In this embodiment, a multimodal knowledge graph is completed based on a multimodal joint feature vector, making full use of multimodal information. The multimodal knowledge graph completion model is trained based on positive and negative samples, which improves the accuracy of knowledge graph completion and solves the problems of not being able to adapt to molecular structure information and not being able to explicitly utilize the features shared between modalities. This can greatly improve the performance of multimodal knowledge graph completion.

[0126] Based on the same disclosed concept, this disclosure proposes another method for completing multimodal knowledge graphs. This method includes: inputting the multimodal knowledge graph to be completed into a trained multimodal knowledge graph completion model to obtain the completed target multimodal knowledge graph. In other words, the trained multimodal knowledge graph completion model can be found in [reference needed]. Figure 9The feature extraction layer in the model extracts features from the multimodal knowledge graph to be completed, obtaining the first feature vector of each modality. Each modality is then grouped into pairs, and the first feature vectors of the two modalities in any group are input into the CA operator layer to output the candidate feature vector of any group. Further, the candidate feature vector of any group is input into the semantic replacement layer to obtain the second feature vector of each of the two modalities in any group. Then, all the second feature vectors are input into the feature fusion layer to obtain the multimodal joint feature vector. Finally, the multimodal joint feature vector is input into the graph completion layer to output the completed multimodal knowledge graph.

[0127] In this embodiment, a multimodal knowledge graph is completed based on a multimodal joint feature vector, making full use of multimodal information. The multimodal knowledge graph completion model is trained based on positive and negative samples, which improves the accuracy of knowledge graph completion and solves the problems of not being able to adapt to molecular structure information and not being able to explicitly utilize the features shared between modalities. This can greatly improve the performance of multimodal knowledge graph completion.

[0128] Figure 10 This is a structural diagram of a multimodal knowledge graph completion device according to an embodiment of this disclosure, as follows: Figure 10 As shown, the multimodal knowledge graph completion device 1000 includes:

[0129] The first acquisition module 1010 is used to extract features from the multimodal knowledge graph and obtain the first feature vector of each modality.

[0130] The second acquisition module 1020 is used to group each modality into pairs, and extract and replace the semantic information of the first feature vectors of the two modalities in any group to obtain the second feature vectors of each of the two modalities in any group.

[0131] The third acquisition module 1030 is used to fuse the second feature vectors corresponding to each group to obtain a multimodal joint feature vector;

[0132] The completion acquisition module 1040 is used to complete the multimodal knowledge graph based on the multimodal joint feature vector and obtain the target multimodal knowledge graph.

[0133] In some implementations, the second acquisition module 1020 is further configured to: extract semantic information from the first feature vectors of two modalities within any group using collaborative attention, and obtain candidate feature vectors for each of the two modalities; select target feature elements that need to be semantically replaced from the candidate feature vectors of the two modalities; determine the position of the target feature element in the candidate feature vectors, and perform semantic information replacement at the same position based on the position, and obtain the second feature vector.

[0134] In some implementations, the second acquisition module 1020 is further configured to: multiply the first feature vectors of the two modes in any group by their respective weight parameters to obtain the weighted feature vectors of the two modes, and multiply the weighted feature vectors of the two modes by a cross product to obtain an affinity matrix; perform scaling transformations on the affinity matrix by rows and columns to obtain a first matrix and a second matrix; and obtain the candidate feature vectors of the two modes based on the first feature vectors, the first matrix, and the second matrix.

[0135] In some implementations, the second acquisition module 1020 is also used to: acquire the semantic information of each feature element in the candidate feature vector; and select the feature element whose semantic information is less than a preset replacement factor as the target feature element.

[0136] In some implementations, the second acquisition module 1020 is further configured to: for any group, based on the position of the target feature element in one of the candidate feature vectors in the group, acquire the feature element to be replaced at the position in another candidate feature vector in the group; and replace the semantic information of the target feature element with the semantic information of the feature element to be replaced.

[0137] In some implementations, the second acquisition module 1020 is also used to: extract and replace semantic information multiple times in parallel on the first feature vectors of two modalities in any group to obtain multiple candidate second feature vectors of the two modalities; and concatenate the multiple candidate second feature vectors to obtain the second feature vectors of each of the two modalities.

[0138] In some implementations, the third acquisition module 1030 is also used to: multiply the second feature vector with the matrix of the preset linear mapping and normalize it to obtain the third feature vector; multiply the third feature vectors of the two modalities element by element to obtain the fourth feature vector corresponding to any group; and multiply the fourth feature vectors of each group element by element to obtain the multimodal joint feature vector.

[0139] In some implementations, the completion module 1040 is also used to: obtain the head entity embedding vector and relation embedding vector of the multimodal knowledge graph; predict the tail entity corresponding to the head entity of the multimodal knowledge graph based on the multimodal joint feature vector, the head entity embedding vector and the relation embedding vector; and complete the multimodal knowledge graph based on the tail entity corresponding to the head entity to obtain the target multimodal knowledge graph.

[0140] In some implementations, the completion module 1040 is also used to: generate a multimodal joint feature map based on the multimodal joint feature vector, the head entity embedding vector, and the relation embedding vector; and predict the tail entities of the head entities of the multimodal knowledge graph based on the multimodal joint feature map to obtain the tail entities corresponding to the head entities.

[0141] In some implementations, the completion module 1040 is also used to: input the multimodal joint feature map into a convolutional layer to obtain the target feature map; input the target feature map into a fully connected layer after flattening to obtain the tail entity embedding vector; and obtain the tail entity corresponding to the head entity based on the tail entity embedding vector.

[0142] In some implementations, the completion module 1040 is also used to: obtain the entity embedding vectors of candidate entities in a multimodal knowledge graph; obtain the similarity between the entity embedding vectors and the tail entity embedding vectors; and determine the tail entity corresponding to the head entity from the candidate entities based on the similarity.

[0143] In this embodiment, semantic information extraction and replacement of the first feature vectors of two modalities within any group can uncover common features between molecular structure information and textual description information. This allows for information exchange to reduce redundancy and enables cross-modal joint learning of important semantic information. Fusing the second feature vectors corresponding to each group reduces the differences between modalities. Based on the multimodal joint feature vectors, the multimodal knowledge graph is completed, fully utilizing multimodal information and improving the accuracy of knowledge graph completion.

[0144] Figure 11 This is a structural diagram of a training device for a multimodal knowledge graph completion model according to an embodiment of this disclosure, as shown below. Figure 11 As shown, the training device 1100 for the multimodal knowledge graph completion model includes:

[0145] The first acquisition module 1110 is used to acquire the sample multimodal knowledge graph and its corresponding sample triples;

[0146] The second acquisition module 1120 is used to train a multimodal knowledge graph completion model based on the sample multimodal knowledge graph to obtain the completed multimodal knowledge graph of the sample multimodal knowledge graph and the predicted triples of the completed multimodal knowledge graph.

[0147] The third acquisition module 1130 is used to adjust the multimodal knowledge graph completion model based on the sample triples and the predicted triples, and return to continue training using the next sample multimodal knowledge graph until the training ends and the target multimodal knowledge graph completion model is obtained; wherein, any triple includes a head entity, a tail entity, and the relationship between the head entity and the tail entity.

[0148] In some implementations, the third acquisition module 1130 is also used to: acquire sample tail entities from sample triples and predict tail entities from predicted triples; acquire the loss between sample tail entities and predicted tail entities; and adjust the multimodal knowledge graph completion model based on the loss.

[0149] In some implementations, the second acquisition module 1020 is also used to: obtain the inverse triples of the sample multimodal knowledge graph based on the sample triples of the sample multimodal knowledge graph, and use them as another sample triples of the sample multimodal knowledge graph; construct positive samples by combining the sample multimodal knowledge graph and the sample triples; construct negative samples by combining the sample multimodal knowledge graph and the inverse triples; and train the multimodal knowledge graph completion model based on the positive and negative samples.

[0150] In some implementations, the second acquisition module 1020 is further configured to: input the sample multimodal knowledge graph into the multimodal knowledge graph completion model; extract features from the sample multimodal knowledge graph using the multimodal knowledge graph completion model to obtain the first feature vector of each modality; group each modality into pairs, and extract and replace semantic information from the first feature vectors of the two modalities within any group to obtain the second feature vectors of each of the two modalities in any group; fuse the second feature vectors corresponding to each group to obtain the multimodal joint feature vector; and output the completed multimodal knowledge graph of the sample multimodal knowledge graph based on the multimodal joint feature vector.

[0151] In this embodiment, a multimodal knowledge graph is completed based on a multimodal joint feature vector, making full use of multimodal information. The multimodal knowledge graph completion model is trained based on positive and negative samples, which improves the accuracy of knowledge graph completion and solves the problems of not being able to adapt to molecular structure information and not being able to explicitly utilize the features shared between modalities. This can greatly improve the performance of multimodal knowledge graph completion.

[0152] Figure 12 This is a structural diagram of a multimodal knowledge graph completion device according to an embodiment of this disclosure, as follows: Figure 12 As shown, the multimodal knowledge graph completion device 1200 includes:

[0153] The completion module 1210 is used to input the multimodal knowledge graph to be completed into the trained multimodal knowledge graph completion model to obtain the completed target multimodal knowledge graph.

[0154] The multimodal knowledge graph completion model is trained using the same training device as described above.

[0155] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0156] In this embodiment, semantic information extraction and replacement of the first feature vectors of two modalities within any group can uncover common features between molecular structure information and textual description information. This allows for information exchange to reduce redundancy and enables cross-modal joint learning of important semantic information. Fusing the second feature vectors corresponding to each group reduces the differences between modalities. Based on the multimodal joint feature vectors, the multimodal knowledge graph is completed, fully utilizing multimodal information and improving the accuracy of knowledge graph completion.

[0157] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0158] Figure 13 A schematic block diagram of an example electronic device 1300 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0159] like Figure 13 As shown, device 1300 includes a computing unit 1301, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 1302 or a computer program loaded from storage unit 1308 into random access memory (RAM) 1303. The RAM 1303 may also store various programs and data required for the operation of device 1300. The computing unit 1301, ROM 1302, and RAM 1303 are interconnected via bus 1304. Input / output (I / O) interface 1305 is also connected to bus 1304.

[0160] Multiple components in device 1300 are connected to I / O interface 1305, including: input unit 1306, such as keyboard, mouse, etc.; output unit 1307, such as various types of monitors, speakers, etc.; storage unit 1308, such as disk, optical disk, etc.; and communication unit 1309, such as network card, modem, wireless transceiver, etc. Communication unit 1309 allows device 1300 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0161] The computing unit 1301 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1301 performs the various methods and processes described above, such as the steps of a method for completing a multimodal knowledge graph or a method for training a multimodal knowledge graph completion model. For example, in some embodiments, the steps of a method for completing a multimodal knowledge graph or the method for training a multimodal knowledge graph completion model can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1308. In some embodiments, part or all of the computer program can be loaded and / or installed on device 1300 via ROM 1302 and / or communication unit 1309. When the computer program is loaded into RAM 1303 and executed by computing unit 1301, it can perform one or more steps of the multimodal knowledge graph completion method or the multimodal knowledge graph completion model training method described above. Alternatively, in other embodiments, computing unit 1301 can be configured by any other suitable means (e.g., by means of firmware) to perform the steps of the multimodal knowledge graph completion method or the multimodal knowledge graph completion model training method.

[0162] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0163] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0164] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0165] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0166] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0167] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0168] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0169] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for completing a multi-modal knowledge graph, comprising: extracting features of a multi-modal knowledge graph to obtain first feature vectors of each modality, wherein the multi-modal knowledge graph is a multi-modal biological knowledge graph, and the modalities of the multi-modal biological knowledge graph include a text information modality, a molecular information modality, and a structure information modality; grouping each modality in pairs, and extracting and replacing semantic information of the first feature vectors of two modalities in any group to obtain second feature vectors of the two modalities in the any group respectively; fusing the second feature vectors corresponding to each group to obtain a multi-modal joint feature vector; completing the multi-modal knowledge graph based on the multi-modal joint feature vector to obtain a target multi-modal knowledge graph; wherein the extracting and replacing semantic information of the first feature vectors of two modalities in any group to obtain second feature vectors of the two modalities in the any group respectively comprises: extracting semantic information of the first feature vectors of the two modalities in the any group by collaborative attention to obtain candidate feature vectors of the two modalities respectively; screening target feature elements requiring semantic information replacement from the candidate feature vectors of the two modalities, wherein the target feature elements are feature elements whose semantic information is less than a preset replacement factor in each feature element of the candidate feature vectors; determining the positions of the target feature elements in the candidate feature vectors, and replacing semantic information at the same positions based on the positions to obtain the second feature vectors.

2. The method of claim 1, wherein, The extracting semantic information of the first feature vectors of the two modalities in the any group by collaborative attention to obtain candidate feature vectors of the two modalities respectively comprises: multiplying the first feature vectors of the two modalities in the any group by respective weight parameters to obtain weighted feature vectors of the two modalities respectively, and performing vector outer product multiplication on the weighted feature vectors of the two modalities to obtain an affinity matrix; performing stretching transformation on the affinity matrix by row and column to obtain a first matrix and a second matrix; obtaining candidate feature vectors of the two modalities respectively according to the first feature vectors of the two modalities, the first matrix, and the second matrix.

3. The method of claim 1, wherein, The screening target feature elements requiring semantic information replacement from the candidate feature vectors of the two modalities comprises: obtaining semantic information of each feature element in the candidate feature vectors; selecting feature elements whose semantic information is less than a preset replacement factor as the target feature elements.

4. The method of claim 1, wherein, The replacing semantic information at the same positions based on the positions comprises: for the any group, obtaining a feature element to be replaced in another candidate feature vector in the group based on the position of the target feature element in one candidate feature vector in the group; and replacing semantic information of the target feature element with semantic information of the feature element to be replaced.

5. The method of claim 1, wherein, The fusing the second feature vectors corresponding to each group to obtain a multi-modal joint feature vector comprises: Multiplying the second feature vector with a matrix of a preset linear mapping and performing normalization processing to obtain a third feature vector; Element-wise multiplication is performed between the third feature vectors of the two modalities to obtain a fourth feature vector corresponding to any group; Element-wise multiplication is performed on the fourth feature vectors of each group to obtain the multi-modal joint feature vector.

6. The method according to any one of claims 1 to 5, wherein, Based on the multi-modal joint feature vector, the multi-modal knowledge graph is completed to obtain a target multi-modal knowledge graph, comprising: Obtain the head entity embedding vector and the relation embedding vector of the multi-modal knowledge graph; According to the multi-modal joint feature vector, the head entity embedding vector and the relation embedding vector, the tail entity corresponding to the head entity of the multi-modal knowledge graph is predicted. Based on the tail entity corresponding to the head entity, the multi-modal knowledge graph is completed to obtain the target multi-modal knowledge graph.

7. The method of claim 6, wherein, According to the multi-modal joint feature vector, the head entity embedding vector and the relation embedding vector, the tail entity corresponding to the head entity of the multi-modal knowledge graph is predicted. According to the multi-modal joint feature vector, the head entity embedding vector and the relation embedding vector, a multi-modal joint feature map is generated. According to the multi-modal joint feature map, the tail entity prediction of the multi-modal knowledge graph head entity is performed to obtain the tail entity corresponding to the head entity.

8. The method of claim 7, wherein, According to the multi-modal joint feature map, the tail entity prediction of the multi-modal knowledge graph head entity is performed to obtain the tail entity corresponding to the head entity. The multi-modal joint feature map is input into a convolution layer to obtain a target feature map. The target feature map is flattened and input into a fully connected layer to obtain a tail entity embedding vector. According to the tail entity embedding vector, the tail entity corresponding to the head entity is obtained.

9. The method of claim 8, wherein, According to the tail entity embedding vector, the tail entity corresponding to the head entity is obtained. Obtain the entity embedding vector of the candidate entity of the multi-modal knowledge graph; Obtain the similarity of the entity embedding vector and the tail entity embedding vector; According to the similarity, the tail entity corresponding to the head entity is determined from the candidate entity.

10. A training method of a multi-modal knowledge graph completion model, comprising: Obtain a sample multi-modal knowledge graph and its corresponding sample triple, the multi-modal knowledge graph is a multi-modal biological knowledge graph, the modalities of the multi-modal biological knowledge graph include: text information modality, molecular information modality and structure information modality; Input the sample multi-modal knowledge graph into the multi-modal knowledge graph completion model; Feature extraction is performed on the sample multi-modal knowledge graph by the multi-modal knowledge graph completion model to obtain first feature vectors of each modality; Each modality is grouped two by two, and semantic information extraction and replacement are performed on the first feature vectors of two modalities in any group to obtain second feature vectors of the two modalities in any group; The second feature vectors corresponding to each group are fused to obtain a multi-modal joint feature vector; outputting a completed multi-modal knowledge graph of the sample multi-modal knowledge graph based on the multi-modal joint feature vector, to obtain a completed multi-modal knowledge graph of the sample multi-modal knowledge graph and a predicted triple of the completed multi-modal knowledge graph; adjusting the multi-modal knowledge graph completion model according to the sample triple and the predicted triple, and returning to continue training using a next sample multi-modal knowledge graph until training is completed to obtain a target multi-modal knowledge graph completion model; wherein any triple includes a head entity, a tail entity, and a relationship between the head entity and the tail entity; wherein the semantic information extraction and replacement of the first feature vectors of the two modalities in any group obtains respective second feature vectors of the two modalities in the any group, including: performing semantic information extraction of the first feature vectors of the two modalities in the any group by using collaborative attention, to obtain respective candidate feature vectors of the two modalities; from the candidate feature vectors of the two modalities, filtering out target feature elements that need to be replaced by semantic information, the target feature elements being feature elements whose semantic information is less than a preset replacement factor in the candidate feature vectors; determining the positions of the target feature elements in the candidate feature vectors, and performing same-position semantic information replacement based on the positions to obtain the second feature vectors.

11. The method of claim 10, wherein, The adjusting of the multi-modal knowledge graph completion model according to the sample triple and the predicted triple includes: obtaining a sample tail entity from the sample triple, and obtaining a predicted tail entity from the predicted triple; obtaining a loss between the sample tail entity and the predicted tail entity; adjusting the multi-modal knowledge graph completion model according to the loss.

12. The method of claim 10, wherein, The training of the multi-modal knowledge graph completion model based on the sample multi-modal knowledge graph further includes: based on a sample triple of the sample multi-modal knowledge graph, obtaining an inverse triple of the sample multi-modal knowledge graph as another sample triple of the sample multi-modal knowledge graph; constructing a positive sample by using the sample multi-modal knowledge graph and the sample triple; constructing a negative sample by using the sample multi-modal knowledge graph and the inverse triple; training the multi-modal knowledge graph completion model based on the positive sample and the negative sample.

13. A method for completing a multi-modal knowledge graph, comprising: inputting a multi-modal knowledge graph to be completed into a trained multi-modal knowledge graph completion model, to obtain a completed target multi-modal knowledge graph; wherein the multi-modal knowledge graph completion model is trained by using the training method of any one of claims 10-12.

14. An apparatus for completing a multi-modal knowledge graph, comprising: a first obtaining module configured to perform feature extraction on a multi-modal knowledge graph to obtain first feature vectors of respective modalities, the multi-modal knowledge graph being a multi-modal biological knowledge graph, and the modalities of the multi-modal biological knowledge graph including a text information modality, a molecular information modality, and a structure information modality. The second obtaining module is configured to group each modality in pairs, and perform semantic information extraction and replacement on the first feature vectors of two modalities in any group to obtain second feature vectors of the two modalities in the any group respectively; The third obtaining module is configured to fuse the second feature vectors corresponding to each group to obtain a multi-modal joint feature vector; The completion obtaining module is configured to complete the multi-modal knowledge graph based on the multi-modal joint feature vector to obtain a target multi-modal knowledge graph; The second obtaining module is further configured to: perform semantic information extraction on the first feature vectors of the two modalities in the any group by using collaborative attention to obtain candidate feature vectors of the two modalities respectively; select a target feature element that needs to be replaced in terms of semantic information from the candidate feature vectors of the two modalities, the target feature element being a feature element whose semantic information is less than a preset replacement factor in each candidate feature vector; determine the position of the target feature element in the candidate feature vector, and perform same-position semantic information replacement based on the position to obtain the second feature vector.

15. The apparatus of claim 14, wherein, The second obtaining module is further configured to: multiply the first feature vectors of the two modalities in the any group by respective weight parameters to obtain weighted feature vectors of the two modalities respectively, and perform vector outer product multiplication on the weighted feature vectors of the two modalities to obtain an affinity matrix; perform stretching transformation on the affinity matrix by row and column respectively to obtain a first matrix and a second matrix; obtain candidate feature vectors of the two modalities respectively according to the first feature vectors of the two modalities, the first matrix, and the second matrix.

16. The apparatus of claim 14, wherein, The second obtaining module is further configured to: obtain semantic information of each feature element in the candidate feature vector; select a feature element whose semantic information is less than a preset replacement factor as the target feature element.

17. The apparatus of claim 14, wherein, The second obtaining module is further configured to: for the any group, obtain a feature element to be replaced in the same position in another candidate feature vector in the group based on the position of the target feature element in one of the candidate feature vectors in the group; replace the semantic information of the target feature element with the semantic information of the feature element to be replaced.

18. The apparatus of claim 14, wherein, The second obtaining module is further configured to: perform multiple semantic information extraction and replacement on the first feature vectors of the two modalities in the any group in parallel to obtain multiple candidate second feature vectors of the two modalities; splice the multiple candidate second feature vectors to obtain second feature vectors of the two modalities respectively.

19. The apparatus of claim 14, wherein, The third obtaining module is further configured to: multiply the second feature vectors by a matrix of a preset linear mapping and perform normalization processing to obtain third feature vectors; perform element-by-element multiplication between the third feature vectors of the two modalities to obtain fourth feature vectors corresponding to the any group; perform element-by-element multiplication on the fourth feature vectors of each group to obtain the multi-modal joint feature vector.

20. The apparatus of any of claims 14-19, wherein, The completion obtaining module is further configured to: obtain head entity embedding vectors and relation embedding vectors of the multi-modal knowledge graph; predict a tail entity corresponding to the head entity of the multi-modal knowledge graph according to the multi-modal joint feature vector, the head entity embedding vector and the relation embedding vector; complete the multi-modal knowledge graph based on the tail entity corresponding to the head entity, and obtain the target multi-modal knowledge graph.

21. The apparatus of claim 20, wherein, The completion obtaining module is further configured to: generate a multi-modal joint feature map according to the multi-modal joint feature vector, the head entity embedding vector and the relation embedding vector; perform tail entity prediction on the head entity of the multi-modal knowledge graph according to the multi-modal joint feature map, and obtain the tail entity corresponding to the head entity.

22. The apparatus of claim 21, wherein, The completion obtaining module is further configured to: input the multi-modal joint feature map into a convolution layer to obtain a target feature map; input the target feature map into a fully connected layer after flattening operation to obtain a tail entity embedding vector; obtain the tail entity corresponding to the head entity according to the tail entity embedding vector.

23. The apparatus of claim 22, wherein, The completion obtaining module is further configured to: obtain an entity embedding vector of a candidate entity of the multi-modal knowledge graph; obtain a similarity between the entity embedding vector and the tail entity embedding vector; determine the tail entity corresponding to the head entity from the candidate entity according to the similarity.

24. A training device of a multi-modal knowledge graph completion model, comprising: a first obtaining module configured to obtain a sample multi-modal knowledge graph and a sample triple corresponding thereto, the multi-modal knowledge graph being a multi-modal biological knowledge graph, and the modalities of the multi-modal biological knowledge graph including a text information modality, a molecular information modality and a structure information modality; a second obtaining module configured to input the sample multi-modal knowledge graph into the multi-modal knowledge graph completion model, extract features of the sample multi-modal knowledge graph by the multi-modal knowledge graph completion model to obtain first feature vectors of the modalities, group the modalities two by two, extract and replace the first feature vectors of two modalities in any group to obtain second feature vectors of the two modalities in the any group respectively, fuse the second feature vectors corresponding to each group to obtain a multi-modal joint feature vector, and output a completed multi-modal knowledge graph of the sample multi-modal knowledge graph based on the multi-modal joint feature vector to obtain a completed multi-modal knowledge graph of the sample multi-modal knowledge graph and a predicted triple of the completed multi-modal knowledge graph; a third obtaining module configured to adjust the multi-modal knowledge graph completion model according to the sample triple and the predicted triple, and continue training using a next sample multi-modal knowledge graph until a target multi-modal knowledge graph completion model is obtained; wherein any triple includes a head entity, a tail entity and a relation between the head entity and the tail entity. The second obtaining module is further configured to perform semantic information extraction on the first feature vector of the two modalities in the any group by using a cooperative attention mechanism, to obtain a candidate feature vector of each of the two modalities; filter a target feature element that needs to be replaced by semantic information from the candidate feature vector of the two modalities, the target feature element being a feature element whose semantic information is less than a preset replacement factor in the candidate feature vector; determine a position of the target feature element in the candidate feature vector, and perform a same-position semantic information replacement based on the position, to obtain the second feature vector.

25. The apparatus of claim 24, wherein, The third obtaining module is further configured to: obtain a sample tail entity from the sample triple, and obtain a predicted tail entity from the predicted triple; obtain a loss between the sample tail entity and the predicted tail entity; adjust the multi-modal knowledge graph completion model according to the loss.

26. The apparatus of claim 24, wherein, The second obtaining module is further configured to: obtain, based on a sample triple of the sample multi-modal knowledge graph, an inverse triple of the sample multi-modal knowledge graph as another sample triple of the sample multi-modal knowledge graph; construct a positive sample by using the sample multi-modal knowledge graph and the sample triple; construct a negative sample by using the sample multi-modal knowledge graph and the inverse triple; train the multi-modal knowledge graph completion model based on the positive sample and the negative sample. 27.A multi-modal knowledge graph completion device, comprising: a completion module configured to input a multi-modal knowledge graph to be completed into a trained multi-modal knowledge graph completion model, to obtain a target multi-modal knowledge graph after completion; wherein the multi-modal knowledge graph completion model is trained by using the training device according to any one of claims 24-26. 28.An electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication;wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-9 or claims 10-12 or claim 13.

29. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to make the computer execute the method according to any one of claims 1-9 or claims 10-12 or claim 13. 30.A computer program product comprising a computer program which, when executed by a processor, implements the steps of the method according to any one of claims 1-9 or claims 10-12 or claim 13.

Images