Method and device for multi-modal knowledge graph completion based on graph neural network and adversarial learning

Abstract

The application discloses a kind of based on graph neural network and multi-modal knowledge graph completion method and device of learning of confrontation, comprising: the triple data in multi-modal knowledge graph is converted into heterogeneous graph, while the feature of entity image information is extracted to obtain original entity modal characteristic;Learning model comprising heterogeneous graph convolution network, modal feature generator and modal feature discriminator is constructed;After learning model is trained using the way of confrontation learning, modal feature generator and modal feature discriminator are used to generate entity modal characteristic for each entity;Entity modal characteristic and entity structure characteristic are used as the initialization result of multi-modal knowledge graph completion model, and multi-modal knowledge graph completion model is trained based on positive and negative sample comparison;Using the entity representation and relationship representation trained in multi-modal knowledge graph completion model and defined scoring function, score is carried out for candidate triple, and based on the scoring result, knowledge graph is completed, to improve the accuracy of completion.

Description

Technical Field

[0001] This invention belongs to the field of knowledge graph representation learning, specifically relating to a multimodal knowledge graph completion method and apparatus based on graph neural networks and adversarial learning. Background Technology

[0002] Knowledge graphs model human knowledge into structured triples such as (head entity, relation, tail entity). These relation triples represent a relationship between the head and tail entities, and a vast number of triples form a semantically rich knowledge graph. Knowledge graphs are an important research area in artificial intelligence, playing a role in various AI scenarios and serving as a crucial link in the transition from perceptual intelligence to cognitive intelligence.

[0003] With the development of knowledge graph-related technologies, information from multiple modalities, such as images and text descriptions, has been introduced into knowledge graphs, and these knowledge graphs are called multimodal knowledge graphs. Multimodal knowledge graph completion technology has flourished in recent years. For example, patent application CN115186110A discloses a multimodal knowledge graph completion method and system based on relation enhancement, and patent application 114564593A discloses a multimodal knowledge graph completion method based on collaborative attention.

[0004] Although knowledge graphs store a vast amount of structured knowledge and information, they generally suffer from incomplete information. This is because knowledge graphs are often constructed manually (e.g., through expert knowledge) or automatically (e.g., through triple extraction), containing only the facts we have observed. Hidden relationships between the entities corresponding to these observed facts often remain undiscovered. This necessitates methods to uncover these unobserved triples within the existing knowledge graph; this task is known as knowledge graph completion. Traditional knowledge graph completion methods are inspired by word vectorization techniques, modeling triple structures as operations such as translation, rotation, and dot product between vectors. They represent entities and relationships in the knowledge graph using vectorization and define a scoring function to calculate the correctness of a triple. Classic methods include TransE, TransH, RotatE, HAKE, and PairRE, among others.

[0005] On the other hand, knowledge graphs built in real-world production and daily life scenarios, such as product knowledge graphs in e-commerce and social knowledge graphs on social platforms, often suffer from missing entity modal information. For example, some entities may lack corresponding image information (e.g., in a product knowledge graph in e-commerce, some products may not have uploaded corresponding images, resulting in missing image information for the entity corresponding to that product). This lack of modal information affects the expressive power of the knowledge graph and impacts various knowledge graph-related tasks, such as knowledge graph completion. Therefore, completing the missing modal information of entities in knowledge graphs plays an indispensable role in the industrial application and implementation of multimodal knowledge graphs.

[0006] Existing multimodal knowledge graph completion models often project the multimodal information of entities into the same representation space as the entity's structural information. They then utilize multiple combined scoring functions to score a triple based on both the entity's structural and multimodal information, using the triple's score to measure its correctness. Classic methods include IKRL, TBKGC, MMKRL, and OTKGE. However, these methods often only consider very ideal scenarios, i.e., the multimodal information of entities in the multimodal knowledge graph is complete, or they use very simple processing methods for missing entity modal information, such as random initialization. Such designs are actually very inefficient in utilizing the modal information of entities in the multimodal knowledge graph and cannot completely solve the impact of missing entity modal information on the prediction results of knowledge graph completion tasks.

[0007] Therefore, if we can design a method for completing entity attributes in a multimodal knowledge graph, and train a set of modality feature generators and discriminators by using adversarial learning to leverage the existing structural information and multimodal information (such as image information) of entities in the multimodal knowledge graph, then we can use the modality feature generator to generate corresponding modality features for data lacking modality information. At the same time, we can also adjust the modality information of entities that do not lack modality information to enrich their semantic information. While completing the entity modality information of the knowledge graph, this will also help downstream tasks such as knowledge graph completion, and promote the implementation and application of knowledge graphs in real production and life. Summary of the Invention

[0008] In view of the above, the purpose of this invention is to provide a method and apparatus for multimodal knowledge graph completion based on graph neural networks and adversarial learning, which can improve the performance of multimodal knowledge graph completion tasks while completing the entity attributes of multimodal knowledge graphs, and promote the implementation and application of knowledge graph technology in real production and life scenarios.

[0009] To achieve the above-mentioned objectives, this embodiment provides a multimodal knowledge graph completion method based on graph neural networks and adversarial learning, comprising the following steps:

[0010] Step 1: Transform the triplet data in the multimodal knowledge graph into a heterogeneous graph, and simultaneously extract features from the entity image information in the multimodal knowledge graph to obtain the original entity modality features;

[0011] Step 2: Construct a learning model that includes a heterogeneous graph convolutional network, a modal feature generator, and a modal feature discriminator. The heterogeneous graph convolutional neural network extracts entity structure features from the entity structure information in the heterogeneous graph. The modal feature generator generates corresponding entity modal features for entities based on random noise and entity structure features. The modal feature discriminator determines whether the original entity modal features and the entity modal features in the input pair correspond to the same entity.

[0012] Step 3: After training the learning model using adversarial learning, use the trained modal feature generator and modal feature discriminator to generate entity modal features for each entity in the multimodal knowledge graph.

[0013] Step 4: Use the entity modal features obtained in Step 3 and the entity structural features obtained by encoding through the heterogeneous graph convolutional network as the initialization results of the multimodal knowledge graph completion model, and train the multimodal knowledge graph completion model based on the comparison of positive and negative samples.

[0014] Step 5: Use the trained entity representations and relation representations and the defined scoring function in the multimodal knowledge graph completion model to score the candidate triples, and perform knowledge graph completion based on the scoring results.

[0015] Preferably, in step 1, the conversion of triple data in the multimodal knowledge graph into a heterogeneous graph includes:

[0016] A multimodal knowledge graph is represented as G = (E, R, T), where E and R represent the entity set and relation set in the multimodal knowledge graph, respectively, and T = {(h, r, t) | h ∈ E, r ∈ R} represents the set of triples in the knowledge graph;

[0017] A heterogeneous graph is constructed based on the set of triples, where nodes and edges correspond to entities and triples in the knowledge graph, respectively.

[0018] Preferably, in step 1, the step of extracting features from the entity image information in the multimodal knowledge graph to obtain the original entity modality features includes:

[0019] Divide the entity set of the multimodal knowledge graph into disjoint subsets E. valid and subset E none Among them, subset E validThe entity in the middle corresponds to several images, that is, the corresponding entity image information, subset E. none The entity in the middle does not have a corresponding entity image information;

[0020] Using a pre-trained visual coding model for subset E valid The entity image information is used to perform feature encoding to extract the original entity modal features, which can be expressed by the formula:

[0021]

[0022] Among them, |I i | represents the i-th entity e i The corresponding image information set I i Number of entities in the image i Represents entity e i The corresponding image information, ViT(·) represents the pre-trained visual coding model, e v,i Represents entity e i The original entity modal characteristics.

[0023] Preferably, in step 2, the heterogeneous graph convolutional neural network extracts entity structure features from the entity structure information in the heterogeneous graph, expressed by the formula:

[0024]

[0025] in, and This represents the entity structure feature obtained after the i-th entity is encoded by the heterogeneous convolutional network at layer l and layer l+1, where σ represents the ReLU activation function. Represents entity e i The set of neighboring nodes, W r W0 is the projection matrix defined for each relation r in the relation set R, and W0 is the projection matrix that projects the entity's own features onto it. Indicates the number of neighboring nodes. This indicates that the j-th entity obtains its structural features after being encoded by the l-th layer heterogeneous convolutional network.

[0026] Preferably, in step 2, the modal feature generator employs at least two fully connected neural networks to generate corresponding entity modal features for the entity based on random noise and entity structural features, expressed by the formula:

[0027] G(z,e s,i )=FC2(σ(FC1([z;e s,i ])))

[0028] Wherein G(z,e) s,i The modal feature generator () represents the generator based on random noise z and solid structural features e. s,iGenerate entity modal features v i FC1(·) and FC2(·) represent two fully connected neural network layers, σ(·) represents the ReLU activation function, and [;] represents the vector concatenation operator.

[0029] Preferably, in step 2, the modality feature discriminator employs at least two fully connected neural networks to determine whether the original entity modality features and the entity modality features of the input pairs correspond to the same entity, as expressed by the formula:

[0030] D(v i |e s,i )=FC2([σ(FC1(v i )),e s,i ])

[0031] Wherein, D(v) i |e s,i ) indicates the determination of the modal features of the original entity e s,i and entity modal features v i The probability of whether they correspond to the same entity.

[0032] Preferably, in step 3, the trained modal feature generator and modal feature discriminator are used to generate entity modal features for each entity in the multimodal knowledge graph, including:

[0033] Use the trained modal feature generator for each entity e i Generate K modal features, and then use the trained modal feature discriminator to compare the K modal features with entity e. i The entity structure features are used to determine whether they correspond to the same entity, and based on the determination result, the modal features are obtained by mean pooling, which is expressed by the formula:

[0034]

[0035] Among them, e v,i Let p represent the entity modal features generated by the i-th entity. j The value of v represents the judgment result of the modal feature discriminator for the j-th modal feature, and can be either 0 or 1. 0 indicates that the modal feature and the entity structure feature do not correspond to the same entity, while 1 indicates that they correspond to the same entity. i,j This represents the j-th modal feature generated by the i-th entity.

[0036] Preferably, in step 4, the representation vector parameters of the multimodal knowledge graph completion model are initialized using the entity modal features obtained in step 3 and the entity structural features obtained by encoding through a heterogeneous graph convolutional network. For the triple (h,r,t), the multimodal knowledge graph completion model uses the following scoring function to obtain the score F(h,r,t) of the triple (h,r,t).

[0037]

[0038] Among them, h s ,h v ,t s ,t v W represents the entity structural features and entity modal features corresponding to the head entity h and tail entity t, respectively. v The modal feature projection matrix is ​​used to project the modal features of an entity onto the structural feature space of the entity. The symbol ° represents the rotation operation on the complex plane. The scoring function F(h,r,t) treats the vector as a complex number in the complex space and performs a rotation operation on the vector r.

[0039] Based on the score F(h,r,t), the contrast loss function L for positive and negative samples is calculated, which is expressed by the formula:

[0040]

[0041] Where T represents the set of triples in the heterogeneous graph, γ represents the artificially defined margin, σ(·) represents the Sigmoid function, N represents the number of negative samples in each triple, and the number of negative samples (h) ′ ,r ′ ,t ′ The positive sample (h,r,t) is obtained by randomly replacing the first and last entities. F(h,r,t) represents the score of the positive sample (h,r,t). ′ ,r ′ ,t ′ ) represents negative samples (h) ′ ,r ′ ,t ′ The score;

[0042] The multimodal knowledge graph completion model is trained using the contrastive loss function L.

[0043] Preferably, in step 5, for each candidate triplet, a score is calculated based on the entity representation, relation representation, and defined scoring function trained in the multimodal knowledge graph completion model, and high-scoring candidate triplets are selected for knowledge graph completion.

[0044] To achieve the above-mentioned objectives, the embodiments also provide a multimodal knowledge graph completion device based on graph neural networks and adversarial learning, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-mentioned multimodal knowledge graph completion method based on graph neural networks and adversarial learning.

[0045] Compared with the prior art, the beneficial effects of the present invention include at least the following:

[0046] To facilitate the implementation and application of multimodal knowledge graphs in real-world production and daily life scenarios, this paper proposes an effective solution to the problem of missing entity modal attributes. Through adversarial learning, a set of modal feature generators and modal feature discriminators are trained to complete and enhance the modal features of all entities in the knowledge graph. These representation vectors are then applied to tasks such as knowledge graph completion, enhancing the prediction and reasoning capabilities of the knowledge graph completion model. This, in turn, promotes the application and implementation of multimodal knowledge graphs in real-world scenarios such as e-commerce and social networks. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 This is a flowchart of the multimodal knowledge graph completion method based on graph neural networks and adversarial learning provided in the embodiment;

[0049] Figure 2 This describes the structure and training process of the learning model provided in the embodiment. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.

[0051] Existing multimodal knowledge graph completion methods in fields such as e-commerce and social networks do not consider the situation of missing entity modalities in real-world scenarios. This leads to performance degradation when modal information is incorporated into the knowledge graph completion model for triple scoring due to the lack of modal information. To address this technical problem, this invention provides a multimodal knowledge graph completion method and apparatus based on graph neural networks and adversarial learning.

[0052] like Figure 1 As shown in the embodiment, the multimodal knowledge graph completion method based on graph neural networks and adversarial learning provided includes the following steps:

[0053] Step 1: Transform the triplet data in the multimodal knowledge graph into a heterogeneous graph, and simultaneously extract features from the entity image information in the multimodal knowledge graph to obtain the original entity modal features.

[0054] A multimodal knowledge graph is represented as G = (E, R, T), where E and R represent the entity set and relation set, respectively, and T = {(h, r, t) | h ∈ E, r ∈ R} represents the set of triples in the knowledge graph. In the multimodal knowledge graph for e-commerce, the head entity h represents a product, the tail entity t represents a product attribute value or product category, and the relation r represents an attribute or category. For social networks, the head entity h represents user A in the social network, the tail entity t represents user B's friend, and the relation r represents the following relationship between users, thus forming a triple of (user A, following, friend B). Besides this, there are many other triple types, such as (user A, interested topics, topic C), (topic C, associated with…, topic D), etc. The multimodal information in a social network knowledge graph includes user avatars, photos, topic-related images, etc. A heterogeneous graph is constructed based on the set of triples T, where nodes and edges correspond to entities and triples in the knowledge graph, respectively.

[0055] Based on whether the modal information of an entity is missing, the entity set of the multimodal knowledge graph is divided into disjoint subsets E. valid and subset E none Among them, subset E valid The entity in the middle corresponds to several images, that is, the corresponding entity image information, subset E. none The entity in the middle does not have corresponding entity image information. For each entity e, two feature vectors ei are defined. s ,e v The dimensions of the entity structural features and the original entity modal features are respectively represented by d. i ,d v .

[0056] Using a pre-trained visual coding model for subset E valid The entity image information is used to perform feature encoding to extract the original entity modal features, which can be expressed by the formula:

[0057]

[0058] Among them, |I i | represents the i-th entity e i The corresponding image information set Ii Number of entities in the image i Represents entity e i The corresponding image information, ViT(·) represents the pre-trained visual encoding model, which can be a pre-trained visual Transformer model, e v,i Represents entity e i The original entity modal characteristics.

[0059] Step 2: Construct a learning model that includes a heterogeneous graph convolutional network, a modal feature generator, and a modal feature discriminator.

[0060] In the learning model, the Heterogeneous Graph Convolutional Neural Network (R-GCN) extracts entity structure features from the entity structure information in the heterogeneous graph. The modality feature generator (G) generates corresponding entity modality features for the entity based on random noise and entity structure features. The modality feature discriminator (D) determines whether the original entity modality features and the entity modality features of the input pair correspond to the same entity.

[0061] Specifically, the heterogeneous graph convolutional neural network uses a single-layer heterogeneous graph convolutional network as an encoder for structural information, which is used to extract entity structural features from the entity structural information in the heterogeneous graph, expressed by the formula:

[0062]

[0063] in, and This represents the entity structure feature obtained after the i-th entity is encoded by the heterogeneous convolutional network at layer l and layer l+1, where σ represents the ReLU activation function. Represents entity e i The set of neighboring nodes, W r W0 is the projection matrix defined for each relation r in the relation set R, and W0 is the projection matrix that projects the entity's own features onto it. Indicates the number of neighboring nodes. This indicates that the j-th entity obtains its structural features after being encoded by the l-th layer heterogeneous convolutional network.

[0064] Specifically, the modal feature generator employs at least two fully connected neural networks to generate corresponding entity modal features for the entity based on random noise and entity structural features, expressed by the formula:

[0065] G(z,e s,i )=FC2(σ(FC1([z;e s,i ])))

[0066] Wherein G(z,e) s,i The modal feature generator () represents the generator based on random noise z and solid structural features e. s,i Generate entity modal features vi FC1(·) and FC2(·) represent two fully connected neural network layers, σ(·) represents the ReLU activation function, z is 64-dimensional random noise sampled from a standard normal distribution, and ,;] represents the vector concatenation operator, indicating that two vectors are concatenated to generate a vector of dimension d. v entity modal features v i .

[0067] Specifically, the modality feature discriminator employs at least two fully connected neural networks to determine whether the original entity modality features and the entity modality features of the input pairs correspond to the same entity, as expressed by the formula:

[0068] D(v i |e s,i )=FC2([σ(FC1(v i )),e s,i ])

[0069] Wherein, D(v) i |e s,i ) indicates the determination of the modal features of the original entity e s,i and entity modal features v i The probability of whether they correspond to the same entity. It should be noted that entity modal features v i It can be generated by the modality feature generator, or it can be obtained by feature extraction based on entity image information. The two fully connected neural network layers FC1(·) and FC2(·) in the modality feature discriminator are different from the two fully connected neural network layers in the modality feature generator.

[0070] Step 3: After training the learning model using adversarial learning, use the trained modal feature generator and modal feature discriminator to generate entity modal features for each entity in the multimodal knowledge graph.

[0071] When training the learning model using adversarial learning, it is divided into two stages. In the first stage, the parameters of the modal feature generator are fixed to train the modal feature discriminator, so that the modal feature discriminator has a stronger ability to distinguish multimodal entity attributes. In the second stage, the parameters of the modal feature discriminator are fixed to train the generator, so that the modal feature generator can generate more realistic entity modal features.

[0072] In each training stage, the structural feature vectors of entities are obtained through message passing and aggregation by the heterogeneous graph convolutional network R-GCN. This ensures that the structural feature vectors of entities are updated during training. This two-stage training method allows the modal feature generator G and the modal feature discriminator D to engage in a game-like interaction, gradually improving their performance during training. The overall training objective can be expressed as:

[0073]

[0074] After training the learning model, the trained modal feature generator and modal feature discriminator are used to generate entity modal features for each entity in the multimodal knowledge graph. The specific process is as follows:

[0075] Use the trained modal feature generator for each entity e i Generate K = 512 distinct dimensions d v The modal features are then used to identify the K modal features and the entity e. i The entity structure features are used to determine whether they correspond to the same entity. Based on the determination result, the modal features are processed by mean pooling to obtain the entity modal features of each entity, which can be expressed by the formula:

[0076]

[0077] Among them, e v,i Let p represent the entity modal features generated by the i-th entity. j The value of v represents the judgment result of the modal feature discriminator for the j-th modal feature, and can be either 0 or 1. 0 indicates that the modal feature and the entity structure feature do not correspond to the same entity, while 1 indicates that they correspond to the same entity. i,j This represents the j-th modal feature generated by the i-th entity.

[0078] Step 4: Use the entity modal features obtained in Step 3 and the entity structural features obtained by encoding through a heterogeneous graph convolutional network as the initialization results of the multimodal knowledge graph completion model, and train the multimodal knowledge graph completion model based on the comparison of positive and negative samples.

[0079] The modal knowledge graph completion model still uses entity structure feature vector e s and entity modal eigenvector e v The entities in the knowledge graph are represented by a structural feature vector r, and each relation in the knowledge graph is represented by a structural feature vector r. Simultaneously, the representation vector parameters of the multimodal knowledge graph completion model are initialized using the entity modal features obtained in step 3 and the entity structural features encoded by a heterogeneous graph convolutional network. For the triple (h,r,t), the multimodal knowledge graph completion model uses the following scoring function to obtain the score F(h,r,t) of the triple (h,r,t);

[0080]

[0081] Among them, h s ,h v ,t s ,t vW represents the entity structural features and entity modal features corresponding to the head entity h and tail entity t, respectively. v This represents the modal feature projection matrix, used to project features of dimension d. v The entity modal features are projected onto a dimension of d. s In the entity structure feature space, the symbol ° represents a rotation operation on the complex plane. The scoring function F(h,r,t) treats the vector as a complex number in the complex space and performs a rotation operation on the vector r. This scoring function is composed of multiple scoring functions to ensure sufficient interaction between representation vectors of different types, enabling implicit alignment between two representation vectors of the same entity.

[0082] The goal of the scoring function is to ensure that, in correct triples (i.e., positive samples), the representation vector of the head entity, after transformation by relation r, is as close as possible to the representation vector of the corresponding tail entity, thus achieving a higher score. Conversely, it aims to minimize the score of incorrect triples (i.e., negative samples). Therefore, during the training process, the multimodal knowledge graph completion model calculates the contrastive loss function L for positive and negative samples based on the scoring F(h,r,t), expressed by the formula:

[0083]

[0084] Where T represents the set of triples in the heterogeneous graph, γ represents the artificially defined margin, σ(·) represents the Sigmoid function, N represents the number of negative samples in each triple, and the number of negative samples (h) ′ ,r ′ ,t ′ The positive sample (h,r,t) is obtained by randomly replacing the first and last entities. F(h,r,t) represents the score of the positive sample (h,r,t). ′ ,r ′ ,t ′ ) represents negative samples (h) ′ ,r ′ ,t ′ The scoring of the model is used to train the multimodal knowledge graph completion model using the contrastive loss function L.

[0085] Step 5: Use the trained entity representations and relation representations and the defined scoring function in the multimodal knowledge graph completion model to score the candidate triples, and perform knowledge graph completion based on the scoring results.

[0086] This paper utilizes a trained multimodal knowledge graph completion model to complete multimodal knowledge graphs (such as e-commerce knowledge graphs and social network knowledge graphs). Specifically, multimodal knowledge graph completion can be divided into two tasks: head entity prediction and tail entity prediction. For head entities, given a head entity h and a relation r, the model needs to predict possible tail entities t. The model will treat all entities as candidates for tail entities and form candidate triples (h, r, t) with the head entity h and the relation r. i ),t i For each candidate triplet, a score is calculated using the entity representation vector and scoring function trained in the completion model. The tail entities are then sorted according to their scores. The tail entity corresponding to the triplet with the highest score is selected as the prediction result of the model, and the knowledge graph is completed.

[0087] Based on the same inventive concept, the embodiment also provides a multimodal knowledge graph completion device based on graph neural networks and adversarial learning, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-mentioned multimodal knowledge graph completion method based on graph neural networks and adversarial learning, including the following steps:

[0088] Step 1: Transform the triplet data in the multimodal knowledge graph into a heterogeneous graph, and simultaneously extract features from the entity image information in the multimodal knowledge graph to obtain the original entity modality features;

[0089] Step 2: Construct a learning model that includes a heterogeneous graph convolutional network, a modal feature generator, and a modal feature discriminator;

[0090] Step 3: After training the learning model using adversarial learning, use the trained modal feature generator and modal feature discriminator to generate entity modal features for each entity in the multimodal knowledge graph.

[0091] Step 4: Use the entity modal features obtained in Step 3 and the entity structural features obtained by encoding through the heterogeneous graph convolutional network as the initialization results of the multimodal knowledge graph completion model, and train the multimodal knowledge graph completion model based on the comparison of positive and negative samples.

[0092] Step 5: Use the trained entity representations and relation representations and the defined scoring function in the multimodal knowledge graph completion model to score the candidate triples, and perform knowledge graph completion based on the scoring results.

[0093] In practical applications, computer memory can be volatile memory located at the local end, such as RAM; it can also be non-volatile memory, such as ROM, FLASH, floppy disks, hard disks, etc.; or it can be a remote storage cloud. The computer processor can be a central processing unit (CPU), microprocessor (MPU), digital signal processor (DSP), or field-programmable gate array (FPGA). These processors can be used to implement the steps of a multimodal knowledge graph completion method based on graph neural networks and adversarial learning.

[0094] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multimodal knowledge graph completion method based on graph neural networks and adversarial learning, characterized in that, Includes the following steps: Step 1: Transform the triplet data in the multimodal knowledge graph into a heterogeneous graph, and simultaneously extract features from the entity image information in the multimodal knowledge graph to obtain the original entity modality features; Step 2: Construct a learning model that includes a heterogeneous graph convolutional network, a modal feature generator, and a modal feature discriminator. The heterogeneous graph convolutional neural network extracts entity structure features from the entity structure information in the heterogeneous graph. The modal feature generator generates corresponding entity modal features for entities based on random noise and entity structure features. The modal feature discriminator determines whether the original entity modal features and the entity modal features in the input pair correspond to the same entity. Step 3: After training the learning model using adversarial learning, use the trained modal feature generator and modal feature discriminator to generate entity modal features for each entity in the multimodal knowledge graph. Step 4: Use the entity modal features obtained in Step 3 and the entity structural features obtained by encoding through the heterogeneous graph convolutional network as the initialization results of the multimodal knowledge graph completion model, and train the multimodal knowledge graph completion model based on the comparison of positive and negative samples. Step 5: Use the trained entity representations and relation representations and the defined scoring function in the multimodal knowledge graph completion model to score the candidate triples, and perform knowledge graph completion based on the scoring results.

2. The multimodal knowledge graph completion method based on graph neural networks and adversarial learning according to claim 1, characterized in that, Step 1, which involves transforming the triplet data in the multimodal knowledge graph into a heterogeneous graph, includes: A multimodal knowledge graph is represented as G = (E, R, T), where E and R represent the entity set and relation set in the multimodal knowledge graph, respectively, and T = {(h, r, t) | h ∈ E, r ∈ R} represents the set of triples in the knowledge graph; A heterogeneous graph is constructed based on the set of triples, where nodes and edges correspond to entities and triples in the knowledge graph, respectively.

3. The multimodal knowledge graph completion method based on graph neural networks and adversarial learning according to claim 1, characterized in that, In step 1, the step of extracting features from the entity image information in the multimodal knowledge graph to obtain the original entity modality features includes: Divide the entity set of the multimodal knowledge graph into disjoint subsets E. valid and subset E none Among them, subset E valid The entity in the middle corresponds to several images, that is, the corresponding entity image information, subset E. none The entity in the middle does not have a corresponding entity image information; Using a pre-trained visual coding model for subset E valid The entity image information is used to perform feature encoding to extract the original entity modal features, which can be expressed by the formula: Among them, |I i | represents the i-th entity e i The corresponding image information set I i Number of entities in the image i Represents entity e i The corresponding image information, ViT(·) represents the pre-trained visual coding model, e v，i Represents entity e i The original entity modal characteristics.

4. The multimodal knowledge graph completion method based on graph neural networks and adversarial learning according to claim 1, characterized in that, In step 2, the heterogeneous graph convolutional neural network extracts the entity structure features of the entity structure information in the heterogeneous graph. The update process of the structure features of each entity is expressed by the formula: in, and This represents the entity structure feature obtained after the i-th entity is encoded by the heterogeneous convolutional network at layer l and layer l+1, where σ represents the ReLU activation function. Represents entity e i The set of neighboring nodes, W r W0 is the projection matrix defined for each relation r in the relation set R, and W0 is the projection matrix that projects the entity's own features onto it. Indicates the number of neighboring nodes. This indicates that the j-th entity obtains its structural features after being encoded by the l-th layer heterogeneous convolutional network.

5. The multimodal knowledge graph completion method based on graph neural networks and adversarial learning according to claim 1, characterized in that, In step 2, the modal feature generator employs at least two fully connected neural networks to generate corresponding entity modal features for the entity based on random noise and entity structural features, expressed by the formula: G(z,e s，i )=FC2(σ(FC1([z;e s，i ]))) Wherein G(z,e) s，i The modal feature generator () represents the generator based on random noise z and solid structural features e. s，i Generate entity modal features v i FC1(·) and FC2(·) represent two fully connected neural network layers, σ(·) represents the ReLU activation function, and [;] represents the vector concatenation operator.

6. The multimodal knowledge graph completion method based on graph neural networks and adversarial learning according to claim 1, characterized in that, In step 2, the modality feature discriminator employs at least two fully connected neural networks to determine whether the original entity modality features and the entity modality features of the input pairs correspond to the same entity, as expressed by the formula: D(v i |e s，i )=FC2([σ(FC1(v i )), and s，i ]) Wherein, D(v) i |e s，i ) indicates the determination of the modal features of the original entity e s，i and entity modal features v i The probability of whether they correspond to the same entity.

7. The multimodal knowledge graph completion method based on graph neural networks and adversarial learning according to claim 1, characterized in that, In step 3, the trained modal feature generator and modal feature discriminator are used to generate entity modal features for each entity in the multimodal knowledge graph, including: Use the trained modal feature generator for each entity e i Generate K modal features, and then use the trained modal feature discriminator to compare the K modal features with entity e. i The entity structure features are used to determine whether they correspond to the same entity. Based on the determination result, the modal features are processed by mean pooling to obtain the entity modal features of each entity, which can be expressed by the formula: Among them, e v，i Let p represent the entity modal features generated by the i-th entity. j The value of v represents the judgment result of the modal feature discriminator for the j-th modal feature, and can be either 0 or 1. 0 indicates that the modal feature and the entity structure feature do not correspond to the same entity, while 1 indicates that they correspond to the same entity. i，j This represents the j-th modal feature generated by the i-th entity.

8. The multimodal knowledge graph completion method based on graph neural networks and adversarial learning according to claim 1, characterized in that, In step 4, the representation vector parameters of the multimodal knowledge graph completion model are initialized using the entity modal features obtained in step 3 and the entity structural features obtained by encoding through a heterogeneous graph convolutional network. For the triple (h, r, t), the multimodal knowledge graph completion model uses the following scoring function to obtain the score F(h, r, t) of the triple (h, r, t). Among them, h s h v , t s , t v W represents the entity structural features and entity modal features corresponding to the head entity h and tail entity t, respectively. v The modal feature projection matrix is ​​used to project the modal features of an entity onto the entity's structural feature space. The symbol is... The scoring function F(h, r, t) represents a rotation operation on the complex plane. It treats the vector as a complex number in the complex space and performs a rotation operation on the vector r. Based on the score F(h, r, t), the contrast loss function L for positive and negative samples is calculated, and expressed by the formula: Where T represents the set of triples in the heterogeneous graph, γ represents the artificially defined boundary, σ(·) represents the Sigmoid function, N represents the number of negative samples in each triple, the negative samples (h′, r′, t′) are obtained by randomly replacing the head and tail entities of the positive samples (h, r, t), F(h, r, t) represents the score of the positive samples (h, r, t), and F(h′, r′, t′) represents the score of the negative samples (h′, r′, t′). The multimodal knowledge graph completion model is trained using the contrastive loss function L.

9. The multimodal knowledge graph completion method based on graph neural networks and adversarial learning according to claim 1, characterized in that, In step 5, for each candidate triple, a score is calculated based on the entity representation, relation representation and the defined scoring function trained in the multimodal knowledge graph completion model, and high-scoring candidate triples are selected for knowledge graph completion.

10. A multimodal knowledge graph completion device based on graph neural networks and adversarial learning, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the multimodal knowledge graph completion method based on graph neural networks and adversarial learning as described in any one of claims 1-9.

Images