Multi-modal knowledge graph completion method based on generative adversarial network

Abstract

The invention relates to the technical field of knowledge maps, and discloses a multi-modal knowledge map completion method based on a generative adversarial network, which is used for solving the problems in the prior art that modal information is unbalanced, a modal fusion strategy is rough, the inference capability is reduced under modal deficiency and the like. Particularly, when only a tail entity prediction task is carried out, good generalization ability and robustness are achieved, a pre-training comparison learning model is used for coding image and text information of an entity, and multi-modal semantic features are extracted and mapped to a unified embedding space to achieve modal alignment; secondly, through a fine-grained modal attention fusion module, dynamically adjusting weights according to contributions of different modals in a tail entity prediction task, and realizing effective integration of information; further introducing a generative adversarial network based on a convolutional structure, and enhancing the robustness of the model to modal deficiency by constructing pseudo-modal features; and then a RotatE scoring function is adopted to model triple semantic rationality, and training optimization is carried out in combination with a self-adversarial negative sampling strategy.

Description

Technical Field

[0001] This invention relates to the field of knowledge graph technology, specifically to a multimodal knowledge graph completion method based on generative adversarial networks. Background Technology

[0002] A knowledge graph is a structured knowledge base that represents facts and semantic relationships using triples (head entity, relation, tail entity) as basic units. It is widely used in search engines, question-answering systems, and recommendation systems, providing efficient support for entity linking and relation reasoning. However, real-world knowledge graphs often contain a large number of missing entity-relationship or fact triples. Knowledge graph completion technology predicts missing or unknown entities or relations based on existing triples in the knowledge graph, filling in the gaps and thus making the knowledge graph more complete.

[0003] With the development of multimodal data technology, traditional knowledge graphs are gradually evolving into knowledge systems that integrate multimodal information such as text and images. However, most current knowledge graph construction methods still rely mainly on structured and textual information, ignoring the rich semantics contained in modalities such as images. Especially in the context of the increasing prevalence of multimodal data, traditional completion methods face significant limitations.

[0004] To address these issues, researchers have proposed the research direction of Multimodal Knowledge Graph Completion (MMKGC), which aims to improve the richness of knowledge representation and the accuracy of reasoning by fusing image, text, and structural information. Although existing studies use graph neural networks, Transformers, multimodal encoders, and other structures for modeling, they generally suffer from two major shortcomings: (1) they ignore the significant imbalance of modal information among different entities; and (2) they fail to adaptively adjust the weights according to the importance of each modal information. Summary of the Invention

[0005] The purpose of this invention is to provide a multimodal knowledge graph completion method based on generative adversarial networks, which solves the problems of unbalanced modal information, coarse modal fusion strategies, and decreased reasoning ability under modal missing conditions in current multimodal knowledge graph completion tasks.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a multimodal knowledge graph completion method based on generative adversarial networks, comprising the following steps: A. Training a knowledge graph completion model:

[0007] A1. By using a contrastive learning pre-trained model to encode images and text, deep semantic features of entities in image and text modalities are extracted and mapped to a unified embedding space to achieve preliminary modal alignment.

[0008] A2. Construct a modal fusion module with a fine-grained attention mechanism, and adaptively adjust the feature fusion weights according to the importance of each modal feature in different task scenarios to achieve effective information integration;

[0009] A3. Introduce a generative adversarial network based on a convolutional structure to simulate missing modalities during training. Construct high-quality pseudo-modal features through a generator to enhance the model's robustness to incomplete modalities.

[0010] A4. Use the RotatE scoring function to measure the rationality of triples, and combine it with a self-adversarial negative sampling strategy to optimize the training process and improve the model's reasoning ability in complex multimodal environments.

[0011] B. Performing completion tasks based on knowledge graph completion models:

[0012] The trained knowledge graph completion model is used for actual knowledge graph completion tasks. First, structural information, image information, and text information are encoded and fused to generate multimodal embedding representations of entities. Then, candidate triples are scored using a scoring function, and the completion result with the highest score is selected. Finally, the completion result with the highest score is chosen.

[0013] Preferably, in step A1, encoding the image and text using a contrastive learning pre-trained model specifically includes:

[0014] A11, Visual input I for each entity e e The feature vector F is generated by encoding the visual encoder part of the pre-trained encoder CLIP (Contrastive Language–Image Pre-training) model. V The processed vectors are projected into the embedding space to obtain the embedding features of the visual modality.

[0015] F V =proj(CLIP) ViT (I e ));

[0016] A12. Text input I for each entity e e The text encoder part of the pre-trained encoder CLIP model is used to encode the text description of the entity, generating a text modal feature vector F. T :

[0017] F T =proj(CLIP) Text (T e ));

[0018] A13. Structural input for each entity ee It is encoded using a trainable embedding layer to generate structural modality features F. S Specifically, each entity in the knowledge graph is first assigned a unique index number, and then discrete entity identifiers are mapped to continuous dense vector representations through an embedding matrix:

[0019] F S =Embedding(S e ).

[0020] Preferably, in step A2, the modal fusion module for constructing a fine-grained attention mechanism adaptively adjusts the feature fusion weights based on the importance of each modal feature in different scenarios, specifically including:

[0021] A21. Perform a unified transformation of the identifier space on visual, textual, and structural modal features, and input modal features F. V F T F S The latent representation a is generated by mapping through independent linear transformations. V a T a S :

[0022] a i =W i F i +b i ,i∈{V,T,S};

[0023] A22. The modal features {a} after linear transformation V a T a S The attention tensor A is formed by stacking elements along the modal dimension. The Softmax function is then applied to tensor A along the modal dimension to generate the modal attention weight matrix W. attn :

[0024] A = stack(a V a T a S )

[0025] W attn =Softmax(A);

[0026] A23. The original input modal features F V F T F S The weighted modal features are obtained by performing element-wise multiplication with the corresponding attention weights.

[0027] Preferably, in step A3, the introduction of a generative adversarial network based on a convolutional structure specifically includes:

[0028] A31. Construct a generator module, with the input being the structural embedding F of the entity. S The noise vector e sampled from the standard normal distribution noise The vector concatenated from ~N(0,1) is first transformed using a linear transformation to generate an initial feature map:

[0029] fc out =reshape(ReLU(W1·concat(e noise ,F S )+b1));

[0030] A32. Input the initial feature map into the ConBlocks module, which contains two convolutional and normalization layers, to extract high-dimensional local feature representations. One layer of the module is represented as follows:

[0031] Conv_out=ReLU(BatchNorm(Conv2d(fC out )));

[0032] A33. Flatten the convolutional output into a one-dimensional vector and map it to an output embedding of the same dimension as the structural features through a linear transformation:

[0033] C onvoutflat =reshape(conv) out, [batchsize,*])

[0034] G n =W4·Conv_out_flat+b4.

[0035] Preferably, in step A4, the step of using the RotatE scoring function to measure the rationality of triples and combining it with a self-adversarial negative sampling strategy to optimize the training process specifically includes:

[0036] A41. Use the RotatE model to embed entities and relationships into the model. Specifically, embed the head and tail entities into F... h F t The relation embedding r = Embedding(r) is modeled as a complex rotation operation modulo 1, such that the predicted representation of the tail entity in the triple is... in This represents element-wise complex multiplication;

[0037] The semantic reasonableness score of the triple (h,r,t) is calculated using the following RotatE scoring function; a higher score indicates a more reasonable prediction.

[0038]

[0039] Where ||·||2 is the l2 norm, i.e.

[0040] A42. Based on the RotatE score, construct a training loss function. For each positive triple (h,r,t)∈T, jointly consider k negative samples, and use the sigmoid function to define and calculate the following loss function:

[0041]

[0042] Where γ is the score margin constant, σ is the sigmoid function, (h′ i ,r′ i ,t′ i ) represents the negative triples generated by negative sampling, where p is the negative sample. i The self-adversarial weights are calculated as follows:

[0043]

[0044] Where β is the temperature parameter, the main training objective of the model is to minimize the loss. kgc ;

[0045] A43. Calculate the loss in adversarial training as follows:

[0046]

[0047] Preferably, the loss is minimized using stochastic gradient descent, with alternating optimization of the loss. kgc and Loss adv The objective function is such that the discriminator can better distinguish between real and fake multimodal features, while the generator continuously improves the quality of fake features, thereby achieving effective learning of multimodal information. The overall training objective is:

[0048] min Loss kgc +min maxλLoss adv .

[0049] This invention provides a multimodal knowledge graph completion method based on generative adversarial networks (GANs). This multimodal knowledge graph completion method based on GANs has the following beneficial effects:

[0050] (1) The multimodal knowledge graph completion method based on generative adversarial networks introduces an image and text pre-training model based on contrastive learning, which makes the multimodal features have stronger semantic consistency in the shared space.

[0051] (2) The multimodal knowledge graph completion method based on generative adversarial networks designs a fine-grained modal attention fusion mechanism to realize the weighted combination of different modal features in the feature dimension, thereby enhancing the accuracy and robustness of semantic representation;

[0052] (3) The multimodal knowledge graph completion method based on generative adversarial networks completes missing modal information by using a convolutional generative adversarial network, thereby improving the model’s adaptability to modal imbalance and information loss and significantly enhancing the generalization ability of multimodal knowledge graph completion. Attached Figure Description

[0053] Figure 1 This is a flowchart of the model training process in an embodiment of the present invention;

[0054] Figure 2 This is a schematic diagram of the algorithm principle for model training in an embodiment of the present invention;

[0055] Figure 3 This is a flowchart of the model prediction process in an embodiment of the present invention. Detailed Implementation

[0056] This invention aims to propose a multimodal knowledge graph completion method based on generative adversarial networks (GANs), addressing issues such as insufficient utilization of multimodal features, imbalanced modal information, and weak reasoning ability for missing modalities in existing technologies. The method includes a training process and a completion process. During training, sample data is first acquired and preprocessed to obtain the entity relationship structure information, text descriptions, and image information of the multimodal knowledge graph. Next, a pre-trained model based on contrastive learning is used to extract features from the text and image information of the entities, and a trainable embedding layer is used to model the structural information and relationships. Then, a fine-grained attention modality fusion module is designed to fuse the fused image, text, and structural modal information of the entities. A GAN is then used to simulate missing modalities, and the RotatE scoring function is used to effectively model the triples. Finally, the knowledge graph completion loss and adversarial loss are calculated separately, combined, and the model is trained alternately using stochastic gradient descent. When performing a specific completion task, the trained model is used to obtain the embedding features of candidate triples, the triple score is calculated using the trained scoring function, and the completion task is completed based on the score ranking.

[0057] like Figure 1-3 As shown, the present invention has the following two specific embodiments.

[0058] Example 1

[0059] S1. Sample Data Acquisition and Preprocessing:

[0060] S11. Prepare multimodal knowledge graph data. Each triple consists of a head entity, a relation, and a tail entity, in the form (h, r, t). For each entity, collect its text description (such as Wikipedia summaries, Baidu Encyclopedia entries) and image information (such as entity images, object photos, etc.).

[0061] S12. Assign unique structured identifiers (such as IDs) to entities and relations, select text and images that can characterize the features of entities and maintain a one-to-one correspondence between text descriptions, image information and entities, and use the supplemented multimodal knowledge graph as the dataset.

[0062] S2. Use a pre-trained model based on contrastive learning to extract features from the text and image information of entities, and use a trainable embedding layer to model structural information and relationships.

[0063] S21. For each entity e, first obtain its corresponding image information I. e Examples include portraits of people in Wikipedia entries, species images in animal encyclopedias, or product images on e-commerce platforms. The images are input into the visual encoder part of a pre-trained image-text contrast model CLIP (e.g., the ViT / B-32 model) to extract deep semantic features of the image modality. To unify the feature embedding space, the extracted raw image features need to be further processed through a learnable linear projection layer for dimensionality mapping, resulting in the final image modality embedding vector, as shown below:

[0064] F V =proj(CLIP) ViT / B-32 (I e ))

[0065] Wherein, CLIP ViT / B-32 A visual encoder using the Vision Transformer architecture, where proj represents a linear transformation used for dimension alignment.

[0066] S22. For each entity e, obtain its corresponding natural language description TeT_eTe, such as an encyclopedia entry summary, definition introduction, or short encyclopedia text. Input this description into the text encoder part of the same CLIP model. This encoder is usually based on the Transformer architecture and can capture semantic information in the text. Similar to image modality processing, to ensure consistency in the embedding space of each modality, the text feature output also needs to be mapped through a linear projection layer of the same dimension to obtain the text modality embedding vector, specifically expressed as:

[0067] F T =proj(CLIP) Text (T e ));

[0068] CLIP Text This represents the text encoder component in the CLIP model, where proj is the linear transformation function. The final result is F... T It can be used as the input of semantic and linguistic representations of entities into the downstream fusion model.

[0069] S23. For the structural information of each entity e (i.e., the entity's ID or index in the knowledge graph), encode it using a trainable embedding matrix to obtain a dense vector representation of the structural modality. Specifically, assign a unique number to each entity in the graph, input this number into the embedding layer, and obtain the entity structural embedding F by looking up a table. S This process can be formalized as follows:

[0070] F S =Embedding(S e );

[0071] Among them, S e This represents the index number of the entity, and the embedding is the parameter matrix of the embedding layer. Through training, this embedding layer can automatically learn to capture the structural relationship features between entities, providing basic structural semantic support for triple modeling;

[0072] S24. For each relation r in a triplet, an embedding layer is also used for structural representation learning. Each relation is treated as an independent discrete item in the knowledge graph and assigned a unique number. This number is input into the relation embedding layer to obtain its structural representation vector r, as follows:

[0073] r = Embedding(r);

[0074] The embedding and entity structure embedding use different parameter matrices to specifically represent the semantic information of the relationship. This embedding vector will be used in conjunction with the entity embedding in the triple scoring function to infer whether the relationship between entities is reasonable.

[0075] S3. The fine-grained attention modality fusion module is designed to fuse the entity's fused image, text, and structural modal information;

[0076] S31. In order to unify the representation dimensions of different modal features, the extracted image modal features F are first... V Text modal features F T and structural modal features F S The inputs are fed into three independent linear transformation layers, resulting in three sets of latent representation vectors:

[0077] a i =W i F i +b i,i∈{V,T,S};

[0078] Among them, W i and b i Let a represent the weight matrix and bias term a for each mode, respectively. V a T a S This involves the feature representations of the three modalities within the same latent space. The purpose of this step is to reconstruct the original modal features so that they can be fused and compared within a unified space.

[0079] S32. The modal latent representation {a} obtained above is... V a T a S Stacking them along the modal dimension forms a three-dimensional tensor A:

[0080] A = stack(a V a T a S )

[0081] Next, the Softmax function is applied to tensor A in the modality dimension to calculate the modality attention weight matrix W. attn :

[0082] W attn =Softmax(A)

[0083] The Softmax function ensures that the sum of all modality weights is 1, giving the weights a probabilistic meaning. The attention mechanism dynamically allocates weights based on the contribution of each modality to the current context, thereby improving the accuracy and effectiveness of feature fusion.

[0084] S33. Based on the attention weights of each modality, adjust the original input modal features F. V F T F S Perform element-wise multiplication to obtain the weighted modal features.

[0085]

[0086] Here, ⊙ represents element-wise multiplication, and represents the weight vector of the corresponding mode.

[0087] Finally, the weighted modal features are summed to form the entity's fused representation vector F. fusion :

[0088]

[0089] This fusion represents F fusionIntegrating information from images, text, and structural modalities, it possesses stronger representational capabilities and robustness, serving as the core input for subsequent generative adversarial modeling and scoring inference.

[0090] S4: Construct a generative adversarial network based on convolutional structures to enhance the model's robustness to missing modalities:

[0091] To improve the model's reasoning ability in situations involving missing or incomplete modal information, this embodiment constructs a Generative Adversarial Network (GAN) based on a convolutional structure. This network consists of a generator and a discriminator, and its main objective is to simulate potential modal missing scenarios in real-world applications during the training phase, generating high-quality pseudo-modal features to fill information gaps and improve overall robustness. The main steps are as follows:

[0092] S41. Generator Design and Pseudo-Feature Generation:

[0093] S411. Input Construction: The generator's input consists of two parts: first, the structural embedding representation of the entity; and second, a noise vector e that follows a standard normal distribution N(0,1). noise The two are combined into a unified input vector through a concat operation to simulate the missing modalities with diversity and uncertainty.

[0094] S412, Initial Feature Map Construction: The concatenated vector is first subjected to dimensionality enhancement and linear transformation through a fully connected linear layer, followed by the application of the non-linear activation function ReLU, with the output being the initial feature map. The formal expression is as follows:

[0095] fc out =reshape(ReLU(W1·concat(e noise ,F S )+b1))

[0096] Where W1 and b1 are the weight matrix and bias term in the generator, respectively, and the reshape operation reconstructs the linear output into a tensor shape suitable for convolution operations.

[0097] S413, Feature Extraction Module (ConBlocks): The initial feature map is input into a convolutional module containing two layers of convolutional operations (Conv2d), batch normalization (BatchNorm), and the ReLU activation function. This module is used to extract local high-dimensional modal features and enhance the semantic expressive power of pseudo-features.

[0098] The operation process for each layer is as follows:

[0099] Conv_out=ReLU(BatchNorm(Conv2d(fC out )))

[0100] After stacking the two convolutional layers mentioned above, the output tensor is the convolutional representation of the pseudomodal features.

[0101] S414. Output Mapping: Finally, the convolution output tensor is flattened into a one-dimensional vector and mapped through a linear transformation to the same embedding space dimension as the target modality (such as an image or text modality), serving as the final generated pseudo-feature representation G. n The calculation process is as follows:

[0102] C onvoutflat =reshape(conv) out, [batchsize,*])

[0103] G n =W4·Conv_out_flat+b4

[0104] Among them, W4 and b4 are the output layer parameters.

[0105] S42. Use the RotatE scoring mechanism as a "discriminator" to achieve adversarial training:

[0106] In this embodiment, instead of designing a separate discriminator network as in the traditional GAN ​​framework, the RotatE scoring function is used to evaluate the quality of pseudo-modal features and construct an adversarial training objective. RotatE is a knowledge graph embedding method based on complex space. Its basic idea is to model relation types by rotating the head entity embedding in complex space to move closer to the tail entity.

[0107] For a triple (h,r,t), the RotatE scoring function is:

[0108]

[0109] in This represents the Hadamard (element-wise multiplication) operation. The pseudo-modality is embedded in G. n Replacing the original modality with triplet scoring enables effective supervision of pseudo-features.

[0110] S43, Adversarial Training Loss Design:

[0111] Using RotatE's scoring function as adversarial supervision, the generator is trained to make its output pseudo-modal features' triple scores in RotatE as close as possible to the true triple scores, thus ensuring semantic consistency and usability. The loss function used is as follows:

[0112]

[0113] S44. RotatE is used as both a decoder and a completion decoder to evaluate the semantic reasonableness of entity relation triples. The loss is as follows:

[0114]

[0115] S45. Joint Optimization Strategy:

[0116] The ultimate training objective is to use RotatE's knowledge graph to complete the main task loss. kgc Fighting Loss Against Generators adv The joint optimization is as follows:

[0117] min Loss kgc +min maxλLoss adv

[0118] Where λ is a hyperparameter that balances the importance of the two tasks.

[0119] Through the above design, the model does not require an additional discriminator structure. It directly utilizes the knowledge graph scoring mechanism to complete the authenticity evaluation and feedback of pseudo-modal features, achieving efficient and robust adversarial training, enhancing multimodal fusion capabilities, and improving completion performance in modality missing scenarios.

[0120] The algorithm principle for model training in the above steps is as follows: Figure 2 As shown.

[0121] Example 2

[0122] The difference from Example 1 is that, as Figure 3 As shown: It includes:

[0123] S1. Construct a candidate triplet set for the tail entity prediction task.

[0124] For a given partial triple (h,r,t'), where the head entity h and relation r are known and the tail entity t is to be predicted, enumerate all possible tail entities t′∈E from the entity set E to construct a candidate triple set:

[0125] C = {(h,r,t')|t'∈E}

[0126] This candidate set covers all possible positions to fill, for selection during subsequent scoring.

[0127] S2. Fuse modal features to generate multimodal entity embeddings.

[0128] For each candidate triplet, the head entity h and the candidate tail entity t′ are respectively obtained through the modality fusion module to obtain their final multimodal embedding representations. The three modal features are fused through a fine-grained attention mechanism to output a unified embedding representation F. h F t The relation r also obtains its vector representation r through the embedding layer.

[0129] S3. Score candidate triples using the RotatE decoder.

[0130] Semantic scoring is performed on all candidate triples (h,r,t') using the RotatE decoding function, which is consistent during the training phase:

[0131]

[0132] This scoring function measures the similarity between the head entity and the tail entity in a triple after relational transformation; a higher score indicates stronger semantic consistency.

[0133] S4. Select the best completion result based on the score.

[0134] Sort all triplet scores in the candidate set and select the highest-scoring triples as the final completion results. A Top-K strategy is used: select the top K triplet scores (e.g., the top 1).

[0135] The final completion result set is:

[0136]

[0137] S55. Update the knowledge graph and output the results.

[0138] The completed tail entity triples are added to the original knowledge graph to construct a new set of knowledge triples. The system uses the completed result as the prediction output for downstream tasks, such as expanding candidate answers in question-answering systems, building relationship chains in recommendation systems, or iterating knowledge graph enhancement training.

[0139] The above description is merely an illustrative embodiment of the present invention and is not intended to limit the scope of the invention. Any equivalent changes and modifications made by those skilled in the art without departing from the concept and principles of the present invention should fall within the scope of protection of the present invention. Furthermore, it should be noted that the components of the present invention are not limited to the overall application described above. Each technical feature described in the specification can be used individually or in combination as needed. Therefore, the present invention naturally covers other combinations and specific applications related to this case.

Claims

1. A multimodal knowledge graph completion method based on generative adversarial networks, characterized in that, Includes the following steps: A. Training the knowledge graph completion model: A1. By using a contrastive learning pre-trained model to encode images and text, deep semantic features of entities in image and text modalities are extracted and mapped to a unified embedding space to achieve preliminary modal alignment. A2. Construct a modal fusion module with a fine-grained attention mechanism, and adaptively adjust the feature fusion weights according to the importance of each modal feature in different task scenarios to achieve effective information integration; A3. Introduce a generative adversarial network based on a convolutional structure to simulate missing modalities during training. Construct high-quality pseudo-modal features through a generator to enhance the model's robustness to incomplete modalities. A4. Use the RotatE scoring function to measure the rationality of triples, and combine it with a self-adversarial negative sampling strategy to optimize the training process and improve the model's reasoning ability in complex multimodal environments. B. Performing completion tasks based on knowledge graph completion models: The trained knowledge graph completion model is used for actual knowledge graph completion tasks. First, structural information, image information, and text information are encoded and fused to generate multimodal embedding representations of entities. Then, candidate triples are scored using a scoring function, and the completion result with the highest score is selected. Finally, the completion result with the highest score is chosen.

2. The method for multimodal knowledge graph completion based on generative adversarial networks according to claim 1, characterized in that: In step A1, encoding images and text through a contrastive learning pre-trained model specifically includes: A11, Visual input I for each entity e e The feature vector F is generated by encoding the visual encoder part of the pre-trained encoder CLIP (Contrastive Language–Image Pre-training) model. V The processed vectors are projected into the embedding space to obtain the embedding features of the visual modality. F V =proj(CLIP ViT (I e )); A12. Text input I for each entity e e The text encoder part of the pre-trained encoder CLIP model is used to encode the text description of the entity, generating a text modal feature vector F. T : F T =proj(CLIP Text (T e )); A13. Structural input for each entity e e It is encoded using a trainable embedding layer to generate structural modality features F. S Specifically, each entity in the knowledge graph is first assigned a unique index number, and then discrete entity identifiers are mapped to continuous dense vector representations through an embedding matrix: F S =Embedding(S e )。 3. The method for multimodal knowledge graph completion based on generative adversarial networks according to claim 1, characterized in that: In step A2, the modal fusion module for constructing a fine-grained attention mechanism adaptively adjusts the feature fusion weights based on the importance of each modal feature in different scenarios, specifically including: A21. Perform a unified transformation of the identifier space on visual, textual, and structural modal features, and input modal features F. V F T F S The latent representation a is generated by mapping through independent linear transformations. V a T a S : a i =W i F i +b i ,i∈{V,T,S}; A22. The modal features {a} after linear transformation V a T a S The attention tensor A is formed by stacking elements along the modal dimension. The Softmax function is then applied to tensor A along the modal dimension to generate the modal attention weight matrix W. attn : A=stack(a V 、a T 、a S ) W attn =Softmax(A); A23. The original input modal features F V F T F S The weighted modal features are obtained by performing element-wise multiplication with the corresponding attention weights. i∈{V,T,S}.

4. The method for multimodal knowledge graph completion based on generative adversarial networks according to claim 1, characterized in that: In step A3, the introduction of a generative adversarial network based on a convolutional structure specifically includes: A31. Construct a generator module, with the input being the structural embedding F of the entity. S The noise vector e sampled from the standard normal distribution noise The vector concatenated from ~N(0,1) is first transformed using a linear transformation to generate an initial feature map: fc out =reshape(ReLU(W1·concat(e noise ,F S )+b1)); A32. Input the initial feature map into the ConBlocks module, which contains two convolutional layers and normalization, to extract high-dimensional local feature representations. One of the modules is represented as follows: Conv_out=ReLU(BatchNorm(Conv2d(fC out ))); A33. Flatten the convolutional output into a one-dimensional vector and map it to an output embedding of the same dimension as the structural features through a linear transformation: C onvoutflat =reshape(conv out, [batchsize,*]) G n =W4·Conv_out_flat+b4。 5. The method for multimodal knowledge graph completion based on generative adversarial networks according to claim 1, characterized in that: In step A4, the use of the RotatE scoring function to measure the rationality of triples and the optimization of the training process by combining a self-adversarial negative sampling strategy specifically include: A41. Use the RotatE model to embed entities and relationships into the model. Specifically, embed the head and tail entities into F... h F t The relation embedding r = Embedding(r) is modeled as a complex rotation operation modulo 1, such that the predicted representation of the tail entity in the triple is... in This represents element-wise complex multiplication; The semantic reasonableness score of the triple (h,r,t) is calculated using the following RotatE scoring function; a higher score indicates a more reasonable prediction. Where ||·||2 is the l2 norm, i.e. A42. Based on the RotatE score, construct a training loss function. For each positive triple (h,r,t)∈T, jointly consider k negative samples, and use the sigmoid function to define and calculate the following loss function: Where γ is the score margin constant, σ is the sigmoid function, (h′ i ,r′ i ,t′ i ) represents the negative triples generated by negative sampling, where p is the negative sample. i The self-adversarial weights are calculated as follows: Where β is the temperature parameter, the main training objective of the model is to minimize the loss. kgc ; A43. Calculate the loss in adversarial training as follows:

6. The method for multimodal knowledge graph completion based on generative adversarial networks according to claim 5, characterized in that, Minimize the loss using stochastic gradient descent, and alternately optimize the loss. kgc and Loss adv The objective function is such that the discriminator can better distinguish between real and fake multimodal features, while the generator continuously improves the quality of fake features, thereby achieving effective learning of multimodal information. The overall training objective is: my Loss kgc +min maxλLoss adv 。

Images