Multi-modal knowledge graph completion method and system, electronic equipment, medium and product

Abstract

The embodiment of the invention provides a multi-modal knowledge graph completion method and system, electronic equipment, a medium and a product, and relates to the technical field of artificial intelligence. The method comprises the following steps: performing feature preprocessing on a plurality of instances of a target entity in a plurality of modalities to generate modal entity representation; according to the modal entity representation, performing processing by using a pre-constructed first loss function and a pre-constructed second loss function, and determining each modal representation after decoupling optimization and alignment optimization; determining a third loss function for link prediction loss according to each optimized modal representation and a gating attention mechanism; constructing a total loss function according to the first loss function, the second loss function, the third loss function and a preset hyper-parameter; and according to a preset framework of multi-modal image fusion and cross-modal alignment and the total loss function, determining a magic hundred-box recommendation model based on multi-modal knowledge graph completion. According to the scheme, the utilization and integration efficiency of the multi-modal features can be improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to a method, system, electronic device, medium, and product for completing a multimodal knowledge graph. Background Technology

[0002] With the development of multimodal data processing technology, the multimodal knowledge graph completion task has become a research hotspot in the field of knowledge graphs. Its core objective is to complete missing entities or relations in knowledge graphs by mining multimodal information. Existing technical solutions for multimodal knowledge graph completion (MMKGC) tasks are mainly divided into two categories: one is single-modal knowledge graph completion methods based on multi-view fusion strategies, and the other is multimodal fusion methods. Single-modal methods extract structural information from the knowledge graph to complete the embedding learning of entities and relations. To capture the diverse features within a modality, related technologies have proposed multi-view fusion strategies, which improve representation learning performance by integrating different feature perspectives from the same modality. Multimodal fusion methods enhance entity representations by fusing heterogeneous modal data such as text and images. One of its key technologies is cross-modal alignment technology, which achieves semantic alignment between different modal data through a contrastive learning mechanism, thereby improving the fusion quality of multimodal information.

[0003] When applied to MMKGC tasks, existing technologies have the following unavoidable technical drawbacks: (1) Existing multi-view fusion methods generally adopt simple weighted average or feature splicing strategies, which fail to deeply explore the complementary information between multiple views, resulting in poor multi-view fusion effect and failing to provide sufficient feature support for entity representation. (2) Existing cross-modal alignment methods lack a constraint mechanism for the consistency of training objectives in different modalities during model training, resulting in insufficient information fusion between modalities and restricting the accuracy of cross-modal entity representation; (3) Existing modal feature decoupling methods fail to achieve effective separation of different modal features, resulting in serious information redundancy between modalities. This not only reduces the running efficiency of the model, but also has a negative impact on the overall performance of knowledge graph completion. Summary of the Invention

[0004] At least one embodiment of this application provides a multimodal knowledge graph completion method, system, electronic device, medium, and product to address the problem of insufficient effective utilization and integration of features in the prior art during multimodal information processing.

[0005] To solve the above-mentioned technical problems, this application is implemented as follows:

[0006] In a first aspect, embodiments of this application provide a multimodal knowledge graph completion method, including:

[0007] Feature preprocessing is performed on multiple instances of the target entity in multiple modalities to generate modal entity representations;

[0008] Based on the modal entity representation, decoupling optimization is performed using a pre-constructed first loss function for modal decoupling, and alignment optimization is performed using a pre-constructed second loss function for modal alignment, thereby determining each modal representation after decoupling optimization and alignment optimization;

[0009] Based on the optimized modal representations and gating attention mechanism, a third loss function is determined for linking prediction loss; the link is used to represent the association between single-modal entities and / or the association between cross-modal entities;

[0010] Based on the first loss function, the second loss function, the third loss function, and the preset hyperparameters, construct the total loss function;

[0011] Based on the preset framework of multimodal image fusion and cross-modal alignment and the total loss function, a Magic Box recommendation model based on multimodal knowledge graph completion is determined.

[0012] Optionally, feature preprocessing is performed on multiple instances of the target entity in multiple modalities to generate modal entity representations, including:

[0013] A pre-trained model is used to extract the feature vector of each instance from the multiple instances;

[0014] Based on the trainable weight matrix, the structural embedding of the target entity, the relationship embedding between the target entities, and the bias term obtained by training and tuning using a preset graph neural network, a nonlinear activation function is used to calculate the gate vector corresponding to the feature vector.

[0015] The gating vector and the feature vector are multiplied element-wise, and the weights of each operation result are calculated using cosine similarity. The feature vectors of all instances are then summed according to the weights to generate a modal entity representation.

[0016] Optionally, based on the modal entity representation, decoupling optimization is performed using a pre-constructed first loss function for modal decoupling to determine the decoupling optimized modal representations, including:

[0017] Based on the modal entity representation, determine the modal entity representation of the target entity under different modes;

[0018] Based on the modal entity representations under different modalities, determine the joint probability distribution corresponding to the modal entity representations under different modalities, and the marginal distribution corresponding to each modal entity representation;

[0019] Obtain a pre-constructed first loss function for modal decoupling; the first loss function is determined based on the joint probability distribution, the marginal distribution, and the reciprocal of the total number of samples corresponding to the target entity;

[0020] By minimizing the loss value of the first loss function, the decoupled optimized modal representations are determined.

[0021] Optionally, based on the modal entity representation, alignment optimization is performed using a pre-constructed second loss function for modal alignment to determine the decoupled optimized modal representations, including:

[0022] Based on the modal entity representation, determine the modal entity representation of the target entity under different modes after decoupling;

[0023] Obtain a pre-constructed second loss function for modal alignment; the second loss function is determined based on the modal entity representations under different modes after decoupling, a preset consistency loss function, and a preset learnable temperature parameter;

[0024] The modal representations after modal alignment optimization are determined by minimizing the loss value of the second loss function.

[0025] Optionally, based on the optimized modal representations and gated attention mechanism, a third loss function is determined for linking the prediction loss, including:

[0026] Based on the optimized modal representations, each modal representation is processed using a preset modal projection function to obtain the target embedding for each modality;

[0027] Based on the gating attention mechanism, the target embedding is weighted by combining learnable weights to obtain the attention weight coefficients of the corresponding modality;

[0028] Based on the attention weight coefficients, the target embeddings of all modalities are weighted and summed to obtain a joint representation;

[0029] Based on the joint characterization, a third loss function is determined for linking the prediction loss.

[0030] Optionally, based on the joint characterization, a third loss function is determined for linking the prediction loss, including:

[0031] Based on the joint representation and the preset scoring function, a set of triples is determined; each triple in the set of triples is used to represent the link score of the head entity, relation, and tail entity; the preset scoring function captures the high-order interaction between modal embeddings through a preset tensor and integrates each embedding representation in the form of the i-th dimension tensor product;

[0032] Based on the set of triples, a set of positive samples and a set of negative samples are determined, and based on the set of positive samples and the set of negative samples, a third loss function for link prediction loss is determined; the set of positive samples is a set of positive sample triples corresponding to known facts in a preset multimodal knowledge graph, and the set of negative samples is a set of non-known link triples generated by randomly replacing tail entities in the set of positive sample triples.

[0033] Secondly, embodiments of this application provide a multimodal knowledge graph completion system, including:

[0034] The first processing module is used to perform feature preprocessing on multiple instances of the target entity in multiple modalities to generate modal entity representations;

[0035] The first determining module is used to perform decoupling optimization using a pre-constructed first loss function for modal decoupling and alignment optimization using a pre-constructed second loss function for modal alignment based on the modal entity representation, thereby determining each modal representation after decoupling optimization and alignment optimization.

[0036] The second determining module is used to determine a third loss function for linking the prediction loss based on the optimized modal representations and the gating attention mechanism; the link is used to represent the association relationship between single-modal entities and / or the association relationship between cross-modal entities;

[0037] The second processing module is used to construct a total loss function based on the first loss function, the second loss function, the third loss function, and preset hyperparameters;

[0038] The third determining module is used to determine the Magic Box recommendation model based on multimodal knowledge graph completion according to the preset framework of multimodal image fusion and cross-modal alignment and the total loss function.

[0039] Thirdly, embodiments of this application provide an electronic device, including a memory, a transceiver, and a processor; the memory is used to store a computer program; the transceiver is used to send and receive data under the control of the processor.

[0040] The processor is used to perform feature preprocessing on multiple instances of the target entity in multiple modalities to generate modal entity representations;

[0041] The processor is further configured to perform decoupling optimization using a pre-constructed first loss function for modal decoupling based on the modal entity representation, and to perform alignment optimization using a pre-constructed second loss function for modal alignment, thereby determining each modal representation after decoupling optimization and alignment optimization;

[0042] The processor is further configured to determine a third loss function for linking prediction loss based on the optimized modal representations and the gated attention mechanism; the link is used to represent the association relationship between single-modal entities and / or the association relationship between cross-modal entities;

[0043] The processor is further configured to construct a total loss function based on the first loss function, the second loss function, the third loss function, and preset hyperparameters;

[0044] The processor is also used to determine a Magic Box recommendation model based on multimodal knowledge graph completion, according to a preset framework of multimodal image fusion and cross-modal alignment and the total loss function.

[0045] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method as described in any one of the first aspects.

[0046] Fifthly, embodiments of this application provide a computer program product including computer instructions that, when executed by a processor, implement the steps of the method as described in any one of the first aspects.

[0047] Compared with existing technologies, the multimodal knowledge graph completion method, system, electronic device, medium, and product provided in this application solve the fundamental problem of inconsistent multimodal data formats by preprocessing the features of multimodal instances of target entities to generate modal entity representations in a unified format; it uses a pre-constructed first loss function to achieve modal decoupling, avoiding mutual interference between modal features, and a second loss function to achieve modal alignment, ensuring cross-modal semantic consistency and effectively solving the problems of modal conflict and semantic disconnect; it introduces a gated attention mechanism and constructs a third loss function to adapt to single and / or cross-modal entity association prediction, dynamically select key features, and improve the accuracy of feature utilization; it combines the three types of loss functions with preset hyperparameters to construct a total loss function, achieving comprehensive model optimization and avoiding insufficient feature integration caused by single-dimensional optimization; based on the preset framework of multimodal image fusion and cross-modal alignment, it forms an end-to-end Magic Box recommendation model, improving the multimodal feature utilization and integration capabilities, while optimizing the knowledge graph completion effect, ultimately improving the accuracy and reliability of the recommendation model. Attached Figure Description

[0048] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0049] Figure 1A flowchart illustrating the multimodal knowledge graph completion method provided in this application embodiment;

[0050] Figure 2 This is a schematic diagram of the structure of IMVIA provided in an embodiment of this application;

[0051] Figure 3 This is a schematic diagram of the structure of the multimodal knowledge graph completion system provided in the embodiments of this application;

[0052] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0053] The terms "first," "second," etc., used in this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same class, without limiting the number of objects; for example, the first object can be one or more. Furthermore, "or" in this application indicates at least one of the connected objects. For example, "A or B" covers three scenarios: Scenario 1: including A but not B; Scenario 2: including B but not A; Scenario 3: including both A and B. The character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0054] The term "instruction" in this application can be either a direct instruction (or explicit instruction) or an indirect instruction (or implicit instruction). A direct instruction can be understood as one in which the sender explicitly informs the receiver of specific information, the operation to be performed, or the requested result, etc.; an indirect instruction can be understood as one in which the receiver determines the corresponding information based on the instruction sent by the sender, or makes a judgment and determines the operation to be performed or the requested result, etc., based on the judgment result.

[0055] To enable those skilled in the art to better understand the embodiments of this application, the following description is provided first:

[0056] For the Multimodal Knowledge Graph Completion (MMKGC) task in the Magic Box, relevant technologies mainly include single-modal knowledge graph completion methods based on multi-view fusion strategies and multimodal fusion methods. Single-modal methods utilize the structural information of the knowledge graph for entity and relation embedding learning, with multi-view fusion strategies proposed to capture intramodal diversity. Multimodal methods enhance entity representation by fusing multimodal data such as text and images, with cross-modal alignment techniques achieving intermodal alignment through contrastive learning.

[0057] MoMoK (Mixture of Modality Knowledge Experts) and MNF (Multi-hop Neighbor Fusion) are two different multimodal knowledge processing and graph-structured data aggregation techniques, applied respectively to expert-level information integration of multimodal knowledge graphs and multi-level neighbor node fusion of graph convolutional networks. MoMoK is a hybrid expert model that combines multimodal knowledge, improving the efficiency and accuracy of multimodal knowledge graph processing by dynamically routing and weighting knowledge experts from different modalities. MoMoK captures unique features of different modalities through multimodal expert models, while MNF is a multi-level neighbor node fusion technique in graph convolutional networks, enhancing node representation by aggregating neighbor information with different hop counts. Both methods attempt to improve the performance of knowledge graph completion through multi-view fusion and cross-modal alignment. Their application scenarios differ: MoMoK focuses on multimodal expert-level information integration, while MNF focuses on multi-level neighbor aggregation of graph-structured data.

[0058] This application provides a method, system, electronic device, medium, and product for multimodal knowledge graph completion. The method and system are based on the same concept. Since the methods and systems solve problems based on similar principles, their implementations can be mutually referenced, and repeated details will not be elaborated further.

[0059] Please refer to Figure 1 This application provides a multimodal knowledge graph completion method, including:

[0060] Step 11: Perform feature preprocessing on multiple instances of the target entity in multiple modalities to generate modal entity representations.

[0061] It should be noted that the target entity here refers to the core entity in the Magic Box recommendation scenario, including but not limited to: film and television content (movies, TV series), users, tags (such as comedy, action, etc.), actors, etc. Multiple modalities include but are not limited to text modalities, such as movie synopses and user reviews; image modalities, such as cover images and screenshots; and audio modalities, such as trailer clips. Multiple instances refer to multiple sets of data for the same entity within the same modality, such as multiple cover images or multiple synopses for a movie. Feature preprocessing employs appropriate encoding methods for different modalities. For example, text is converted into semantic vectors using models such as BERT, images use ResNet to extract visual features, and audio is converted into audio features using MFCC. Modal entity representations are used to represent the initial feature vectors of each entity in each modality, such as the text feature vector and image feature vector for movie A, addressing the problem of inconsistent raw data formats and the inability to directly compute them.

[0062] This application addresses the issue of heterogeneous multimodal data formats by generating modal entity representations through feature preprocessing. It transforms raw multimodal data into a computable, unified feature form, laying the foundation for subsequent processing.

[0063] Step 12: Based on the modal entity representation, perform decoupling optimization using a pre-constructed first loss function for modal decoupling, and perform alignment optimization using a pre-constructed second loss function for modal alignment, to determine each modal representation after decoupling optimization and alignment optimization.

[0064] In this application, based on the modal entity representations, decoupling optimization is performed using a pre-constructed first loss function (modal decoupling loss), and alignment optimization is performed using a pre-constructed second loss function (modal alignment loss), ultimately determining the decoupled and aligned modal representations. The core purpose of the first loss function is to separate the private features of different modalities, such as the semantic logic of the text modality and the color composition of the image modality, avoiding mutual interference between features across modalities. This is achieved by minimizing the mutual information between entity representations of different modalities, thereby enhancing the correlation between features within the same modality and weakening the correlation between private features of different modalities. The core purpose of the second loss function is to ensure that the common semantics describing the same entity in different modalities remain consistent. This is achieved by narrowing the distance between the feature representations of the same entity in different modalities, ensuring no disconnect in cross-modal semantics. After joint optimization using dual loss functions, each modal representation retains its unique value, such as the plot information of the text and the visual appeal of the image, while also achieving cross-modal semantic unification. This lays the foundation for subsequent multimodal feature integration and effectively solves the key problem of modal feature interference and cross-modal semantic disconnect in traditional methods.

[0065] Step 13: Based on the optimized modal representations and gating attention mechanism, determine the third loss function used to link the prediction loss; the link is used to represent the association between single-modal entities and / or the association between cross-modal entities.

[0066] In this application, based on the optimized modal representations, features are dynamically filtered using a gated attention mechanism to determine the third loss function used for link prediction loss. Linkage specifically refers to the association between single-modal entities, such as movie Y belonging to type Z, and the association between cross-modal entities, such as movie Y corresponding to poster image P. The gated attention mechanism introduces a trainable weight adjustment module to dynamically allocate the weights of each modal feature according to the needs of the recommendation scenario: for example, when recommending animations to children, the weight of image modalities (cover cuteness, character design) is automatically increased, while the weight of text modalities (plot summary) is decreased, achieving the filtering of secondary features and the focus on key features. Based on the features filtered by the gated attention mechanism, the third loss function takes entity relationship prediction in knowledge graph completion as its core objective. By quantifying the difference between the predicted rationality and the actual association of triples (head entity-relationship-tail entity), it drives the model to learn accurate entity association rules. This step, through dynamic weight allocation and relationship prediction loss constraints, solves the problems of unreasonable feature weight allocation and interference from invalid features, enabling the model to focus on the core entity relationship reasoning requirements of the recommendation task.

[0067] It should be noted that gated attention mechanisms are classic techniques for feature fusion and weight allocation in deep learning. Their core principle involves dynamically adjusting feature contributions through gating units, and they have been widely applied in natural language processing, computer vision, and recommendation systems. This application, based on the principles of gated attention mechanisms, is specifically adapted to the particular scenario of movie recommendation, focusing on the judgment criteria for weight adjustment and modal adaptability. Traditional gated attention mechanisms often adjust weights based on the relevance of features themselves, while this application further integrates the requirements of the recommendation scenario into the weight allocation logic. This leverages the stability of existing technologies and solves the problem of static modal weights in recommendation tasks through scenario-based adaptation.

[0068] Step 14: Construct the total loss function based on the first loss function, the second loss function, the third loss function, and the preset hyperparameters;

[0069] Step 15: Based on the preset framework of multimodal image fusion and cross-modal alignment and the total loss function, determine the Magic Box recommendation model based on multimodal knowledge graph completion.

[0070] In this embodiment, multi-dimensional losses are integrated to achieve global model optimization. Step 14 constructs a total loss function to address the one-sidedness of a single optimization objective. Hyperparameters play a crucial role in balancing the importance of different losses. The total loss function avoids the problem of a single loss causing the model to focus only on modal alignment while neglecting recommendation relationship prediction, ensuring comprehensive optimization of the model from feature quality (such as decoupling and alignment) to link prediction. Step 115 completes model training and deployment based on the preset framework and the total loss function. The preset framework for multimodal image fusion and cross-modal alignment refers to the technical architecture integrating steps 11 to 14, such as a neural network including a feature encoder, decoupling and alignment module, gating attention module, and link prediction module. The model optimizes parameters through backpropagation using the total loss function, ultimately outputting a model that can utilize the multimodal knowledge graph completion results for recommendation. This step 15 integrates the preceding optimizations into a deployable recommendation model, ultimately solving the problem of insufficient effective utilization and integration of multimodal features and improving recommendation accuracy.

[0071] This application captures intramodal variability through a multi-view representation mechanism. By introducing a strategy combining information decoupling and contrastive learning, each modality-specific learner can focus on feature extraction while maintaining consistency in training objectives across all modalities. Finally, a relation-aware gated fusion network is designed to integrate multimodal decisions while considering relational context, thereby enhancing the model's reasoning ability in complex MMKGC tasks.

[0072] It should be noted that the preset framework in this application adopts the unified framework proposed in this application, which combines intra-modality multi-view fusion with cross-modality alignment (IMVIA), as referenced. Figure 2The multimodal knowledge graph construction and reasoning framework shown, taking multimodal input information as an example, integrates information from three modalities: text, vision, and structure, achieving effective utilization and integration of multimodal features. In the input layer, multimodal data is accessed: text modality (Text), visual modality (Visual), and structural modality (Structural), which presents the structural relationships between entities in the form of knowledge graph subgraphs. In the encoding layer, multimodal feature extraction is performed. The text encoder transforms the input text into sequence features, capturing the semantic information of the text. The image encoder extracts multi-layer features from each image and cross-associates them with the subgraph features of the structural modality, achieving preliminary fusion of visual and structural features. The subgraph encoder transforms the knowledge graph subgraphs into a sequence of structural features, capturing the structural relationships between entities. Contrastive learning and modality alignment are performed at the feature optimization layer. Contrastive learning is used to align and decouple multimodal features: integrate multi-view visual features, textual features, and structural features. Contrastive learning optimizes the semantic consistency of different modal features while decoupling irrelevant features, ensuring that the modalities are aligned without interference.

[0073] In the fusion and prediction layer, multimodal knowledge reasoning is performed. In the fusion module, the completion and reasoning of the multimodal knowledge graph are achieved through the following steps: generating entity embeddings and relation embeddings, corresponding to the head, tail, and relation entities in the knowledge graph, respectively. Multimodal embeddings are fused using techniques such as Tucker decomposition to output an overall prediction score, while also supporting multi-dimensional sub-task predictions, such as multimodal scoring, structural prediction, visual prediction, and text prediction, thus achieving joint reasoning of multimodal knowledge. This framework, through a complete design encompassing multimodal encoding, contrastive learning alignment and decoupling, multimodal embedding fusion, and knowledge reasoning, achieves unified representation of multimodal features (textual, visual, and structural features are all transformed into computable embeddings); semantic alignment between modalities (contrastive learning ensures semantic consistency in the description of the same entity across different modalities); and effective feature integration (the fusion module jointly uses multimodal embeddings for knowledge graph completion). Ultimately, this architecture can fully utilize multimodal information to improve the performance of knowledge graphs in tasks such as entity linking and relationship prediction, and solves the problem of insufficient effective utilization and integration of multimodal features.

[0074] It should be noted that this application can define a knowledge graph as follows: It consists of a collection of entities Relation set and multimodal data Composition. Defining entities As a node in a knowledge graph, it represents a specific object or concept, defined as follows: It is a physical entity In modality The next specific form of expression (such as image, text, etc.), its first Each instance is denoted as , can be regarded as the first Each view. The goal of this task is to learn multimodal entity embeddings. To predict missing triples ,in For the head entity, For the relationship, It is a tail entity.

[0075] In MMKGC, entities are typically represented in various forms, such as images, text, or other forms of media. Within each modality, an entity... This may be associated with multiple views or instances, such as different images captured under different conditions or multiple text descriptions from different sources. How can these multi-view representations be effectively aggregated into a single modality to form a unified, robust entity representation? Thus accurately capturing entities In modal This application employs an intra-modal multi-view fusion method, comprising three steps: feature extraction, gating control, and feature aggregation. The key to intra-modal multi-view fusion lies in effectively capturing the diverse variability of entities within a single modality (such as images or text), avoiding information loss caused by simple averaging or splicing. Through a dynamic aggregation mechanism (including feature extraction, gating control, and feature aggregation), the contribution weight of each view (instance) is adaptively adjusted in conjunction with the relational context, forming a unified and robust modality-specific entity representation. This solves the problem of unsatisfactory multi-view fusion in related technologies, improving the richness and accuracy of entity representation. Specifically, by introducing gating vectors and similarity weighting, it ensures that task-related view features are prioritized, thus making it suitable for complex scenarios of multimodal data in movie and TV recommendation, such as image or text descriptions from different sources.

[0076] Optionally, feature preprocessing is performed on multiple instances of the target entity in multiple modalities to generate modal entity representations, including:

[0077] A pre-trained model is used to extract the feature vector of each instance from the multiple instances;

[0078] Based on the trainable weight matrix, the structural embedding of the target entity, the relationship embedding between the target entities, and the bias term obtained by training and tuning using a preset graph neural network, a nonlinear activation function is used to calculate the gate vector corresponding to the feature vector.

[0079] The gating vector and the feature vector are multiplied element-wise, and the weights of each operation result are calculated using cosine similarity. The feature vectors of all instances are then summed according to the weights to generate a modal entity representation.

[0080] In this embodiment, during the intramodal multi-view fusion process, the layers are linked together through a "layer-by-layer processing and feedback" mechanism: the output of the previous step directly serves as the input of the next step, forming a pipeline processing chain from the original instance to the fused representation. Simultaneously, the relational context (the concatenation of entity structure embedding and relation embedding) runs throughout the entire process, acting as a regulatory factor to ensure the task-oriented nature of the fusion process. In feature extraction, for entities... In modality The next An example The pre-trained model CLIP is used to extract features. The pre-trained model CLIP extracts features using the following formula: ;in, This represents the raw instance data of the input, which is the raw material for feature extraction. 'm' represents modality, such as text modality or image modality, used to distinguish different types of information. 'i' represents entity, corresponding to the target entity. , used to locate the core entity to which the feature belongs, k represents the instance number, which represents the kth specific sample of the same entity in the same modality; This represents the feature encoder for modality m in the pre-trained model CLIP; This represents the output of feature extraction, i.e., the modal entity feature vector obtained after CLIP encoding; It is modal Feature dimensions; Representing the eigenvector The mathematical space properties.

[0081] Furthermore, the extracted feature vectors This is directly passed to the gating control module as the basic input for calculating the gating vector. This is achieved by introducing a gating vector. To adjust each feature The contribution of the gating vector is as follows: ;in, It is a trainable weight matrix that is dynamically optimized through model training, and the initial values ​​are initialized using Gaussian randomization. It is a concatenation of the embedding obtained by vectorizing the structural information of entities and the embedding of relationships; It is a bias term obtained through training and tuning of a graph neural network; It is a non-linear activation function, such as the Sigmoid or ReLU function.

[0082] Furthermore, the gating vector With feature vectors These are fed into the feature aggregation module as a control factor for weighted calculation. The weight of each feature is calculated using a similarity function. And by weighted summation, we obtain the modality-specific modal entity representation. Weights of each feature Represented as: Optional, modal entity representation. This can be expressed by the formula: .in, Cosine similarity; This represents element-wise product. It ensures that the feature is most consistent with the relational context. It has been given greater importance.

[0083] It's important to note that the key to cross-modal decoupling and alignment lies in ensuring that each modality focuses on learning features while maintaining consistency in training objectives across all modalities. Through a dual strategy, modal decoupling minimizes mutual information to reduce redundancy, while modal alignment synchronizes objectives through contrastive learning. This addresses the issues of information redundancy and objective inconsistency among modalities in related technologies, improving the accuracy of entity representations and the model's inference capabilities in MMKGC tasks (such as movie and TV recommendation). Cross-modal decoupling and alignment directly rely on the output of step 11 as its input. Specifically, intra-modal multi-view fusion generates modality-specific entity representations through feature extraction, gating control, and feature aggregation steps. This refers to the fusion representation of multiple views within a single modality. These representations serve as the starting point input for cross-modal decoupling and alignment, and are used to further address information redundancy and goal consistency issues between modalities.

[0084] Furthermore, by combining information decoupling loss and contrastive learning loss, a balance between modal independence and complementarity is ensured. Optionally, based on the modal entity representation, decoupling optimization is performed using a pre-constructed first loss function for modal decoupling to determine the decoupled optimized modal representations, including:

[0085] Based on the modal entity representation, determine the modal entity representation of the target entity under different modes;

[0086] Based on the modal entity representations under different modalities, determine the joint probability distribution corresponding to the modal entity representations under different modalities, and the marginal distribution corresponding to each modal entity representation;

[0087] Obtain a pre-constructed first loss function for modal decoupling; the first loss function is determined based on the joint probability distribution, the marginal distribution, and the reciprocal of the total number of samples corresponding to the target entity;

[0088] By minimizing the loss value of the first loss function, the decoupled optimized modal representations are determined.

[0089] In this embodiment, considering the need to fuse a large amount of text and image information in the movie recommendation process, it is crucial in MMKGC to ensure that each modality focuses on effectively learning features. To achieve effective deentanglement of inter-modal learning, this application introduces a first loss function for modal decoupling. The object of decoupling is the modality-specific entity representation of any two different modalities. Based on the modal entity representation, different modal entity representations can be obtained. and (Fusion representations of different modalities) minimize the mutual information between learned representations, used to unravel the learning process under different modalities to maximize their individual contributions. First loss function. Defined as: Where m and n both represent modes, and m≠n limits the consideration to cross-modal combinations only, rather than within the same mode; The total number of samples corresponding to the target entity is represented by the output of step 11, which is the modal entity representation generated after feature preprocessing of multiple instances of each target entity in each modality. For example, if there are 100 target entities, each entity corresponds to 3 modalities, such as text, vision, and structure, then the total number of samples D is |D|=100×3=300. This indicates that the loss is averaged over all samples to avoid the influence of differences in sample size on the loss value and to ensure the stability of the loss scale. The joint probability distribution of the encoded features of entity i under modality m and modality n; and It is their marginal distribution; Mutual information (MI) measures the dependency between two random variables: a larger value indicates a stronger correlation between the two variables (i.e., two modal features); a value of 0 indicates that the two variables are completely independent (without redundancy). By minimizing the loss based on mutual information, this model encourages each modal encoder to learn specialized features, thereby reducing cross-modal redundancy.

[0090] Optionally, based on the modal entity representation, alignment optimization is performed using a pre-constructed second loss function for modal alignment to determine the decoupled optimized modal representations, including:

[0091] Based on the modal entity representation, determine the modal entity representation of the target entity under different modes after decoupling;

[0092] Obtain a pre-constructed second loss function for modal alignment; the second loss function is determined based on the modal entity representations under different modes after decoupling, a preset consistency loss function, and a preset learnable temperature parameter;

[0093] The modal representations after modal alignment optimization are determined by minimizing the loss value of the second loss function.

[0094] In this embodiment, the core of modal decoupling is to separate the private features of each modality, such as text semantics and image color, to avoid feature interference. However, decoupling alone can easily lead to deviations in the learning objectives of different modalities. That is, the representation of the same entity in different modalities may show semantic disconnect. For example, the text may describe an entity in a science fiction movie, but its image features may be biased towards a comedic style. Therefore, achieving modal alignment based on decoupling is crucial. Its goal is to ensure that the core semantics of different modalities describing the same entity remain consistent, providing a unified semantic benchmark for subsequent multimodal feature integration and recommendation tasks.

[0095] This application achieves modality alignment through contrastive learning, with the core component being a pre-constructed second loss function (denoted as...). This function uses the decoupled modal entity representation as the basis for computation and constructs the optimization objective by quantifying the semantic consistency of different modal representations.

[0096] This application aligns different modalities through comparative learning, such as using decoupled modal representations. and The training objective is the second loss function. Defined as: ;in, These are preset, learnable temperature parameters. It is a loss function that guarantees consistency; It represents the total number of modalities (such as the total number of modalities like text, image, and audio). This is used to average the loss across all cross-modal pairs, avoiding the influence of differences in the number of modes on the loss value and ensuring the stability of the loss scale; This represents the similarity score from mode m to mode n; This represents the inverse similarity score from mode n to mode m. By combining focused modal feature deentanglement and cross-modal alignment loss, IMVIA ensures that each modality contributes its expertise while maintaining a consistent prediction objective across all modalities.

[0097] It should be noted that the alignment optimization calculation process is as follows: Input data: Obtain the multimodal representation of the target entity after decoupling optimization; Score calculation: For any two different modalities (m≠n), calculate the bidirectional similarity scores for "modality m to modality n" and "modality n to modality m" respectively; Loss calculation: Input the bidirectional scores into the consistency loss function. Two sets of loss results are obtained and summed to form the alignment loss for that mode pair; Global average: The alignment loss of all mode pairs is summed and multiplied by Average it to get the final result. Loss value; parameter optimization: to minimize The loss value is the target, and the trainable parameters of modality feature extraction, decoupling and other processes are adjusted through backpropagation to make the semantic representations of different modalities tend to be consistent.

[0098] Optionally, based on the optimized modal representations and gated attention mechanism, a third loss function is determined for linking the prediction loss, including:

[0099] Based on the optimized modal representations, each modal representation is processed using a preset modal projection function to obtain the target embedding for each modality;

[0100] Based on the gating attention mechanism, the target embedding is weighted by combining learnable weights to obtain the attention weight coefficients of the corresponding modality;

[0101] Based on the attention weight coefficients, the target embeddings of all modalities are weighted and summed to obtain a joint representation;

[0102] Based on the joint characterization, a third loss function is determined for linking the prediction loss.

[0103] It is important to note that the key to multimodal joint decision-making lies in integrating all information from text, vision, structure, and joint modalities to achieve high-quality knowledge graph completion. By fusing joint representations and predicting scores through links, the complementary nature of multimodal approaches is fully utilized, and higher-order interaction capture is considered within the context of relationships. This addresses the problem of insufficient modal fusion in existing technologies and improves the model's reasoning ability in complex MMKGC tasks (such as movie and TV show recommendations in the Magic Box). This application provides a relationship-aware gated attention mechanism and a Tucker scoring function to ensure a dynamic and efficient decision-making process.

[0104] In this application, a representation aggregation and scoring feedback mechanism are linked together. Based on the optimized modal representations, each modal representation is processed through a preset modal projection function to obtain the target embedding of each modality. The embeddings of head entities, relations, and tail entities serve as regulatory factors throughout the process, ensuring task-oriented decision-making. In the MMKGC task, high-quality knowledge graph completion must utilize all information provided by textual, visual, structural, and joint modal representations. Specifically, in the joint representation fusion process, the joint representation calculation method of this application aggregates information from each modality. Specific embeddings to form a unified entity representation. .

[0105] Specifically, a weighted summation method is used to integrate the representations of different modalities into a joint representation: ;in, The multimodal joint feature of the i-th sample is the final unified feature obtained by fusing features from all modalities (such as text, image, and audio). It is used as input for subsequent tasks (classification, prediction, retrieval, etc.) and integrates complementary information from each modality to avoid the limitations of a single modality. M represents the set of all modalities, and m represents a single modality in the set M. This represents the original modal features of the i-th sample in the m-th modality, i.e., the optimized m-th modal representation; The preset modal projection function for the m-th mode; These are learnable weights. This representation enhances multimodal complementarity through an attention fusion mechanism.

[0106] Furthermore, based on the joint characterization, a third loss function is determined for linking the prediction loss, including:

[0107] Based on the joint representation and the preset scoring function, a set of triples is determined; each triple in the set of triples is used to represent the link score of the head entity, relation, and tail entity; the preset scoring function captures the high-order interaction between modal embeddings through a preset tensor and integrates each embedding representation in the form of the i-th dimension tensor product;

[0108] Based on the set of triples, a set of positive samples and a set of negative samples are determined, and based on the set of positive samples and the set of negative samples, a third loss function for link prediction loss is determined; the set of positive samples is a set of positive sample triples corresponding to known facts in a preset multimodal knowledge graph, and the set of negative samples is a set of non-known link triples generated by randomly replacing tail entities in the set of positive sample triples.

[0109] In this embodiment, link prediction is the core task of multimodal knowledge graph completion. The goal is to predict missing entity-relationship-entity links in the knowledge graph by learning the association patterns between entities through a model. Based on optimized multimodal features and joint embeddings, this application constructs a third loss function by quantifying reasonableness through a scoring function and optimizing prediction accuracy through a loss function. After obtaining textual, visual, structural, and joint representations, this application uses the Tucker scoring function to calculate the reasonableness of triples. This function effectively integrates multimodal embedding features and captures high-order cross-modal interaction information. The calculation formula is: ;in, Using the core tensor, it captures higher-order interactions between modality-specific embeddings. Indicates the first Multidimensional tensor product; This is used to reflect the degree of matching between the head entity h and the tail entity t, which are linked through relation r. The higher the score, the more likely the entity association represented by the triple is to be valid.

[0110] To optimize the accuracy of the Tucker scoring function in determining reasonableness, this application defines a third loss function. Through supervised learning using positive and negative triples, the model learns realistic entity association patterns. The third loss function... The formula is: ;in, and These are the sets of positive triples and the sets of negative triples, respectively. The set of positive triples... The positive samples are derived from known facts derived from the knowledge graph, specifically the existing complete triples (h, r, t) in the training dataset. These are observed entity-relation-entity links in the knowledge graph and are used for supervised learning to capture real-world patterns. In the MMKGC task, these positive samples are extracted directly from multimodal knowledge graphs (such as the Magic Box Movie & TV dataset), ensuring coverage of text, image, and other modalities. The negative triple set... Generated using a negative sampling strategy to simulate missing or faulty links.

[0111] It should be noted that the quality of the sample set directly determines the supervision effect of the loss function. This application constructs a loss function through real fact extraction and controlled negative sampling. and This ensures the effectiveness of the monitoring signal. (Set of positive triples) The data originates from known facts in multimodal knowledge graphs, specifically the complete triples (h, r, t) observed in the training dataset. These triples represent real entity-relation-entity links. In the Magic Box movie recommendation task... Extract directly from the Magic Box movie and TV dataset to ensure coverage of multimodal associations such as text (e.g., movie title-director-person name) and images (e.g., movie poster-corresponding work-movie title).

[0112] Set of negative triples Generated using a negative sampling strategy to simulate missing or broken links in a knowledge graph. Specifically, in each training batch, for each positive triplet... Uniformly sample an entity different from t from the entity set E. Replace the original tail entity to generate negative samples To ensure the authenticity of negative samples, triples already present in T₊ should be filtered out during sampling to avoid generating false negative samples.

[0113] The optimization objective of the third loss function in this application is to minimize The value of , through supervised learning, enables the model to achieve two main capabilities: first, to output a high reasonableness score for positive triples, making Approaching 0, the loss contribution is small; secondly, it outputs a lower reasonableness score for negative triples, making Approaching zero, the loss contribution is small. Through the optimization of this loss function, the model can effectively learn the mapping relationship between multimodal features and entity association patterns, accurately predict missing links in the knowledge graph, and provide core relational reasoning capabilities for the Magic Box recommendation task.

[0114] Furthermore, the final training objective of the model in this application combines three key components: multimodal disentanglement loss. Alignment loss and knowledge graph completion loss These components work together to ensure that the model effectively captures modality-specific features, learns to rank training objectives across modality features, and accurately predicts missing links in the knowledge graph.

[0115] Specifically, the total loss function for multiple tasks The definition is as follows: ;in, , These are hyperparameters that can be tuned based on the application data. Empirical values ​​can be used, such as 0.1 and 0.05. This application minimizes this total loss function while training all models.

[0116] This application uses a gating mechanism to aggregate multi-view features within a modality, ensuring that the diversity within each modality is fully captured; this application introduces information decoupling and contrastive learning strategies to ensure that each modality focuses on learning features while maintaining consistency in cross-modal training objectives; this application employs a relation-aware gating fusion network, which fully considers relational context and enhances the ability to integrate multimodal information.

[0117] This application achieves dynamic aggregation of multi-instance features within a modality through a gating mechanism. Features for each instance are extracted using a pre-trained model, such as information extracted from images from different perspectives or text descriptions from multiple sources. The weights of each instance are calculated by combining contextual information about entities and relationships, ensuring that task-relevant features receive higher importance. Modality-specific entity representations are generated through weighted fusion. This mechanism adaptively captures intramodal diversity, avoiding information loss caused by simple averaging or splicing, thereby improving the richness and accuracy of entity representations.

[0118] This application employs a two-stage optimization strategy to address the issues of intermodal redundancy and goal inconsistency. By decoupling the loss constraint to ensure the independence of different modal representations, each modality is forced to focus on extracting its unique features. By aligning the prediction distributions of different modalities through contrastive losses, the consistency of their goals during training is ensured. This design preserves modal specificity while enhancing cross-modal complementarity, avoiding the performance degradation caused by goal conflict in traditional methods.

[0119] This application designs a relation-aware gated attention mechanism that dynamically integrates representations from various modalities to generate joint embeddings. Specifically, it uses learnable weights and projection functions to nonlinearly weight the contributions of different modalities and incorporates relational context to regulate the fusion process. During link prediction, a higher-order interaction scoring function is used to capture the complex relationships between entities and relations. This design significantly improves the model's performance on complex relational reasoning tasks, outperforming traditional bilinear or translational models.

[0120] Compared with related technologies, this application has at least the following advantages:

[0121] (1) This application dynamically aggregates features of multiple instances through a gating mechanism and adaptively adjusts instance weights in conjunction with relational context, which significantly improves the ability to model intramodal diversity.

[0122] (2) This application introduces decoupling loss and symmetric contrast loss, which not only ensures the independence of modal features, but also aligns the objectives between modalities, thereby improving cross-modal complementarity.

[0123] (3) This application designs a relationship-aware gating attention mechanism that dynamically adjusts the multimodal fusion weights, thereby enhancing the ability to reason about complex relationships.

[0124] The various methods described above are based on embodiments of this application. A system for implementing the above methods will be further provided below.

[0125] Please refer to Figure 3 This application also provides a multimodal knowledge graph completion system, including:

[0126] The first processing module 31 is used to perform feature preprocessing on multiple instances of the target entity in multiple modalities to generate modal entity representations;

[0127] The first determining module 32 is used to perform decoupling optimization using a pre-constructed first loss function for modal decoupling and alignment optimization using a pre-constructed second loss function for modal alignment based on the modal entity representation, and to determine each modal representation after decoupling optimization and alignment optimization.

[0128] The second determining module 33 is used to determine a third loss function for linking the prediction loss based on the optimized modal representations and the gated attention mechanism; the link is used to represent the association relationship between single-modal entities and / or the association relationship between cross-modal entities;

[0129] The second processing module 34 is used to construct a total loss function based on the first loss function, the second loss function, the third loss function, and preset hyperparameters;

[0130] The third determining module 35 is used to determine the Magic Box recommendation model based on multimodal knowledge graph completion according to the preset framework of multimodal image fusion and cross-modal alignment and the total loss function.

[0131] Optionally, the first processing module 31 described above includes:

[0132] The first processing unit is used to extract the feature vector of each instance among the plurality of instances using a pre-trained model;

[0133] The second processing unit is used to calculate the gate vector corresponding to the feature vector based on the trainable weight matrix, the structural embedding of the target entity, the relational embedding between the target entities, and the bias term obtained by training and tuning using a preset graph neural network, through a nonlinear activation function.

[0134] The third processing unit is used to perform element-wise multiplication of the gated vector and the feature vector, calculate the weight of each operation result through cosine similarity, and sum the feature vectors of all instances according to the weights to generate a modal entity representation.

[0135] Optionally, the first determining module 32 described above is specifically used for:

[0136] Based on the modal entity representation, determine the modal entity representation of the target entity under different modes;

[0137] Based on the modal entity representations under different modalities, determine the joint probability distribution corresponding to the modal entity representations under different modalities, and the marginal distribution corresponding to each modal entity representation;

[0138] Obtain a pre-constructed first loss function for modal decoupling; the first loss function is determined based on the joint probability distribution, the marginal distribution, and the reciprocal of the total number of samples corresponding to the target entity;

[0139] By minimizing the loss value of the first loss function, the decoupled optimized modal representations are determined.

[0140] Optionally, the first determining module 32 described above is also specifically used for:

[0141] Based on the modal entity representation, determine the modal entity representation of the target entity under different modes after decoupling;

[0142] Obtain a pre-constructed second loss function for modal alignment; the second loss function is determined based on the modal entity representations under different modes after decoupling, a preset consistency loss function, and a preset learnable temperature parameter;

[0143] The modal representations after modal alignment optimization are determined by minimizing the loss value of the second loss function.

[0144] Optionally, the second determining module 33 described above includes:

[0145] The eighth processing unit is used to process each modal representation according to the optimized modal representations through a preset modal projection function to obtain the target embedding of each modality;

[0146] The ninth processing unit is used to perform weight calculation on the target embedding according to the gating attention mechanism and in combination with learnable weights to obtain the attention weight coefficients of the corresponding modality.

[0147] The acquisition unit is used to perform a weighted summation of the target embeddings of all modalities based on the attention weight coefficients to obtain a joint representation;

[0148] A determining unit is configured to determine a third loss function for linking the prediction loss based on the joint characterization.

[0149] Optionally, the aforementioned determining unit is specifically used for:

[0150] Based on the joint representation and the preset scoring function, a set of triples is determined; each triple in the set of triples is used to represent the link score of the head entity, relation, and tail entity; the preset scoring function captures the high-order interaction between modal embeddings through a preset tensor and integrates each embedding representation in the form of the i-th dimension tensor product;

[0151] Based on the set of triples, a set of positive samples and a set of negative samples are determined, and based on the set of positive samples and the set of negative samples, a third loss function for link prediction loss is determined; the set of positive samples is a set of positive sample triples corresponding to known facts in a preset multimodal knowledge graph, and the set of negative samples is a set of non-known link triples generated by randomly replacing tail entities in the set of positive sample triples.

[0152] It should be noted that the system in this embodiment corresponds to the system described above for multimodal knowledge graph completion. The implementation methods in the above embodiments are all applicable to the embodiments of this system and can achieve the same technical effect. The system provided in this application embodiment can implement all the method steps implemented in the above method embodiments and can achieve the same technical effect. Therefore, the parts that are the same as those in the method embodiments and the beneficial effects will not be described in detail here.

[0153] Another embodiment of this application provides an electronic device, referring to... Figure 4 As shown, it includes a memory 420, a transceiver 410, and a processor 400; the memory 420 is used to store computer programs; the transceiver 410 is used to send and receive data under the control of the processor 400.

[0154] The processor 400 is used to perform feature preprocessing on multiple instances of the target entity in multiple modalities to generate modal entity representations;

[0155] The processor 400 is further configured to perform decoupling optimization using a pre-constructed first loss function for modal decoupling based on the modal entity representation, and to perform alignment optimization using a pre-constructed second loss function for modal alignment, thereby determining each modal representation after decoupling optimization and alignment optimization;

[0156] The processor 400 is further configured to determine a third loss function for linking the prediction loss based on the optimized modal representations and the gated attention mechanism.

[0157] The processor 400 is further configured to construct a total loss function based on the first loss function, the second loss function, the third loss function, and preset hyperparameters;

[0158] The processor 400 is also used to determine a Magic Box recommendation model based on multimodal knowledge graph completion according to the preset framework of multimodal image fusion and cross-modal alignment and the total loss function.

[0159] Optionally, the processor 400 described above is also used for:

[0160] A pre-trained model is used to extract the feature vector of each instance from the multiple instances;

[0161] Based on the trainable weight matrix, the structural embedding of the target entity, the relationship embedding between the target entities, and the bias term obtained by training and tuning using a preset graph neural network, a nonlinear activation function is used to calculate the gate vector corresponding to the feature vector.

[0162] The gating vector and the feature vector are multiplied element-wise, and the weights of each operation result are calculated using cosine similarity. The feature vectors of all instances are then summed according to the weights to generate a modal entity representation.

[0163] Optionally, the processor 400 described above is also used for:

[0164] Based on the modal entity representation, determine the modal entity representation of the target entity under different modes;

[0165] Based on the modal entity representations under different modalities, determine the joint probability distribution corresponding to the modal entity representations under different modalities, and the marginal distribution corresponding to each modal entity representation;

[0166] Obtain a pre-constructed first loss function for modal decoupling; the first loss function is determined based on the joint probability distribution, the marginal distribution, and the reciprocal of the total number of samples corresponding to the target entity;

[0167] By minimizing the loss value of the first loss function, the decoupled optimized modal representations are determined.

[0168] Optionally, the processor 400 described above is also used for:

[0169] Based on the modal entity representation, determine the modal entity representation of the target entity under different modes after decoupling;

[0170] Obtain a pre-constructed second loss function for modal alignment; the second loss function is determined based on the modal entity representations under different modes after decoupling, a preset consistency loss function, and a preset learnable temperature parameter;

[0171] The modal representations after modal alignment optimization are determined by minimizing the loss value of the second loss function.

[0172] Optionally, the processor 400 described above is also used for:

[0173] Based on the optimized modal representations, each modal representation is processed using a preset modal projection function to obtain the target embedding for each modality;

[0174] Based on the gating attention mechanism, the target embedding is weighted by combining learnable weights to obtain the attention weight coefficients of the corresponding modality;

[0175] Based on the attention weight coefficients, the target embeddings of all modalities are weighted and summed to obtain a joint representation;

[0176] Based on the joint characterization, a third loss function is determined for linking the prediction loss.

[0177] Optionally, the processor 400 described above is also used for:

[0178] Based on the joint representation and the preset scoring function, a set of triples is determined; each triple in the set of triples is used to represent the link score of the head entity, relation, and tail entity; the preset scoring function captures the high-order interaction between modal embeddings through a preset tensor and integrates each embedding representation in the form of the i-th dimension tensor product;

[0179] Based on the set of triples, a set of positive samples and a set of negative samples are determined, and based on the set of positive samples and the set of negative samples, a third loss function for link prediction loss is determined; the set of positive samples is a set of positive sample triples corresponding to known facts in a preset multimodal knowledge graph, and the set of negative samples is a set of non-known link triples generated by randomly replacing tail entities in the set of positive sample triples.

[0180] Continue as Figure 4 As shown, the electronic device in this application embodiment includes a transceiver 410, a processor 400, a memory 420, and a program or instructions stored in the memory 420 and executable on the processor 400. When the processor 400 executes the program or instructions, it implements the various processes of the above-described multimodal knowledge graph completion method embodiment and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0181] The transceiver 410 is used to receive and send data under the control of the processor 400.

[0182] Among them, Figure 4In this context, the bus architecture may include any number of interconnected buses and bridges, specifically linking various circuits together, represented by one or more processors (processor 400) and memory (memory 420). The bus architecture may also link together various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides an interface. The transceiver 410 may be multiple elements, including transmitters and receivers, providing a unit for communicating with various other devices over a transmission medium. The processor 400 is responsible for managing the bus architecture and general processing, and the memory 420 may store data used by the processor 400 during operation.

[0183] This application also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the various processes of the above-described multimodal knowledge graph completion method embodiments and achieves the same technical effects. To avoid repetition, it will not be described again here. The computer-readable storage medium may include read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0184] This application also provides a computer program product, including computer instructions. When the computer instructions are executed by a processor, they implement the various processes of the above-described multimodal knowledge graph completion method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0185] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0186] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0187] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A method for completing a multimodal knowledge graph, characterized in that, include: Feature preprocessing is performed on multiple instances of the target entity in multiple modalities to generate modal entity representations; Based on the modal entity representation, decoupling optimization is performed using a pre-constructed first loss function for modal decoupling, and alignment optimization is performed using a pre-constructed second loss function for modal alignment, thereby determining each modal representation after decoupling optimization and alignment optimization; Based on the optimized modal representations and gating attention mechanism, a third loss function is determined for linking prediction loss; the link is used to represent the association between single-modal entities and / or the association between cross-modal entities; Based on the first loss function, the second loss function, the third loss function, and the preset hyperparameters, construct the total loss function; Based on the preset framework of multimodal image fusion and cross-modal alignment and the total loss function, a Magic Box recommendation model based on multimodal knowledge graph completion is determined.

2. The method according to claim 1, characterized in that, Feature preprocessing is performed on multiple instances of the target entity in multiple modalities to generate modal entity representations, including: A pre-trained model is used to extract the feature vector of each instance from the multiple instances; Based on the trainable weight matrix, the structural embedding of the target entity, the relationship embedding between the target entities, and the bias term obtained by training and tuning using a preset graph neural network, a nonlinear activation function is used to calculate the gate vector corresponding to the feature vector. The gating vector and the feature vector are multiplied element-wise, and the weights of each operation result are calculated using cosine similarity. The feature vectors of all instances are then summed according to the weights to generate a modal entity representation.

3. The method according to claim 1, characterized in that, Based on the modal entity representation, decoupling optimization is performed using a pre-constructed first loss function for modal decoupling to determine the decoupling optimized modal representations, including: Based on the modal entity representation, determine the modal entity representation of the target entity under different modes; Based on the modal entity representations under different modalities, determine the joint probability distribution corresponding to the modal entity representations under different modalities, and the marginal distribution corresponding to each modal entity representation; Obtain a pre-constructed first loss function for modal decoupling; the first loss function is determined based on the joint probability distribution, the marginal distribution, and the reciprocal of the total number of samples corresponding to the target entity; By minimizing the loss value of the first loss function, the decoupled optimized modal representations are determined.

4. The method according to claim 1, characterized in that, Based on the modal entity representation, alignment optimization is performed using a pre-constructed second loss function for modal alignment to determine the decoupled optimized modal representations, including: Based on the modal entity representation, determine the modal entity representation of the target entity under different modes after decoupling; Obtain a pre-constructed second loss function for modal alignment; the second loss function is determined based on the modal entity representations under different modes after decoupling, a preset consistency loss function, and a preset learnable temperature parameter; The modal representations after modal alignment optimization are determined by minimizing the loss value of the second loss function.

5. The method according to claim 1, characterized in that, Based on the optimized modal representations and gated attention mechanism, a third loss function is determined for linking the prediction loss, including: Based on the optimized modal representations, each modal representation is processed using a preset modal projection function to obtain the target embedding for each modality; Based on the gating attention mechanism, the target embedding is weighted by combining learnable weights to obtain the attention weight coefficients of the corresponding modality; Based on the attention weight coefficients, the target embeddings of all modalities are weighted and summed to obtain a joint representation; Based on the joint characterization, a third loss function is determined for linking the prediction loss.

6. The method according to claim 5, characterized in that, Based on the joint characterization, a third loss function is determined for linking the prediction loss, including: Based on the joint representation and the preset scoring function, a set of triples is determined; each triple in the set of triples is used to represent the link score of the head entity, relation, and tail entity; the preset scoring function captures the high-order interaction between modal embeddings through a preset tensor and integrates each embedding representation in the form of the i-th dimension tensor product; Based on the set of triples, a set of positive samples and a set of negative samples are determined, and based on the set of positive samples and the set of negative samples, a third loss function for link prediction loss is determined; the set of positive samples is a set of positive sample triples corresponding to known facts in a preset multimodal knowledge graph, and the set of negative samples is a set of non-known link triples generated by randomly replacing tail entities in the set of positive sample triples.

7. A multimodal knowledge graph completion system, characterized in that, include: The first processing module is used to perform feature preprocessing on multiple instances of the target entity in multiple modalities to generate modal entity representations; The first determining module is used to perform decoupling optimization using a pre-constructed first loss function for modal decoupling and alignment optimization using a pre-constructed second loss function for modal alignment based on the modal entity representation, thereby determining each modal representation after decoupling optimization and alignment optimization. The second determining module is used to determine a third loss function for linking the prediction loss based on the optimized modal representations and the gating attention mechanism; the link is used to represent the association relationship between single-modal entities and / or the association relationship between cross-modal entities; The second processing module is used to construct a total loss function based on the first loss function, the second loss function, the third loss function, and preset hyperparameters; The third determining module is used to determine the Magic Box recommendation model based on multimodal knowledge graph completion according to the preset framework of multimodal image fusion and cross-modal alignment and the total loss function.

8. An electronic device, characterized in that, It includes a memory, a transceiver, and a processor; the memory is used to store computer programs; the transceiver is used to send and receive data under the control of the processor. The processor is used to perform feature preprocessing on multiple instances of the target entity in multiple modalities to generate modal entity representations; The processor is further configured to perform decoupling optimization using a pre-constructed first loss function for modal decoupling based on the modal entity representation, and to perform alignment optimization using a pre-constructed second loss function for modal alignment, thereby determining each modal representation after decoupling optimization and alignment optimization; The processor is further configured to determine a third loss function for linking prediction loss based on the optimized modal representations and the gated attention mechanism; the link is used to represent the association relationship between single-modal entities and / or the association relationship between cross-modal entities; The processor is further configured to construct a total loss function based on the first loss function, the second loss function, the third loss function, and preset hyperparameters; The processor is also used to determine a Magic Box recommendation model based on multimodal knowledge graph completion, according to a preset framework of multimodal image fusion and cross-modal alignment and the total loss function.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, Includes computer instructions that, when executed by a processor, implement the steps of the method as described in any one of claims 1 to 6.

Images