Decoupling expert multi-modal knowledge graph completion method based on community perception

Abstract

The invention discloses a decoupling expert multi-mode knowledge graph completion method based on community perception, and belongs to the technical field of artificial intelligence and knowledge graphs. According to the method, specific features of each modal are processed through a modal specific expert network, cross-modal shared features are extracted through a modal shared feature expert network, the specific features and the shared features of the modals are fused through a specific-shared feature fusion mechanism, and finally, missing triples are predicted, and a knowledge graph is complemented. According to the method, the specific network and the encoder are designed, auxiliary tasks are combined, modal features are accurately separated, redundant interference is reduced, cooperative work is carried out through the specific network, image information is fully mined, modal feature interaction is deeply learned, implicit relation between power equipment entities can be more accurately captured, and complementation accuracy is improved. Various modal data are fully fused, processing is performed according to modal characteristics, knowledge consistency is ensured, non-text data values can be mined in enterprise-level knowledge graph completion, and completion reliability is ensured.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and knowledge graph technology, and specifically relates to a decoupled expert multimodal knowledge graph completion method based on community perception, which is applied to the fields of multimodal understanding, knowledge graph completion, etc. Background Art

[0002] In the field of multimodal knowledge graph completion, community perception is an effective means to explore the intrinsic connections between entities. It can discover connections that are difficult to detect based on traditional relationships, and has demonstrated important value in knowledge graph scenarios such as social networks.

[0003] Current multimodal knowledge graph completion (MMKGC) methods primarily rely on multimodal feature fusion, but these methods often fail to effectively decouple modality-specific and shared features, leading to embedding redundancy and performance degradation. Furthermore, existing methods often ignore the implicit high-order semantic correlations between entities, which limits the effectiveness of knowledge graph completion.

[0004] Patent CN 119577061 A constructs a multimodal power knowledge base, involving steps such as data classification, entity extraction, AI model alignment, graph database construction, and joint embedding. However, this method lacks modal feature decoupling, making it difficult to capture implicit semantic associations when processing power equipment knowledge, resulting in significant deviations in completion results. Patent CN 119622040 A focuses on acquiring enterprise data to build a knowledge base, including data processing, knowledge extraction, graph construction, retrieval and recommendation, and update and maintenance. However, this method focuses on text data, underutilizes non-text data, and lacks a mechanism for detecting knowledge semantic consistency. Patent CN 119250076 A completes the graph by converting images into text and learning representations using knowledge graph embedding and a feedforward neural network model. However, image-to-text conversion is prone to information loss and lacks interactive feature learning. Patent CN 119783799 A, by contrast, primarily optimizes the Transformer architecture and incorporates dynamic prompt templates and multi-granular cross-modal aggregation. However, prompt template generation and adjustment are inefficient when processing large-scale graphs, and the flexibility of multimodal data fusion is limited. Summary of the Invention

[0005] This paper proposes a community-aware decoupled expert multimodal knowledge graph completion method (INCADE), which aims to solve the following problems:

[0006] Modal feature decoupling problem: Existing methods fail to effectively separate modal-specific features and shared features, resulting in embedding redundancy and interference.

[0007] The problem of ignoring implicit semantic relevance: Existing methods fail to capture the implicit high-order semantic relevance between entities, resulting in poor knowledge graph completion results.

[0008] The purpose of this invention is to improve the accuracy and robustness of multimodal knowledge graph completion by introducing community-aware entity encoders and auxiliary tasks (such as modality memory and modality forgetting) to achieve effective decoupling of modality-specific features and shared features, and to capture implicit high-order semantic correlations between entities.

[0009] To solve the above technical problems, the specific technical solution of the community-aware decoupled expert multimodal knowledge graph completion method of the present invention is as follows:

[0010] Step 1: Obtain multimodal data input;

[0011] Step 2: Process the specific features of each modality through a modality-specific expert network; dynamically learn the mapping relationship between entities and entity communities in each modality through an entity community mapper, and obtain the specific features of each modality entity based on the fusion of community information and entities;

[0012] Step 3: Extract cross-modal shared features through the modality shared feature expert network; project the multimodal features into a unified representation space, then model the shared information of all modalities, and aggregate them to obtain cross-modal shared features;

[0013] Step 4: Fusion of modality-specific features and shared features through the specific-shared feature fusion mechanism;

[0014] Step 5: Predict missing triplets based on the fused features and complete the knowledge graph.

[0015] The modality-specific expert network is specifically as follows:

[0016] Extract the original modal features from the multimodal data and construct a knowledge graph. Under a specific single modality, calculate the representation of each entity community through the entity community mapper. The formula is as follows:

[0017]

[0018] in Representing an entity Assigned to the community The probability of , i is the entity index, traversing all entities in the knowledge graph from 1 to the total number of entities n, j is the community index, and the total number of communities is k;

[0019] Two communities and Implicit correlation between Calculated by the following formula:

[0020]

[0021] where φ is the transformation function implemented by the multilayer perceptron, is the activation function;

[0022] The intensities are then quantized using a convolution operation:

[0023]

[0024] in, For the community and The quantified strength of the implicit association between Conv( ) is a one-dimensional convolutional layer, σ is a sigmoid function;

[0025] Community Aggregate information from other communities and then use the residual operation to update the representation of the entity community as follows:

[0026]

[0027]

[0028] in, Represents a community after aggregating information from other communities, represents the updated entity community, ψ is the residual operation implemented by the gated recurrent unit;

[0029] Finally, entity e i The representation of is updated by integrating information from the corresponding community:

[0030]

[0031] in For entity e i specific characteristics.

[0032] The modality shared feature expert network is specifically as follows:

[0033] First, each modality is projected into a unified representation space:

[0034]

[0035] in, ( ) is the shared feature projection layer, is the original feature of the i-th entity in the m-th mode, is the shared spatial feature of the i-th entity in the m-th mode after projection, where m is the mode index and M is the total number of modes;

[0036] Then aggregate the shared entity features of all modalities to obtain the comprehensive modal shared features :

[0037]

[0038]

[0039] in, represents the shared feature aggregation weight of the mth modality, ( )and are the transformation layers and learnable weights, Is a shared space feature A simplified representation of is the element-wise product operator; exp( ) is the exponential function, and tanh(·) is the hyperbolic tangent function.

[0040] The specific-shared feature fusion mechanism is as follows:

[0041]

[0042]

[0043] in, It is a multimodal entity representation; Indicates the The fusion weight of the views, Q represents the set of modality-specific views and modality-shared views; Modality-specific features obtained for modality-specific expert networks or modality-shared feature expert networks or modal shared features ,in is the specific feature under the mth mode; ( ) is the transformation layer, are learnable weights.

[0044] The step 5 is specifically as follows:

[0045] After predicting the missing triples based on the multimodal entity representation, the rationality of the triples is measured using a scoring function:

[0046]

[0047] Among them, the triple (h, r, t) indicates that there is a relationship r between the head entity h and the tail entity t, S(h, r, t) is the triple scoring function, x i，i=1,2,3 represents the tensor product along the i-th mode, where the mode corresponds to the direction of the tensor dimension, and W is the core tensor learned during training;

[0048] Then the training optimization is performed through the joint loss function; the joint loss function is as follows:

[0049]

[0050] Among them, v is the view index, represents the multimodal view, λ1 and λ2 represent the adjustment weights; represents the entity prediction loss function under the vth view, which includes the head entity loss and the tail entity loss;

[0051] The loss function for head entity prediction is:

[0052]

[0053] Where (h, r, t)∈T represents the traversal of the triple set T in the training knowledge graph, taking the real head entity h, relation r, and tail entity t, ∑ h′∈E exp(S(h′,r,t)) represents the score of the triple (h′,r,t) after replacing the head entity with h′ for all entities.

[0054] The loss function for tail entity prediction is:

[0055]

[0056] Traverse the training triples (h, r, t) and calculate the score probability of all entities E after replacing the tail entity with t′;

[0057] represents the modal memory loss, which is calculated as follows:

[0058]

[0059] Where m is the modality index, traversing various modes from 1 to |M|, and |M| is the total number of modes; is the unique hot encoding label corresponding to m modes, with a dimension of {0,1}|M|, where only the mth bit is 1, indicating the true mode to which the feature belongs; log( ) is a logarithmic function; is the modality-specific classification layer, implemented by a multilayer perceptron; It is the entity feature representation of the i-th entity under m modes, which is used to classify and judge the mode to which it belongs;

[0060] is the modality forgetting loss, calculated as follows:

[0061]

[0062] Among them D KL ( ) is the KL divergence loss, v (m) = represents a uniform distribution over all modes; ( ) is the modality-shared classification layer.

[0063] The beneficial effects of the present invention are as follows:

[0064] By designing a specific network and encoder, combined with auxiliary tasks, the present invention can accurately separate modal features and reduce redundant interference, thereby more accurately capturing the implicit connections between power equipment entities and improving completion accuracy.

[0065] The present invention can fully integrate multiple modal data, process them according to the modal characteristics, ensure knowledge consistency, and mine the value of non-text data in enterprise-level knowledge graph completion to ensure completion reliability.

[0066] The present invention fully mines image information through the collaborative work of specific networks, deeply studies the interaction of modal features, has strong generalization capabilities on different data sets and tasks, and makes completion more efficient and accurate.

[0067] With the help of the entity community perception encoder, the present invention can deeply explore the potential connections between entities, efficiently complete them in complex large-scale graph scenarios, and does not rely on complex prompt templates, making it more adaptable. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 Schematic diagram of the process of the present invention;

[0069] Figure 2 Schematic diagram of the modality-specific expert network structure;

[0070] Figure 3 Schematic diagram of the modal regularization process. DETAILED DESCRIPTION

[0071] In order to better understand the purpose, structure and function of the present invention, the following is a further detailed description of the community-aware decoupled expert multimodal knowledge graph completion method of the present invention in conjunction with the accompanying drawings.

[0072] This paper proposes a community-aware decoupled expert multimodal knowledge graph completion method (INCADE), whose overall architecture is as follows: Figure 1 As shown in the figure. This method achieves effective decoupling of modality-specific features and shared features through a modality-specific expert network, a modality-shared feature expert network, and a specific-shared feature fusion mechanism. It also captures the implicit high-order semantic correlation between entities and significantly improves the accuracy of multimodal knowledge graph completion. The specific steps of this method are as follows:

[0073] Step 1: Obtain multimodal data input, including structural, visual, and textual modalities;

[0074] Step 2: Process the specific features of each modality through a modality-specific expert network;

[0075] Step 3: Extract cross-modal shared features through the modality shared feature expert network;

[0076] Step 4: Fusion of modality-specific features and shared features through the specific-shared feature fusion mechanism;

[0077] Step 5: Predict missing triplets and complete the knowledge graph.

[0078] Furthermore, the multimodal data in step 1 specifically includes:

[0079] Structural modality: data that can represent structural relationships, which can be in the form of tree structures, knowledge graphs, tables, etc.; visual modality: data represented by images, graphics, etc.; textual modality: semantic data represented by natural language or text sequences.

[0080] Furthermore, the modality-specific expert network in step 2 is as follows Figure 2 As shown, the specific structure is as follows:

[0081] To effectively learn modality-specific entity embeddings, we design an innovative entity community-aware encoder that aims to capture the implicit long-range dependencies between entities within each modality. In a multimodal knowledge graph, different modalities contain unique information, such as image details in the visual modality and semantic information in the textual modality. The entity relationships within this information often exhibit long-range dependencies, making them difficult to capture effectively using traditional methods. This encoder addresses this issue.

[0082] For the three main modalities—structural, visual, and textual—raw modal features are extracted from the original modal information. For the structural modality, the R-GCN (Relational Graph Convolutional Network) is pre-trained on the entire knowledge graph to initialize the raw structural modal features. During subsequent training, these feature representations are continuously optimized by learning directly from triplets within the batch, making them more accurately reflect the information in the structural modality.

[0083] In order to capture the implicit semantic correlation of visual and textual modalities, this paper designs an entity community mapper to dynamically learn the mapping relationship between entities and entity communities. Refers to a specific object in the knowledge graph (such as a football-related organization, a country or region famous for football, etc.). i is used as an entity index to traverse all entities from 1 to n (n is the total number of entities) in the knowledge graph. A community is a virtual grouping based on semantic relevance clustering (such as the "football club" community). The entity community mapper uses the mapping matrix C∈R n×k represents the relationship between entities and communities, where n and k represent the number of entities and communities, respectively. Unlike traditional static assignment methods, the entity-community mapper allows entities to belong to multiple communities with different probabilities. This dynamic mapping relationship can more flexibly and accurately express the complex connections between entities and communities.

[0084] Under a specific single modality, the entity community mapper calculates the representation of each entity community by the following formula:

[0085]

[0086] in Representing an entity Assigned to the community The probability of .

[0087] In order to capture the implicit high-order correlations between entity communities, this embodiment constructs a fully connected graph in which all entity communities are connected to each other; and Implicit correlation between Calculated by the following formula:

[0088]

[0089] where φ is a transformation function implemented by a multi-layer perceptron (MLP), for Activation function.

[0090] To distinguish different implicit correlations, a convolution operation is used to quantify their strength:

[0091]

[0092] in, For the community and The quantified strength of the implicit association between Conv( ) is a one-dimensional convolutional layer, and σ is a sigmoid function, which limits the strength of implicit correlation to between 0 and 1, thereby quantifying the strength of implicit correlation and facilitating subsequent calculation and analysis.

[0093] For the community , which aggregates information from other communities: Represent the community after aggregating information from other communities, and then use the residual operation to update the representation of the entity community:

[0094] in, represents the updated entity community, and ψ is a residual operation implemented by a gated recurrent unit (GRU). GRU can effectively process sequential data. It retains the original information through residual connections while integrating information from other communities, making the community representation richer and more accurate.

[0095] Finally, entity e i The representation of is updated by integrating information from the corresponding community:

[0096]

[0097] in For entity e i In order to encourage the learning of better modality-specific entity representations, a modality memory auxiliary task is introduced to ensure that valuable modality-specific information is retained. The modality memory loss is defined as follows:

[0098]

[0099] is the modality memory loss, which is used to constrain the model to learn modality-specific entity representations and encourage the model to retain unique information of each modality; m is the modality number index, which traverses various modalities from 1 to |M| (|M| is the total number of modalities, for example, when structural, visual, and textual modalities coexist, the number is 3); is the unique hot encoding label corresponding to m modes, with a dimension of {0,1}|M|, where only the mth bit is 1, indicating the true mode to which the feature belongs; is its transpose, which is used to extract the probability value corresponding to the true mode in the classification probability; log( ) is a logarithmic function that converts the classification probability into cross entropy loss; It is a modality-specific classification layer, implemented by a multi-layer perceptron. It inputs the features of an entity under a certain modality and outputs the probability distribution of the feature belonging to each modality. It is the entity feature representation of the i-th entity under m modes, which is used to classify and determine the mode to which it belongs.

[0100] Furthermore, the modality-sharing feature expert network in step 3 is specifically as follows:

[0101] To capture the entity representation shared by modalities, we design a shared feature expert network that first projects multiple modal features into a unified representation space and then models the shared information of all modalities. The projection process for each modal feature is as follows:

[0102]

[0103] in ( ) is a shared feature projection layer, shared by all modalities, and is also an entity community-aware encoder, shared by all modalities. is the original feature of the i-th entity in the m-th mode. The projection layer represents the shared spatial features of the i-th entity in the m-th modality. This aims to converge features from different modalities into a unified space, facilitating subsequent mining of cross-modal shared information. The projection layer traverses the various modalities (such as structure, vision, and text) in the modal set M to complete the shared spatial mapping of all modal features, supporting the subsequent shared information modeling process. This projection layer maps features from different modalities into a common space, enabling the integration and representation of shared information from these modalities within this unified space.

[0104] To guide the learning of modality-shared entity representations, this paper introduces a modality forgetting task to constrain the shared information flow between different modalities. The modality forgetting loss is defined as follows:

[0105]

[0106] Among them D KL ( ) is the Kullback-Leibler (KL) divergence loss, which is used to measure the difference between two probability distributions, v (m) = represents a uniform distribution over all modes; ( ) is the modality sharing classification layer, which confuses the classification results based on shared features, enabling the model to better extract the real shared information and avoid the model from being overly dependent on the specific information of a certain modality.

[0107] By aggregating the shared entity features of all modalities, a comprehensive modality shared entity representation is obtained :

[0108]

[0109]

[0110] in Represents the shared feature aggregation weight of the mth modality, which is used to measure the impact of the shared features of this modality on the final comprehensive representation The contribution of ( )and are the transformation layers and learnable weights, Indicates the shared spatial features of the entity after being mapped by the shared feature projection layer under the mth mode, which is Simplified representation of ; is the element-wise product operator. exp( ) is the exponential function, that is, exp(x)=e x (e is a natural constant) It is used to amplify feature differences, allowing the weight calculation of different modes to better "highlight the key and suppress the secondary." tanh(·) Hyperbolic tangent function.

[0111] Furthermore, the specific-shared feature fusion mechanism in step 4 is specifically as follows:

[0112] Through the modality-specific expert network and the modality-shared expert network, the specific features under the mth modality can be obtained (representing the overall specific semantic abstraction of the mth modality) and shared features , respectively representing the heterogeneous representation within each modality and the consistent representation across modalities. In order to fully integrate this information, the specific-shared feature fusion process is as follows:

[0113]

[0114]

[0115] in Indicates the The fusion weight of the view (modality-specific / shared view) is used to measure the contribution of the view feature to the final multimodal entity representation. contribution. ( ) is the transformation layer, are learnable weights, The modality-specific features obtained from the previous modality-specific expert network and modality-shared feature expert network or modal shared features . Represents a collection of modality-specific and modality-shared views, where V represents a modality-specific view that focuses on the unique information of a single modality. This fusion mechanism can dynamically fuse specific features and shared features according to the importance of different modalities, making the final multimodal entity representation It can better capture the complex semantic interactions between different modalities and make full use of the advantageous information of each modality.

[0116] The dual-expert network strengthens modality specificity and cross-modal generality through classification tasks and distribution constraints. The specific expert network incorporates grouping characteristics and guides input embedding processing with the help of group labels (such as modality type labels), and focuses on learning modality-specific features through spectral loss and specific loss constraints:

[0117]

[0118] Where L is the graph Laplace matrix, C is the entity-community mapping matrix, and tr( ) represents the sum of the main diagonal elements of the matrix in brackets; the spectral loss uses graph Laplace regularization to ensure that the modality-specific features are consistent with the semantics of the knowledge graph structure; the specific loss Through the modality classification task Precisely encode modality specificity. The shared expert network defines the learning direction based on specific shared labels (such as cross-modal common semantic labels) and extracts cross-modal shared features through the KL divergence loss constraint. The KL divergence loss forces the distribution of shared features to approach a uniform distribution, allowing them to "forget" modality specificity and retain only cross-modal commonalities. This loss also uniformly encompasses the semantics of the original "sharing loss."

[0119] like Figure 3 The collaborative process of the dual expert network shown in the figure corresponds to the canonical diagram. In the left branch, multimodal embedding is performed by a specific expert network, combining spectral loss and specific loss to output modality-specific features. In the right branch, multimodal embedding is performed through a shared expert network, combined with KL divergence loss to output cross-modal shared features. The two are guided by "specific-shared labels" in both directions and obtained by dynamic weighted fusion. , which serves as input to step 5, uses the Tucker scoring function to calculate the rationality score of the triple combination, thereby predicting the missing head or tail entity. This mechanism allows the model to simultaneously grasp "modality-specific details" and "cross-modality commonalities," resolving multimodal semantic conflicts and information redundancy issues. This provides a more robust entity representation for knowledge graph completion (such as missing triple prediction), ultimately improving model performance and effectiveness.

[0120] Furthermore, the step 5 is specifically as follows:

[0121] In order to complete the multimodal knowledge graph, it is necessary to predict the missing triples. This paper adopts a specific scoring function and loss function to achieve this goal. Tucker is used as the scoring function to measure the rationality of the triples:

[0122]

[0123] The triple (h, r, t) indicates that there is a relationship r between the head entity h and the tail entity t. S(h, r, t) is the triple score function used to measure the rationality of the semantic association h−r−t. i represents the tensor product along the i-th mode, where the mode corresponds to the tensor dimension. This operation allows for deep association of tensor elements with entity / relationship embeddings within the dimension, providing a mathematical foundation for characterizing the semantic interactions between the head entity h, relation r, and tail entity t. This allows the score S(h, r, t) to accurately measure the semantic rationality of the triple. W is a core tensor learned during training and a parameter learned during model training, used to capture the complex interactions between entities, relations, and entities.

[0124] For entity prediction, a training target loss is defined. The loss function for head entity prediction is:

[0125]

[0126] Where (h, r, t)∈T represents the traversal of the triple set T in the training knowledge graph, taking the real head entity h, relation r, and tail entity t, ∑ h′∈E exp(S(h′,r,t)) represents the calculation of the score of the triple (h′,r,t) after replacing the head entity with h′ for all entities E (the entity set of the knowledge graph), and is converted into a probability through exp(·). By minimizing The scoring probability of the real head entity h will be made as large as possible, while the scoring probability of other entities h′ will be made as small as possible, so as to learn the embedding vector that can distinguish between real and wrong head entities.

[0127] The loss function for tail entity prediction is:

[0128]

[0129] Traverse the training triples (h, r, t), calculate the score probability of "after replacing the tail entity with t'" for all entities E, maximize the score probability of the true tail entity t, and learn the embedding vector that distinguishes true / false tail entities.

[0130] In order to adapt to the visual learning needs of multimodal knowledge graphs (different modal views such as text views and image views each encode the unique semantics of entities), the entity prediction loss function under a single modality is defined. :

[0131]

[0132] Combine all modal views and two auxiliary tasks (modal memory loss and modality forgetting loss ), the overall training goal of INCADE is:

[0133]

[0134] Where V mm represents multimodal views, while λ1 and λ2 regulate the weights of the two auxiliary tasks. By jointly optimizing the knowledge graph completion task and the auxiliary tasks, INCADE effectively decouples modality-specific and modality-shared entity representations, enabling the model to capture multi-view semantic interactions and thus make more accurate predictions. During training, by continuously adjusting model parameters, the overall training objective is gradually reduced, thereby optimizing model performance and improving the accuracy of multimodal knowledge graph completion.

[0135] It will be understood that the present invention is described by way of some embodiments, and it will be appreciated by those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are intended to be protected by the present invention.

Claims

1. A community-aware decoupled expert multimodal knowledge graph completion method, characterized by: The following steps are involved: Step 1: Obtain multimodal data input; Step 2: Process the specific features of each modality through a modality-specific expert network; The mapping relationship between entities and entity communities in each modality is dynamically learned through the entity community mapper, and the specific features of each modality entity are obtained based on the fusion of community information and entities; Step 3: Extract cross-modal shared features through the modality shared feature expert network; project the multimodal features into a unified representation space, then model the shared information of all modalities, and aggregate them to obtain cross-modal shared features; Step 4: Fusion of modality-specific features and shared features through the specific-shared feature fusion mechanism; Step 5: Predict missing triplets based on the fused features and complete the knowledge graph.

2. A community-aware decoupled expert multimodal knowledge graph completion method according to claim 1, characterized in that: The modality-specific expert network is specifically as follows: Extract the original modal features from the multimodal data and construct a knowledge graph. Under a specific single modality, calculate the representation of each entity community through the entity community mapper. The formula is as follows: ； in Representing an entity Assigned to the community The probability of , i is the entity index, traversing all entities in the knowledge graph from 1 to the total number of entities n, j is the community index, and the total number of communities is k; Two communities and Implicit correlation between Calculated by the following formula: ； where φ is the transformation function implemented by the multilayer perceptron, is the activation function; The intensities are then quantized using a convolution operation: ； in, For the community and The quantified strength of the implicit association between Conv( ) is a one-dimensional convolutional layer, σ is a sigmoid function; Community Aggregate information from other communities and then use the residual operation to update the representation of the entity community as follows: ； ； in, Represents a community after aggregating information from other communities, represents the updated entity community, ψ is the residual operation implemented by the gated recurrent unit; Finally, entity e i The representation of is updated by integrating information from the corresponding community: ； in For entity e i specific characteristics.

3. The method for completing multimodal knowledge graphs based on community awareness and decoupling of experts according to claim 2 is characterized in that: The modality shared feature expert network is specifically as follows: First, each modality is projected into a unified representation space: ； in, ( ) is the shared feature projection layer, is the original feature of the i-th entity in the m-th mode, is the shared spatial feature of the i-th entity in the m-th mode after projection, where m is the mode index and M is the total number of modes; Then aggregate the shared entity features of all modalities to obtain the comprehensive modal shared features : ； ； in, represents the shared feature aggregation weight of the mth modality, ( )and are the transformation layers and learnable weights, Is a shared space feature A simplified representation of is the element-wise product operator; exp( ) is the exponential function, and tanh(·) is the hyperbolic tangent function.

4. The method for completing multimodal knowledge graphs based on community awareness and decoupling of experts according to claim 3 is characterized in that: The specific-shared feature fusion mechanism is as follows: ； ； in, It is a multimodal entity representation; Indicates the The fusion weight of the views, Q represents the set of modality-specific views and modality-shared views; Modality-specific features obtained for modality-specific expert networks or modality-shared feature expert networks or modal shared features ,in is the specific feature under the mth mode; ( ) is the transformation layer, are learnable weights.

5. The method for completing multimodal knowledge graphs based on community awareness and decoupling of experts according to claim 4 is characterized in that: The step 5 is specifically as follows: After predicting the missing triples based on the multimodal entity representation, the rationality of the triples is measured using a scoring function: ； Among them, the triple (h, r, t) indicates that there is a relationship r between the head entity h and the tail entity t, S(h, r, t) is the triple scoring function, x i，i=1,2,3 represents the tensor product along the i-th mode, where the mode corresponds to the direction of the tensor dimension, and W is the core tensor learned during training; Then the training optimization is performed through the joint loss function.

6. The method for completing multimodal knowledge graphs based on community awareness and decoupling of experts according to claim 5 is characterized in that: The joint loss function is as follows: ； Among them, v is the view index, represents the multimodal view, λ1 and λ2 represent the adjustment weights; represents the entity prediction loss function under the vth view, which includes the head entity loss and the tail entity loss; The loss function for head entity prediction is: ； Where (h, r, t)∈T represents the traversal of the triple set T in the training knowledge graph, taking the real head entity h, relation r, and tail entity t, ∑ h′∈E exp(S(h′,r,t)) represents the score of the triple (h′,r,t) after replacing the head entity with h′ for all entities. The loss function for tail entity prediction is: ； Traverse the training triples (h, r, t) and calculate the score probability of all entities E after replacing the tail entity with t′; represents the modal memory loss, which is calculated as follows: ； Where m is the modality index, traversing various modes from 1 to |M|, and |M| is the total number of modes; is the unique hot encoding label corresponding to m modes, with a dimension of {0,1}|M|, where only the mth bit is 1, indicating the true mode to which the feature belongs; log( ) is a logarithmic function; is the modality-specific classification layer, implemented by a multilayer perceptron; It is the entity feature representation of the i-th entity under m modes, which is used to classify and judge the mode to which it belongs; is the modality forgetting loss, calculated as follows: ； Among them D KL ( ) is the KL divergence loss, v (m) = represents a uniform distribution over all modes; ( ) is the modality-shared classification layer.

Images