Multi-modal entity alignment method and system based on alignment sub-graph and electronic equipment

Abstract

The invention relates to a multi-modal entity alignment method and system based on an alignment sub-graph and electronic equipment, and belongs to the technical field of data processing, and the method comprises the steps: based on entities and entity relation information in two knowledge graphs, generating the alignment sub-graph of the entities by defining an alignment rule and extracting an alignment path; dynamically combining the aligned sub-graphs, and for each node in the combined aligned sub-graphs, updating an interpretable path-based message transmission process by aggregating information from neighbor nodes so as to calculate path scores of the aligned sub-graphs; characterizing the multi-modal attributes of the entities for the aligned sub-graphs through an attribute encoder, characterizing the perceived multi-modal attributes of the entities through an aligned attention mechanism, and calculating attribute alignment scores among different entities in the multi-modal knowledge graph; and finally, multi-modal entity pairs are aligned based on entity similarity evaluation comprehensive scores. According to the method, the multi-modal attribute information of the entity can be effectively utilized to improve the alignment accuracy.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and more specifically to a multimodal entity alignment method, system, and electronic device based on aligned subgraphs. Background Technology

[0002] Among existing knowledge graph alignment techniques, embedding-based entity alignment methods are favored due to their wide application in knowledge integration and cross-linguistic information retrieval. However, these methods have significant limitations, especially when dealing with structurally similar entity pairs with low alignment relevance. They rely too heavily on the embedding distance between entities, neglecting the underlying logical rules, resulting in poor interpretability of the alignment results and susceptibility to noise interference.

[0003] Traditionally, to address these shortcomings, researchers have attempted to incorporate entity attribute information or employ iterative strategies to improve entity alignment performance. For example, they might utilize entity attribute features (such as name and description) to construct more nuanced entity representations; or they might iterate to gradually refine the entity alignment results, hoping to converge to a more accurate alignment state. While these methods have improved entity alignment to some extent, they still face the challenge of effectively utilizing cross-graph logic rules, especially when balancing alignment accuracy and interpretability is required.

[0004] Entity alignment is a complex process involving a deep understanding and precise matching of potential relationships between entities. Current technologies are often limited by focusing only on surface structural similarity and attribute overlap, neglecting the deeper logical rules that may govern entity alignment. This is particularly evident when dealing with large-scale, multimodal knowledge graphs. For example, in cross-linguistic knowledge graph alignment, relying solely on textual or structural similarity cannot fully capture the true relationships between entities, because the same entity may be described differently in different linguistic contexts, or the neighborhood structure of an entity may vary due to differences in the corpus. This necessitates entity alignment algorithms that can more intelligently understand and apply the logical rules between entities.

[0005] Furthermore, existing entity alignment methods typically assume that anchor links (i.e., known entity alignment information) are sufficiently abundant to directly guide the entity alignment process. However, in real-world applications, anchor links are often scarce. This necessitates that entity alignment algorithms effectively utilize limited anchor link information, as well as the multimodal attribute information of entities, to construct and strengthen the entity alignment path, thereby improving alignment accuracy.

[0006] In summary, existing technologies fail to fully explore and utilize the underlying logical rules of entity alignment when dealing with the problem, resulting in limited accuracy and interpretability of the alignment results. This limitation is particularly pronounced when dealing with structurally similar entity pairs with low alignment relevance, and in scenarios where anchor links are scarce. Therefore, a novel entity alignment method is needed that not only uncovers deep-seated logical rules from entity alignment but also effectively utilizes the multimodal attribute information of entities to improve alignment accuracy while maintaining interpretability, even in situations where anchor links are scarce. Summary of the Invention

[0007] The purpose of this application is to provide a multimodal entity alignment method, system, and electronic device based on aligned subgraphs. This method improves the performance of multimodal knowledge graph alignment, especially when dealing with large datasets. The application of dynamic programming algorithms significantly reduces computation time, while interpretable attention mechanisms enhance the transparency of model decisions.

[0008] To achieve the above objectives, this application provides the following technical solution: This application proposes a multimodal entity alignment method based on aligned subgraphs, including: Based on entity and entity relationship information in two knowledge graphs, alignment subgraphs of entities are generated by defining alignment rules and extracting alignment paths; The alignment subgraph is dynamically merged. For each node in the merged alignment subgraph, the message passing process based on the interpretable path is updated by aggregating information from neighboring nodes, and then the path score of the alignment subgraph is calculated. The alignment subgraph is characterized by an attribute encoder to represent the multimodal attributes of entities and by an alignment attention mechanism to represent the multimodal attributes of entities in response to perception. The attribute alignment score between different entities in the multimodal knowledge graph is calculated. Based on the path score and attribute alignment score of the aligned subgraph, a comprehensive entity similarity evaluation score is obtained, and multimodal entity pairs are aligned based on the comprehensive entity similarity evaluation score.

[0009] Furthermore, the alignment rules include three types: single-hop rules, symmetrical K-hop rules, and asymmetrical K-hop rules; Single-hop rules are used to describe when entities pair Through the same relationship Alignment when connecting to an already aligned entity pair; The symmetric K-jump rule is used to focus on scenarios where entity pairs are connected to aligned entity pairs through the same k-step relation sequence; The asymmetric K-hop rule is used to extend to the case where aligned entity pairs are connected via semantically equivalent but different length paths.

[0010] Furthermore, the extraction alignment path includes: The alignment path is based on anchor point connections, which include existing alignment seeds and connection points established based on the similarity of entity multimodal attributes. The alignment path reverses the relationships in the knowledge graph to perform bidirectional search, identifies and retains paths that conform to the alignment rules, and forms a complete alignment path structure by integrating information.

[0011] Furthermore, the generated entity alignment subgraph includes: Extract all valid paths with a length not exceeding K, merge paths with common nodes or edges, and construct a K-jump aligned subgraph. An alignment subgraph is generated by aggregating all possible alignment paths. .

[0012] Furthermore, the alignment subgraphs are dynamically merged, including: When multiple target entities share the same ancestor node, a dynamic programming method is used to share the node and dynamically merge the aligned subgraphs. The merged aligned subgraphs are then... Includes from arrive All paths, formally represented as .

[0013] Furthermore, in the message passing process based on interpretable paths, a unidirectional path message passing strategy is adopted to retain specific path information, and an interpretable attention mechanism is used to prioritize the edges on important paths.

[0014] Furthermore, the calculation of the path score for the aligned subgraph includes: A knowledge graph feature fusion function is introduced, and sequence information is processed through a recurrent neural network. Finally, a multilayer perceptron is used to calculate the path score of the aligned subgraph.

[0015] Furthermore, the alignment subgraph is characterized by an attribute encoder to represent the multimodal attributes of the entity, including: Fully connected layers employing specific modes of attribute encoders Perform feature transformation on aligned subgraphs to map features from different modalities to a unified space; each modality Through the weight matrix Initial features Projecting onto a unified representation space to characterize multimodal properties, the transformation formula is as follows: ; The different modalities include text modality and image modality; wherein, the initial features of text modality come from the output of BERT model, and the initial features of image modality come from pre-trained visual model.

[0016] Furthermore, the method of characterizing the multimodal attributes of an entity in response to perception through the alignment attention mechanism includes: For entity pairs and The pairwise sensing multimodal representation is obtained through weighted aggregation: Among them, attention weight It is obtained by calculating the similarity between attribute types and applying the softmax function.

[0017] Furthermore, the attribute alignment scoring between different entities in the multimodal knowledge graph includes: After obtaining the multimodal attribute representations of entity pairs, the multimodal attributes of the entity pairs are processed by a multilayer perceptron to classify the entities... and After performing element-wise multiplication on the multimodal representation, the alignment score is calculated using an MLP: .

[0018] This application also proposes a multimodal entity alignment system based on aligned subgraphs, and a multimodal entity alignment method based on aligned subgraphs, comprising: The alignment subgraph extraction module is used to generate alignment subgraphs of entities based on entity and entity relationship information in two knowledge graphs by defining alignment rules and extracting alignment paths. A path-based graph neural network module is used to dynamically merge the alignment subgraph. For each node in the merged alignment subgraph, the message passing process based on the interpretable path is updated by aggregating information from neighboring nodes, and then the path score of the alignment subgraph is calculated. The node-level multimodal attention module is used to characterize the multimodal attributes of entities in the alignment subgraph through an attribute encoder, and to characterize the multimodal attributes of entities to perception through an alignment attention mechanism, and to calculate the attribute alignment score between different entities in the multimodal knowledge graph. The multimodal entity pair alignment module is used to obtain a comprehensive entity similarity assessment score based on the path score and attribute alignment score of the aligned subgraph, and to align multimodal entity pairs based on the comprehensive entity similarity assessment score.

[0019] This application also proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the above-described multimodal entity alignment method based on aligned subgraphs.

[0020] This application also proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described multimodal entity alignment method based on aligned subgraphs.

[0021] This application also proposes a computer program product comprising computer instructions that instruct a computer to execute the above-described multimodal entity alignment method based on aligned subgraphs.

[0022] Compared with the prior art, this application has the following advantages: This application extracts alignment subgraphs containing structural information about the surrounding entities from two knowledge graphs. Paths are constructed using anchor links, and irrelevant neighbor information is filtered out to obtain key support for structured relationships. The extracted alignment subgraphs are processed using an interpretable path-based message passing mechanism. Edges on important paths are ranked using a unidirectional path message passing strategy and an attention mechanism. Dynamic programming is employed to improve message passing efficiency, ensuring that entity representations reflect alignment details. A unified multimodal attention mechanism is implemented to integrate multimodal attribute information such as text and images of entities. Multimodal anchor links are created, and the alignment attention mechanism prioritizes shared attributes between entities, reducing interference from mismatched attributes and improving the accuracy of multimodal entity alignment. This method significantly improves the performance of multimodal knowledge graph alignment, especially when dealing with large datasets. The application of dynamic programming significantly reduces computation time, while the interpretable attention mechanism enhances the transparency of model decisions. The accurate construction of alignment subgraphs and the effective integration of multimodal attributes enable the model to more accurately capture the similarities and differences between entities, thus achieving higher accuracy and stability in entity alignment tasks. By combining optimization strategies and attribute processing mechanisms, this method not only improves computational efficiency but also enhances the interpretability and credibility of alignment results, providing a solid foundation for subsequent multimodal knowledge discovery and reasoning. Attached Figure Description

[0023] Figure 1 A flowchart of a multimodal entity alignment method based on aligned subgraphs is provided in this application; Figure 2 A flowchart illustrating the multimodal entity alignment method based on alignment subgraph rule mining provided in a specific embodiment of this application; Figure 3 This application provides a schematic diagram of a multimodal entity alignment system based on aligned subgraphs. Detailed Implementation

[0024] The technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. The components of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application. It should be noted that similar reference numerals and letters in the following drawings indicate similar items; therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. Furthermore, in the description of this application, the terms "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0025] This application addresses the problems existing in the prior art by providing a multimodal entity alignment method, system, and electronic device based on aligned subgraphs, which not only improves the accuracy of entity alignment but also enhances the interpretability and robustness of the model.

[0026] Firstly, this application provides a multimodal entity alignment method based on aligned subgraphs, including: S1, based on entity and entity relationship information in two knowledge graphs, generates an alignment subgraph of entities by defining alignment rules and extracting alignment paths; S2, dynamically merge the alignment subgraph. For each node in the merged alignment subgraph, update the message passing process based on the interpretable path by aggregating information from neighboring nodes, and then calculate the path score of the alignment subgraph. S3, the alignment subgraph is characterized by the multimodal attributes of entities through an attribute encoder and by the alignment attention mechanism to characterize the multimodal attributes of entities to perception, and the attribute alignment score between different entities in the multimodal knowledge graph is calculated. S4. Based on the path score and attribute alignment score of the aligned subgraph, the comprehensive entity similarity evaluation score is obtained, and the multimodal entity pairs are aligned based on the comprehensive entity similarity evaluation score.

[0027] Based on the above scheme, this application achieves an intelligent upgrade of knowledge graph alignment by constructing a multimodal entity alignment framework based on alignment subgraphs. Specifically, the framework first extracts alignment subgraphs from two knowledge graphs to capture the structured relationship information between entities. Simultaneously, it combines a path-based graph neural network to evaluate the importance of each potential alignment rule. When a key alignment rule path is discovered, a multimodal attention mechanism is automatically activated, utilizing the entity's text, image, and other attribute information to obtain more accurate alignment results. The model design of this application focuses on three key modules: an alignment subgraph extraction module, a path-based graph neural network module, and a node-level multimodal attention module. First, through the alignment subgraph extraction algorithm and path importance evaluation technology, the model is able to identify and retain the structural information most valuable for entity alignment. During the alignment process, for entity pairs with high structural similarity and many shared attributes, matching is prioritized through the rule paths in the alignment subgraph; for entity pairs with insufficient structural information but rich multimodal attributes, indicating the need for more attribute-level evidence, the multimodal attention mechanism is activated for in-depth analysis; and for entity pairs with both partial structural matching and related attributes, a weighted fusion of structural information and multimodal attributes is used to ensure the reliability and accuracy of the alignment results.

[0028] In practical applications, this approach can promote cross-modal knowledge integration and enhance the understanding and reasoning capabilities of artificial intelligence systems, especially in fields such as search engines, recommendation systems, and natural language processing, demonstrating great potential and value.

[0029] The steps S1 to S4 are explained in detail below: S1, based on entity and entity relationship information in two knowledge graphs, generates an alignment subgraph of entities by defining alignment rules and extracting alignment paths, specifically including: The input for this step is entity and relation information from two knowledge graphs. By defining alignment rules and extracting alignment paths, a subgraph structure containing all possible alignment information is constructed, thereby achieving structured feature extraction for cross-knowledge graph entity alignment.

[0030] This application defines alignment subgraph extraction as a process of identifying and extracting structured information related to entity alignment from two knowledge graphs. For example... Figure 1 As shown, the core idea is to define different types of alignment rules and construct alignment paths based on these rules, ultimately forming a subgraph structure containing complete alignment information. This process mainly consists of three steps: defining alignment rules, constructing alignment paths, and generating aligned subgraphs.

[0031] 1) The process of defining alignment rules includes: Alignment rules include three types: single-hop rules, symmetric k-hop rules, and asymmetric k-hop rules. Single-hop rules describe when entities... Through the same relationship Alignment is determined when connecting to aligned entity pairs. The symmetric k-hop rule focuses on scenarios where entity pairs are connected to aligned entity pairs via the same k-step relation sequence. The asymmetric k-hop rule further extends to cases where entities are connected to aligned entity pairs via semantically equivalent paths of different lengths. These three rules collectively constitute the fundamental criteria for entity alignment, providing a theoretical basis for subsequent path construction.

[0032] 2) The process of constructing the alignment path includes: Alignment path The construction process first relies on anchor connections, which include both existing alignment seeds and connection points established based on the similarity of entity multimodal attributes. During construction, the system reverses the relationships in the knowledge graph to achieve bidirectional search, while identifying and retaining paths that conform to alignment rules. By integrating this information, a complete alignment path structure is ultimately formed. This path construction method ensures that all possible associations between entities are fully captured.

[0033] 3) The process of generating aligned subgraphs includes: Align subgraphs The generation of the alignment subgraph is achieved by aggregating all possible alignment paths. In this process, the system first extracts all valid paths with a length not exceeding K, then merges paths with common nodes or edges, ultimately constructing a K-hop alignment subgraph. This method not only ensures the integrity of structural information, but also controls computational complexity by limiting the path length K, achieving a good balance between efficiency and effectiveness.

[0034] Through the above process, this application can effectively extract and integrate the structured features required for entity alignment. This subgraph-based alignment method provides rich structural information support for subsequent entity alignment tasks, helping to improve the accuracy and reliability of alignment. In particular, by limiting the path length K, both the integrity of structural information and computational complexity are ensured, achieving a balance between efficiency and effectiveness.

[0035] S2, dynamically merge the aligned subgraphs. For each node in the merged aligned subgraph, update the message passing process based on the interpretable path by aggregating information from neighboring nodes, and then calculate the path score of the aligned subgraph, specifically including: This module achieves entity alignment across knowledge graphs through a dynamic merging strategy and a message passing mechanism based on interpretable paths. It effectively handles alignment relationships between entities from different knowledge graphs and captures complex association patterns between entities using a neural network structure.

[0036] This application defines an alignment subgraph neural network as a neural network structure that performs message passing on merged alignment subgraphs. Its core idea is to achieve efficient and interpretable entity alignment by dynamically merging alignment paths and designing an interpretable message passing mechanism. This process mainly consists of three steps: dynamic merging of alignment subgraphs, message passing based on interpretable paths, and calculation of alignment scores.

[0037] 1) The dynamic merging process of aligned subgraphs includes: The dynamic merging process of aligned subgraphs aims to solve the problem of merging subgraphs from knowledge graphs. Entities in arrive The problem of calculating the alignment score of all entities in the equation. and Shared ancestor At that time, from Nodes affected by the incoming path The representation can be used for both entities simultaneously. By sharing node representations through dynamic programming, the system can significantly reduce computational overhead. (Merged aligned subgraph) Includes from arrive All paths, formally represented as .

[0038] 2) Message passing process based on interpretable paths, including: Message passing mechanisms ensure that information is transmitted from entity to entity by designing specific update rules. Towards Effective propagation is achieved. For each node in the merged aligned subgraph, its path representation is updated by aggregating information from neighboring nodes, using the following formula: The attention weights are calculated using an interpretable attention mechanism to ensure that the most relevant path information is processed first.

[0039] 3) The calculation process for the alignment score includes: After completion After the layer update, the system obtained all entities. The final expression These representations contain key path-based information for calculating entity pairs. The alignment score is calculated using a multilayer perceptron and is represented as follows: .

[0040] Through the above process, this application can effectively achieve entity alignment across knowledge graphs. In particular, by introducing an interpretable attention mechanism and a dynamic merging strategy, both the interpretability of the alignment results and computational efficiency are guaranteed. Theoretical analysis shows that there exists a parameter setting and threshold in the neural network that allows the aligned subgraph constructed based on attention weights to accurately reflect the path structure that conforms to the alignment rules, thus providing a theoretical guarantee for the interpretability of the model.

[0041] S3, the alignment subgraph is characterized by an attribute encoder to represent the multimodal attributes of entities, and by an alignment attention mechanism to represent the multimodal attributes of entities in response to perception. An attribute alignment score is calculated between different entities in the multimodal knowledge graph, specifically including: This module implements attribute alignment scoring between different entities in a multimodal knowledge graph by using an attribute encoder and an alignment attention mechanism. It effectively processes attribute information from different modalities such as text and images, and highlights the importance of shared attributes through the attention mechanism, thereby obtaining accurate alignment scores.

[0042] This application defines alignment modality score calculation as a unified process for processing multimodal attributes and calculating entity alignment similarity. Its core idea is to achieve unified representation and effective fusion of different modal attributes through specific encoding transformations and attention mechanisms. This process mainly consists of three steps: attribute encoding transformation, alignment attention calculation, and generation of multimodal attribute alignment scores.

[0043] 1) The attribute encoding conversion process includes: In multimodal knowledge graphs, different types of attributes (such as text and images) have different representation spaces, which poses a challenge to attribute alignment. To address this issue, this application employs fully connected layers specific to each modality for feature transformation. For each modality... Through the weight matrix Initial features Projecting onto a unified representation space, the transformation formula is as follows: The initial features for the text modality come from the output of the BERT model, while the initial features for the image modality come from a pre-trained visual model, ensuring high-quality representation of features from different modalities.

[0044] 2) Alignment attention calculation process, including: To address the interference caused by mismatches in the types and quantities of different attributes during entity alignment, this application designs an alignment attention mechanism. This mechanism effectively reduces the impact of irrelevant attributes by calculating the importance weights of shared attributes between entities. For entity pairs... and The pairwise sensing multimodal representation is obtained through weighted aggregation: Among them, attention weights By calculating the similarity between attribute types and applying the softmax function, it is ensured that shared attributes receive higher weights.

[0045] 3) The process of generating multimodal attribute alignment scores includes: After obtaining the multimodal representations of entity pairs, this application processes the combination of these representations using a multilayer perceptron (MLP) to generate the final alignment score. Specifically, the entity pairs are... and After performing element-wise multiplication on the multimodal representation, the alignment score is calculated using an MLP: .

[0046] S4. Based on the path score and attribute alignment score of the aligned subgraph, the comprehensive entity similarity evaluation score is obtained, and the multimodal entity pairs are aligned based on the comprehensive entity similarity evaluation score.

[0047] Through the above process, this application can effectively handle the entity alignment problem in multimodal knowledge graphs. In particular, by introducing attribute encoding transformation and alignment attention mechanisms, it not only solves the problem of inconsistent attribute representations across different modalities but also highlights the importance of shared attributes, ensuring the accuracy of alignment scores. This design can not only handle complex multimodal attributes but also adapt to differences in attribute types and quantities among different entities, providing reliable technical support for entity alignment in multimodal knowledge graphs.

[0048] Furthermore, this application significantly improves the performance and efficiency of entity alignment by combining a dynamic merging strategy and an interpretable message passing mechanism. Especially when processing large-scale knowledge graphs, the dynamic merging process of aligned subgraphs effectively reduces computational overhead while maintaining the accuracy of the alignment results. By designing a message passing mechanism based on interpretable paths, this application not only provides high-quality alignment results but also ensures the interpretability of the results, providing reliable technical support for knowledge graph integration and knowledge discovery.

[0049] Furthermore, this application addresses the problem of unified representation of attributes across different modalities in multimodal knowledge graphs. By designing a specific attribute encoder and alignment attention mechanism, this application can effectively process attribute information from different modalities such as text and images, and highlights the importance of shared attributes through the attention mechanism. This design not only adapts to differences in the types and quantities of attributes between different entities, but also improves the robustness of the model while ensuring alignment accuracy. When dealing with complex multimodal knowledge graphs, the method in this application can provide stable and reliable alignment results.

[0050] Furthermore, by comprehensively utilizing graph structure information and multimodal attribute information, the comprehensiveness and accuracy of entity alignment results are ensured. This method can intelligently balance the importance of structural and attribute information when processing knowledge graphs with complex relational structures and rich attribute information, thereby improving alignment performance. In practical applications, such as cross-language knowledge graph alignment and multi-source knowledge fusion, the method presented in this application demonstrates excellent adaptability and stability.

[0051] Secondly, this application discloses a multimodal entity alignment system based on alignment subgraph rule mining, comprising: Alignment Subgraph Extraction Module: This module takes entities and their relationships from two knowledge graphs as input and constructs an alignment subgraph for a specific entity using an alignment subgraph extraction algorithm. By using anchor links as bridges, it propagates along paths across the knowledge graph, collecting all possible alignment rule paths and filtering out irrelevant neighbor information, thus providing crucial structural information support for subsequent alignment evaluation.

[0052] The path-based graph neural network module: This module uses an interpretable path-based graph neural network for rule mining and scoring based on the extracted alignment subgraph. It employs a unidirectional path message passing strategy to preserve specific path information and utilizes an interpretable attention mechanism to prioritize edges on important paths. Dynamic programming is used to improve message passing efficiency, ensuring that the generated entity representation accurately captures key alignment details.

[0053] Node-level Multimodal Attention Module: This module integrates multimodal attribute information of entities, including text and images, through a unified multimodal attention mechanism. Addressing the scarcity of anchor links in the real world, this module facilitates the extraction of aligned subgraphs by creating new multimodal anchor links and integrating auxiliary anchor data. Simultaneously, it introduces an alignment attention mechanism, prioritizing shared attributes between entities and reducing interference from mismatched attributes, thereby improving the accuracy of multimodal entity alignment.

[0054] The multimodal entity pair alignment module is used to obtain a comprehensive entity similarity assessment score based on the path score and attribute alignment score of the aligned subgraph, and to align multimodal entity pairs based on the comprehensive entity similarity assessment score.

[0055] Based on the above scheme, the system in this application solves the problem of cross-knowledge graph entity alignment by designing an alignment subgraph neural network module and an alignment modality score calculation module. In traditional entity alignment methods, it is often difficult to effectively handle the structural differences and multimodal attributes between different knowledge graphs, which not only affects the accuracy of alignment but also limits the practical application scope of the method.

[0056] The present application will now be described in further detail with reference to the accompanying drawings and embodiments, so that those skilled in the art can implement it based on the description. This embodiment uses a cross-language knowledge graph alignment task as an example to illustrate the entire implementation process. In this example, the input consists of two knowledge graphs with different language annotations, and the output is the alignment relationship between entities across the graphs, including four modules: The alignment subgraph extraction module takes entities and their related structural information from two knowledge graphs to be aligned as input. Through this module, it dynamically merges and effectively encodes the local structures surrounding the entities. As the scale of knowledge graphs continues to expand, traditional full-graph processing methods face severe challenges in terms of computational efficiency and memory consumption. To address this issue, this application adopts a rule-based subgraph construction strategy. By selectively retaining the most representative structural information, it significantly reduces computational complexity while ensuring alignment effectiveness.

[0057] This application defines alignment subgraph extraction as a rule-based structural information extraction process. Its core idea is to identify and extract local structures related to entity alignment through predefined alignment rules. This process mainly includes three key steps: defining alignment rules, constructing alignment paths, and generating alignment subgraphs.

[0058] 1) The process of defining alignment rules includes: This application defines three basic alignment rules for capturing possible alignment relationships between entities: (i) Single-hop rule: Given a knowledge graph and Entity pairs in and If they are respectively through the same relationship Connect to the aligned entity pair and Then we can infer and It is aligned. Formalized as:

[0059] (ii) Symmetric k-hop rule: If entity pairs and They are connected to their k-hop neighbors using the same relation sequence. and If these neighbors are aligned, then it can be inferred that the original entity pair is also aligned.

[0060] (iii) Asymmetric k-hop rule: Consider entity pairs and The case of connecting aligned entity pairs through semantically equivalent relation sequences with varying numbers of hops.

[0061] 2) The process of constructing the alignment path includes: Based on the above alignment rules, this application introduces the concept of an alignment path, which is defined as a sequence of relations connecting pairs of entities to be aligned. Formalized as:

[0062] in This represents an anchor link, used to connect two knowledge graphs.

[0063] 3) The process of generating aligned subgraphs includes: Alignment subgraphs are constructed by aggregating all possible alignment paths. Given entity pairs... Its K-jump aligned subgraph is defined as:

[0064] Through the above process, this application can effectively extract and integrate local structural information related to entity alignment. This rule-based subgraph extraction method not only significantly reduces computational complexity but also ensures the relevance and completeness of the extracted information. Experimental results show that this method achieves excellent performance on multiple cross-lingual knowledge graph alignment datasets, confirming the effectiveness and practical value of this application.

[0065] The entity alignment module based on the alignment subgraph neural network achieves efficient entity alignment after acquiring entity information from two knowledge graphs through a dynamic merging strategy and a message passing mechanism based on interpretable paths. For multiple target entities with a common ancestor node, a dynamic programming method is used to share node representations to reduce computational overhead. Simultaneously, a novel message passing mechanism is designed to ensure effective information transmission within the alignment subgraph, thereby improving the accuracy and efficiency of alignment.

[0066] This application defines the alignment subgraph neural network as a graph-based deep learning model. Its core idea is to achieve entity alignment across knowledge graphs through dynamic merging and message passing. This process mainly includes the following three key steps: implementation of the dynamic merging strategy, message passing based on interpretable paths, and calculation of alignment scores.

[0067] 1) The implementation process of the dynamic merging strategy includes: This application proposes an innovative dynamic merging strategy for solving the problem of merging from knowledge graphs. Entities in To knowledge graph The problem involves calculating the alignment score of all entities in the graph. The merged alignment subgraph is defined as follows:

[0068] This strategy reduces redundant computation by identifying and utilizing shared structures, especially when When multiple target entities share the same ancestor node, the representation of that node can be reused, significantly improving computational efficiency.

[0069] 2) The message passing process based on interpretable paths includes: This application designs a novel message passing mechanism, the core of which is the path representation calculation formula:

[0070] The attention weights are calculated using an interpretable attention mechanism:

[0071] To address the heterogeneity between different knowledge graphs, this application introduces a knowledge graph feature fusion function:

[0072] And process sequence information using a recurrent neural network:

[0073] The final alignment score is calculated using a multilayer perceptron:

[0074] The interpretability of this application is theoretically guaranteed by Theorem 1, namely, the existence of parameter settings. and threshold This makes the attention weights The resulting subgraph is structurally equivalent to the subgraph formed by paths containing alignment rules. This theoretical guarantee ensures the reliability and interpretability of the method in practical applications.

[0075] Through the above technical solution, this application not only significantly improves the efficiency and accuracy of entity alignment in knowledge graphs, but also ensures the interpretability of the alignment results. This method is particularly suitable for entity alignment tasks in large-scale cross-lingual knowledge graphs, providing reliable technical support for cross-lingual fusion of knowledge graphs.

[0076] The alignment score calculation module based on multimodal attributes mainly comprises two key components: an attribute encoder and an alignment attention score, used to handle entity alignment problems in multimodal knowledge graphs (MMKGs). This module improves alignment accuracy by unifying the representation spaces of different modalities and introducing an attention mechanism.

[0077] 1) Multimodal attribute alignment score includes: The attribute encoder section uses a fully connected layer with a specific modality. Feature projection is performed to map features from different modalities to a unified space. The specific implementation is as follows:

[0078] in, Representing different modalities such as text and visual, For modality The initial attribute features are derived from the BERT model. For the text modality, the initial embeddings come from the BERT model, while for the image modality, the initial embeddings come from a pre-trained visual model.

[0079] The alignment attention score section introduces an innovative attention mechanism that prioritizes entities. and The shared attributes between them effectively reduce the noise impact caused by different attribute types and quantities. Pairwise sensing multimodal representation The calculation formula is:

[0080] Attention score Calculated using the following formula:

[0081] The final multimodal attribute alignment score is obtained by processing the combinatorial representation of entity pairs using a multilayer perceptron (MLP):

[0082] 2) The model training strategy and objective function include: This application employs innovative training strategies and optimization objectives, mainly including the following two aspects: During the testing phase, this application uses all alignment seeds as anchor links to extract aligned subgraphs. To accurately simulate this scenario, alignment seeds... Randomly divided into two subsets: and They accounted for 75% and 25% respectively. Used as a known anchor link, auxiliary pair Make predictions.

[0083] This application proposes two model variants: ASGEA-Stru and ASGEA-MM. ASGEA-Stru uses only graph structure properties and adopts... Calculate the alignment score. ASGEA-MM, by introducing AMS, extends this to obtain a comprehensive score:

[0084] For the training set Each pair of entities in The loss function is defined as:

[0085] in Representation and inclusion The knowledge graph is relative to the knowledge graph. This application uses stochastic gradient descent for optimization to minimize the aforementioned loss function.

[0086] Through the above technical solution, this application not only achieves effective fusion of multimodal attributes, but also significantly improves the accuracy of entity alignment through attention mechanisms and optimized training strategies. This method is particularly suitable for knowledge graph alignment tasks involving multiple modal attributes such as text and images.

[0087] A third objective of this application is to provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the aforementioned multimodal entity alignment method based on aligned subgraphs. It also includes a communication interface and a bus.

[0088] When the processor executes the computer program, it implements the aforementioned multimodal entity alignment method based on aligned subgraphs, specifically including: Based on entity and entity relationship information in two knowledge graphs, alignment subgraphs of entities are generated by defining alignment rules and extracting alignment paths; The alignment subgraph is dynamically merged. For each node in the merged alignment subgraph, the message passing process based on the interpretable path is updated by aggregating information from neighboring nodes, and then the path score of the alignment subgraph is calculated. The alignment subgraph is characterized by an attribute encoder to represent the multimodal attributes of entities and by an alignment attention mechanism to represent the multimodal attributes of entities in response to perception. The attribute alignment score between different entities in the multimodal knowledge graph is calculated. Based on the path score and attribute alignment score of the aligned subgraph, a comprehensive entity similarity evaluation score is obtained, and multimodal entity pairs are aligned based on the comprehensive entity similarity evaluation score.

[0089] In this embodiment, the computer-readable storage medium is a non-volatile storage medium, specifically a ROM, RAM, disk, or optical disk. In this embodiment, the computer-readable storage medium is a 512GB NVMe SSD, which uses 3D NAND flash memory technology and features high-speed read / write performance and long lifespan.

[0090] When the computer program is executed by the processor, it implements the aforementioned multimodal entity alignment method based on aligned subgraphs, specifically including: Based on entity and entity relationship information in two knowledge graphs, alignment subgraphs of entities are generated by defining alignment rules and extracting alignment paths; The alignment subgraph is dynamically merged. For each node in the merged alignment subgraph, the message passing process based on the interpretable path is updated by aggregating information from neighboring nodes, and then the path score of the alignment subgraph is calculated. The alignment subgraph is characterized by an attribute encoder to represent the multimodal attributes of entities and by an alignment attention mechanism to represent the multimodal attributes of entities in response to perception. The attribute alignment score between different entities in the multimodal knowledge graph is calculated. Based on the path score and attribute alignment score of the aligned subgraph, a comprehensive entity similarity evaluation score is obtained, and multimodal entity pairs are aligned based on the comprehensive entity similarity evaluation score.

[0091] A fifth objective of this application is to provide a computer program product comprising computer instructions that instruct a computer to execute the multimodal entity alignment method based on aligned subgraphs.

[0092] In this embodiment, the computer program product is a software package, including an installer, main program, configuration files, help documentation, and sample data. The software package is distributed as an ISO image file and can be installed on the target computer via CD or USB flash drive.

[0093] The computer instructions in the computer program product are written in multiple programming languages, including C++ (core algorithms and hardware interfaces), Python (data analysis and visualization), and JavaScript (web interface). The program architecture adopts a modular design, including the following main modules: Core module: Responsible for implementing the core algorithm of the multimodal entity alignment method based on aligned subgraphs; Hardware interface module: responsible for communicating with sensors and control devices; Data storage module: responsible for storing runtime data in the database; Data analysis module: responsible for analyzing historical data and extracting useful information; Visualization module: Responsible for displaying data in chart form; Web service module: Provides a web interface for users to remotely access and control the system.

[0094] The computer instructions direct the computer to execute the aforementioned multimodal entity alignment method based on aligned subgraphs, specifically including: Based on entity and entity relationship information in two knowledge graphs, alignment subgraphs of entities are generated by defining alignment rules and extracting alignment paths; The alignment subgraph is dynamically merged. For each node in the merged alignment subgraph, the message passing process based on the interpretable path is updated by aggregating information from neighboring nodes, and then the path score of the alignment subgraph is calculated. The alignment subgraph is characterized by an attribute encoder to represent the multimodal attributes of entities and by an alignment attention mechanism to represent the multimodal attributes of entities in response to perception. The attribute alignment score between different entities in the multimodal knowledge graph is calculated. Based on the path score and attribute alignment score of the aligned subgraph, a comprehensive entity similarity evaluation score is obtained, and multimodal entity pairs are aligned based on the comprehensive entity similarity evaluation score.

[0095] This application's multimodal entity alignment method significantly improves the accuracy, interpretability, and cross-modal processing capabilities of entity alignment by combining path information and multimodal attribute encoding. Dynamically merging aligned subgraphs and using an interpretable attention mechanism ensure the algorithm remains efficient and accurate even when processing complex data. The interconnectedness of different embodiments demonstrates the innovative aspects of the technical solution at different levels and its benefits in practical applications, providing a powerful tool for constructing and improving multimodal knowledge graphs and promoting information integration and decision support in the field of artificial intelligence.

[0096] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0097] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0098] This application may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application may take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, readable storage media, optical storage, etc.) containing computer-usable program code.

[0099] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0100] Obviously, the described embodiments are only some, not all, of the embodiments in this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort should fall within the scope of protection of this application.

[0101] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A multimodal entity alignment method based on aligned subgraphs, characterized in that, include: Based on entity and entity relationship information in two knowledge graphs, alignment subgraphs of entities are generated by defining alignment rules and extracting alignment paths; The alignment subgraph is dynamically merged. For each node in the merged alignment subgraph, the message passing process based on the interpretable path is updated by aggregating information from neighboring nodes, and then the path score of the alignment subgraph is calculated. The alignment subgraph is characterized by an attribute encoder to represent the multimodal attributes of entities and by an alignment attention mechanism to represent the multimodal attributes of entities in response to perception. The attribute alignment score between different entities in the multimodal knowledge graph is calculated. Based on the path score and attribute alignment score of the aligned subgraph, a comprehensive entity similarity evaluation score is obtained, and multimodal entity pairs are aligned based on the comprehensive entity similarity evaluation score.

2. The multimodal entity alignment method based on aligned subgraphs according to claim 1, characterized in that, The alignment rules include three types: single-hop rules, symmetrical K-hop rules, and asymmetrical K-hop rules. Single-hop rules are used to describe when entity pairs Through the same relationship Alignment when connecting to an already aligned entity pair; The symmetric K-jump rule is used to focus on scenarios where entity pairs are connected to aligned entity pairs through the same k-step relation sequence; The asymmetric K-hop rule is used to extend to the case where aligned entity pairs are connected via semantically equivalent but different length paths.

3. The multimodal entity alignment method based on aligned subgraphs according to claim 1, characterized in that, The extracted alignment path includes: The alignment path is based on anchor point connections, which include existing alignment seeds and connection points established based on the similarity of entity multimodal attributes. The alignment path reverses the relationships in the knowledge graph to perform bidirectional search, identifies and retains paths that conform to the alignment rules, and forms a complete alignment path structure by integrating information.

4. The multimodal entity alignment method based on aligned subgraphs according to claim 1, characterized in that, The alignment subgraph of the generated entity includes: Extract all valid paths with a length not exceeding K, merge paths with common nodes or edges, and construct a K-jump aligned subgraph. An alignment subgraph is generated by aggregating all possible alignment paths. .

5. The multimodal entity alignment method based on aligned subgraphs according to claim 1, characterized in that, Dynamically merging the aligned subgraphs includes: When multiple target entities share the same ancestor node, a dynamic programming method is used to share the node and dynamically merge the aligned subgraphs. The merged aligned subgraphs are then... Includes from arrive All paths, formally represented as .

6. The multimodal entity alignment method based on aligned subgraphs according to claim 1, characterized in that, In the message passing process based on interpretable paths, a unidirectional path message passing strategy is adopted to retain specific path information, and an interpretable attention mechanism is used to prioritize the edges on important paths.

7. The multimodal entity alignment method based on aligned subgraphs according to claim 1, characterized in that, The calculation of the path score for the aligned subgraph includes: A knowledge graph feature fusion function is introduced, and sequence information is processed through a recurrent neural network. Finally, a multilayer perceptron is used to calculate the path score of the aligned subgraph.

8. The multimodal entity alignment method based on aligned subgraphs according to claim 1, characterized in that, The alignment subgraph is characterized by an attribute encoder to represent the multimodal properties of entities, including: Fully connected layers employing specific modes of attribute encoders Perform feature transformation on aligned subgraphs to map features from different modalities to a unified space; each modality Through the weight matrix Initial features Projecting onto a unified representation space to characterize multimodal properties, the transformation formula is as follows: ; The different modalities include text modality and image modality; wherein, the initial features of text modality come from the output of BERT model, and the initial features of image modality come from pre-trained visual model.

9. The multimodal entity alignment method based on aligned subgraphs according to claim 1, characterized in that, The method of characterizing the multimodal attributes of an entity in response to perception through the alignment attention mechanism includes: For entity pairs and The pairwise sensing multimodal representation is obtained through weighted aggregation: Among them, attention weight It is obtained by calculating the similarity between attribute types and applying the softmax function.

10. The multimodal entity alignment method based on aligned subgraphs according to claim 1, characterized in that, The attribute alignment scoring between different entities in a multimodal knowledge graph includes: After obtaining the multimodal attribute representations of entity pairs, the multimodal attributes of the entity pairs are processed by a multilayer perceptron to classify the entities... and After performing element-wise multiplication on the multimodal representation, the alignment score is calculated using an MLP: .

11. A multimodal entity alignment system based on aligned subgraphs, characterized in that, The multimodal entity alignment method based on aligned subgraphs according to any one of claims 1-10 includes: The alignment subgraph extraction module is used to generate alignment subgraphs of entities based on entity and entity relationship information in two knowledge graphs by defining alignment rules and extracting alignment paths. A path-based graph neural network module is used to dynamically merge the alignment subgraph. For each node in the merged alignment subgraph, the message passing process based on the interpretable path is updated by aggregating information from neighboring nodes, and then the path score of the alignment subgraph is calculated. The node-level multimodal attention module is used to characterize the multimodal attributes of entities in the alignment subgraph through an attribute encoder, and to characterize the multimodal attributes of entities to perception through an alignment attention mechanism, and to calculate the attribute alignment score between different entities in the multimodal knowledge graph. The multimodal entity pair alignment module is used to obtain a comprehensive entity similarity assessment score based on the path score and attribute alignment score of the aligned subgraph, and to align multimodal entity pairs based on the comprehensive entity similarity assessment score.

12. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the multimodal entity alignment method based on aligned subgraphs as described in any one of claims 1-10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the multimodal entity alignment method based on aligned subgraphs as described in any one of claims 1 to 10.

14. A computer program product, the computer program product comprising computer instructions, characterized in that, The computer instructions instruct the computer to execute the multimodal entity alignment method based on aligned subgraphs as described in any one of claims 1-10.

Images