Graph neural network and dynamic relationship interactive graph entity linking method
By constructing a multimodal entity link model with multi-scale dynamic relationship interaction, the problem of insufficient utilization of multimodal features is solved, and more efficient entity linking effect is achieved, and the stability and performance of the model are enhanced.
Patent Information
- Application Number
- CN202510603229.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-12
- Publication Date
- 2025-08-19
AI Technical Summary
The existing multimodal entity link model does not fully utilize multimodal features, which affects performance improvement.
A multimodal entity link model based on multi-scale dynamic relationship interaction is constructed, including a fine-grained relational interaction module, a global attention fusion module, a feedforward neural network module, a Softmax normalization module and a similarity calculation module. Multimodal features are extracted through pre-trained models, and the model parameters are optimized using the loss function until the specified round is reached.
Make full use of multimodal features to improve entity linking effects, avoid single modal dominance decision making, and enhance the stability and efficiency of the model.
Smart Images
Figure CN120509469A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a graph entity linking method for interaction between a graph neural network and dynamic relationships. Background Art
[0002] As the core carrier of structured semantic knowledge, knowledge graphs enable systematic expression and associative reasoning of knowledge through triples of entities, attributes, and relationships. They possess strong capabilities in semantic understanding of complex information, knowledge fusion, and logical reasoning. By constructing domain knowledge networks, knowledge graphs significantly improve the accuracy of semantic search, the relevance of intelligent recommendations, and the interpretability of question-answering systems, becoming the infrastructure for AI to achieve cognitive intelligence. Multimodal graphs expand knowledge graphs, offering enhanced expressive capabilities. By integrating multidimensional data such as images, videos, voice, and sensors, they deeply associate unstructured data such as visual features, spatiotemporal information, and acoustic signals with semantic knowledge, achieving a leap from unimodal symbolic reasoning to multimodal perceptual collaboration. Despite rapid development, high-quality, large-scale multimodal knowledge graphs are still lacking.
[0003] This gave rise to the task of knowledge graph fusion, and entity linking is a key technical approach. It aims to expand the information in the knowledge base by linking mentions in documents to entities in the knowledge base, thereby constructing a larger-scale graph. Early multimodal entity linking models focused on fusing and expanding multimodal data, for example, by combining visual and textual information to enhance textual mentions and predict matching entities. GHMFC extracts hierarchical features from text and vision through a multimodal co-attention mechanism. It obtains weighted visual features guided by textual information and generates weighted textual features guided by visual information. It then uses gated fusion to assess the importance of hierarchical features from different modalities and integrates them into a final multimodal representation of mentions. With the development of large language models, recent multimodal entity linking models also employ pre-trained large models for feature representation. Researchers typically use the BERT model for textual information processing and the CLIP model for visual feature encoding. GEMEL uses LLaMA for language processing and CLIP for visual encoding, effectively combining these two models to achieve entity linking. DRIN no longer fuses multimodal features, but extracts features from the interactive relationship between text and visual features respectively, mines implicit features from fine-grained interactions, and enhances the final representation.
[0004] Although the development of multimodal entity linking models has achieved good results, there are also many challenges. The current models are still insufficient in mining and utilizing multimodal features, which affects further improvement of performance. Summary of the Invention
[0005] The purpose of the present invention is to provide a graph entity linking method that interacts with graph neural networks and dynamic relationships, aiming to solve the problem of insufficient utilization of multimodal features in existing entity linking methods.
[0006] To achieve the above objectives, the present invention provides a graph entity linking method for interacting with a graph neural network and dynamic relationships, comprising the following steps:
[0007] Construct a training dataset for completing the multimodal knowledge graph entity linking task;
[0008] Extract multimodal features of mentions and entities in the training dataset to obtain input features;
[0009] Construct a multimodal entity linking model based on multi-scale dynamic relationship interaction and initialize the model parameters;
[0010] Input the input features into the network to obtain multi-scale output features;
[0011] Update model parameters based on loss function optimization;
[0012] Repeat the steps of inputting input features into the network, obtaining multi-scale output features, and optimizing and updating model parameters according to the loss function until the specified round is reached;
[0013] The test data is input into the trained model, and the one with the greatest similarity is selected as the final link result according to the link strategy.
[0014] The steps of constructing a multimodal entity linking model based on multi-scale dynamic relationship interaction and initializing model parameters further include:
[0015] The entity linking model includes a fine-grained relationship interaction module, a global attention fusion module, a feedforward neural network module, a Softmax normalization module, a similarity calculation module and a two-stage entity linking module based on a decision threshold;
[0016] The fine-grained relationship interaction module is used to complete the relationship interaction between fine-grained features such as text or visual features, and extract the feature representation after feature interaction;
[0017] The global attention fusion module is used to complete feature fusion and obtain fusion features;
[0018] The feedforward neural network module is used to complete the feature space transformation of the hidden layer features;
[0019] The Softmax normalization module is used to normalize the hidden layer output;
[0020] The similarity calculation module is used to calculate the similarity between output features of different scales;
[0021] The two-stage entity linking module based on decision threshold is used to complete the final entity linking task goal.
[0022] The step of extracting multimodal features of mentions and entities in the training dataset to obtain input features further includes:
[0023] For text data, we use the pre-trained BERT model to extract the corresponding features. The model can be obtained from the open source address to obtain the mentioned text feature representation. and text feature representation of candidate entities ;
[0024] For image data, the ResNet network is used to extract features from the image. The model can be obtained from the open source address to obtain the mentioned visual feature representation and visual feature representation of candidate entities .
[0025] The step of inputting the input features into the network to obtain multi-scale output features further includes:
[0026] Initialize the number of training rounds for model training;
[0027] Extract batches from the input features. Each set of data includes the textual and visual features of the mentions and the textual and visual features of the corresponding candidate entities. All features are then constructed into a sampling subgraph. The vertices in the subgraph are different features, and the edges are the interactions between the features of each modality. Specifically, the subgraph includes the relationship between the mentioned text and the entity text, the relationship between the mentioned image and the entity text, the relationship between the mentioned text and the entity image, and the relationship between the mentioned image and the entity image.
[0028] The sampled subgraph is input into the fine-grained relational interaction network module of the model to complete the relational interaction propagation. For example, the text features of the mentions receive the text and image features from the corresponding candidate entities during the interaction to obtain the features after the relational interaction.
[0029] Perform dimension expansion and broadcast replication on the mention features in the interactive output features to make them consistent with the candidate entity feature dimensions;
[0030] Calculate the similarity between the mentions in the interactive output features and all candidate entity features, and get 4 sets of similarities, which are 、 、 and ;
[0031] The original text features, original image features, text output features after interaction, and image output features mentioned are input into the global attention fusion module to learn the importance of cross-modal features and obtain fused features;
[0032] For candidate entities, the original text features, original image features, text output features after interaction, and image output features are input into the global attention fusion module to learn the importance of cross-modal features and obtain the candidate entity fusion features;
[0033] Perform dimension expansion and broadcast replication on the mentioned fusion features to make them consistent with the candidate entity feature dimensions;
[0034] The similarity between the mention and the fused features of each candidate entity is calculated to obtain the fused feature similarity.
[0035] Wherein, the model parameters are optimized and updated according to the loss function, and the steps further include:
[0036] Compare the fusion feature similarity between the mention and all candidate entities, and calculate the difference between the highest and second highest fusion feature similarity values;
[0037] If the difference is greater than the threshold , then take the candidate entity with the highest similarity as the result, then compare the entity link result with the corresponding correct answer to complete the model loss calculation;
[0038] If the difference is greater than or equal to the threshold , then compare the similarities of the four groups of interactive features and take the candidate entity with the largest sum of similarities as the entity linking result;
[0039] Optimize and update the parameters of the knowledge graph entity linking model based on the model loss, and increase the current number of training rounds by 1.
[0040] The present invention provides a graph entity linking method for the interaction of a graph neural network and dynamic relationships. It extracts and utilizes features of different granularities of multimodal data through a multi-scale relationship interaction network, avoids the information limitations brought by a single feature, makes full use of feature representation, improves the entity linking effect, and considers all possible modal combinations at the same time, avoids a single modality dominating the decision, converges faster, has strong stability, and avoids the problem of excessive resource consumption caused by the increase in model complexity. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art.
[0042] Figure 1 This is a step diagram of the graph entity linking method for the interaction of a graph neural network and dynamic relationships according to the first embodiment of the present invention. DETAILED DESCRIPTION
[0043] The embodiments of the present invention are described in detail below. Examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, but should not be understood as limiting the present invention.
[0044] The first embodiment of this application is:
[0045] See also Figure 1 ,in Figure 1 This is a step diagram of the graph entity linking method for the interaction of a graph neural network and dynamic relationships according to the first embodiment of the present invention.
[0046] The present invention provides a graph entity linking method for interacting a graph neural network with dynamic relationships, comprising the following steps:
[0047] S1: Construct a training dataset for completing the multimodal knowledge graph entity linking task;
[0048] S2: Extract multimodal features of mentions and entities in the training dataset to obtain input features;
[0049] S3: Construct a multimodal entity linking model based on multi-scale dynamic relationship interaction and initialize the model parameters;
[0050] S4: Input the input features into the network to obtain multi-scale output features;
[0051] S5: Optimize and update model parameters according to the loss function;
[0052] S6: Repeat S4 and S5 until the specified round is reached;
[0053] S7: Input the test data into the trained model and select the one with the greatest similarity as the final link result according to the link strategy.
[0054] In step S2 of the embodiment of the present invention, the mention and the entity in the knowledge base both have multimodal data of text and image. Multimodal entity linking needs to link the mention to the correct entity in the knowledge base based on the features. It includes the following sub-steps:
[0055] S21: For text data, use the pre-trained BERT model to extract the corresponding features. The model can be obtained from the open source address to obtain the mentioned text feature representation and text feature representation of candidate entities ;
[0056] S22: For image data, the ResNet network is used to extract features from the image. The model can be obtained from the open source address to obtain the mentioned visual feature representation and visual feature representation of candidate entities .
[0057] In step S3 of this embodiment, the entity linking model includes a fine-grained relationship interaction module, a global attention fusion module, a feedforward neural network module, a Softmax normalization module, a similarity calculation module and a two-stage entity linking module based on a decision threshold.
[0058] Among them, the fine-grained relationship interaction module is used to complete the relationship interaction between fine-grained features such as text or visual features, and extract the feature representation after feature interaction; the global attention fusion module is used to complete feature fusion and obtain fusion features; the feedforward neural network module is used to complete the feature space transformation of hidden layer features; the Softmax normalization module is used to complete the hidden layer output normalization operation; the similarity calculation module is used to calculate the similarity between output features of different scales; and the two-stage entity linking module based on decision threshold is used to complete the final entity linking task goal. The specific details will be introduced in step S5.
[0059] S3 includes the following sub-steps:
[0060] S31: Build a fine-grained interaction module. This module is actually a multi-layer graph convolutional network that is used to complete the interaction between fine-grained features such as text or visual features and learn to update the representation of nodes. The network update formula is shown below:
[0061]
[0062] Where, is the activation function, represents the feature representation of all vertices, Indicates the Layer network. Residual connections are used to update features, improve the expressiveness of the model, and enhance the training stability of deep networks. At the same time, considering that the four interactive relationships will also change with node updates, the adjacency matrix A is also set as a trainable parameter. The adjacency matrix update formula is as follows:
[0063]
[0064] Where M is a (0, 1) matrix. If the (i, j) relationship is one of the four types of interaction relationships, then the value is 1, otherwise it is 0.
[0065] S32: Construct a global attention fusion module to complete feature fusion and obtain fused features. This module completes attention calculation using the following formula, and then completes weighted fusion, ultimately obtaining the fusion result of the input sequence. In this embodiment, for mentions or entities, four types of features are fused: initial text, initial image, text after relationship interaction, and image. The attention formula is as follows:
[0066]
[0067] in, , q is the query vector, k is the keyword vector, both are obtained by linear transformation of a single embedding representation. is the vector dimension.
[0068] The attention output is then processed into the final fused feature representation through a fully connected layer.
[0069] S33: Construct a similarity calculation module for completing similarity calculation. The similarity calculation operation is performed between different features of the mention and the candidate entity. In this embodiment, the cosine similarity function is used as the measurement function.
[0070] S34: Construct a two-stage entity linking module based on decision threshold to complete the final entity linking result.
[0071] After the model is built, the relevant model parameters are initialized. The parameter settings are based on previous research historical experience, including specific parameters such as learning rate lr, number of training rounds epoch, batch size batch_size, etc.
[0072] In this embodiment, step S4 includes the following sub-steps:
[0073] S41: Initialize the number of training rounds for model training;
[0074] S42: Extract batches from the input features. Each set of data includes the textual features and visual features of the mentions and the textual features and visual features of the corresponding candidate entities. All features are constructed into a sampling subgraph. The vertices in the subgraph are different features, and the edges are the interactive relationships between the features of each modality. Specifically, the subgraph includes the relationship between the mentioned text and the entity text, the relationship between the mentioned image and the entity text, the relationship between the mentioned text and the entity image, and the relationship between the mentioned image and the entity image.
[0075] In this embodiment, the four types of edges are constructed as follows:
[0076] S421: Use the BERT model to extract text features of mentions and entities respectively, and then calculate feature similarity as text-to-text edges;
[0077] S422: Use the CLIP model to calculate the similarity between text and image as the initial edge weight of the text-to-image relationship;
[0078] S423: CLIP model is also used to calculate the similarity between image and text as the initial edge weight of the image-to-text relationship;
[0079] S424: Select the Faster R-CNN model to extract the top k visual detection targets in the two images, then calculate the weighted similarity of the detected targets in the two images, and use this similarity value to initialize the image-to-image relationship.
[0080] The feature representation is taken as a vertex set and the four types of relationships are taken as edge sets to form a sampling subgraph.
[0081] S43: Input the sampled subgraph into the fine-grained relational interaction network module of the model to complete relational interaction propagation. For example, the mentioned text features receive the text and image features from the corresponding candidate entities during the interaction to obtain the features after the relational interaction.
[0082] S44: Perform dimension expansion and broadcast replication on the mention feature in the interactive output feature to make it consistent with the dimension of the candidate entity feature;
[0083] S45: Calculate the similarity between the mention in the interactive output feature and all candidate entity features, and get 4 groups of similarities, which are 、 、 and ;
[0084] S46: Input the original text features, original image features, text output features after interaction, and image output features mentioned into the global attention fusion module to learn the importance of cross-modal features and obtain fused features;
[0085] S47: Perform the same operation as S46 on the candidate entity to obtain the candidate entity fusion feature;
[0086] S48: Perform dimension expansion broadcast replication on the mentioned fusion features to make them consistent with the candidate entity feature dimensions;
[0087] S49: Calculate the similarity between the mention and the fusion features of each candidate entity to obtain the fusion feature similarity.
[0088] Then, this embodiment S5 includes the following sub-steps:
[0089] S51: comparing the similarity between the fused features of the mention and all candidate entities, and calculating the difference between the highest and second highest fused feature similarities;
[0090] S52: If the difference is greater than the threshold , then take the candidate entity with the highest similarity as the result and jump directly to S54, otherwise go to S53;
[0091] S53: Compare the similarities of the four groups of interactive features and take the candidate entity with the largest sum of similarities as the entity linking result;
[0092] S54: Compare the entity linking result with the corresponding correct answer to complete the model loss calculation. In this embodiment, the loss adopts the marginal ranking loss, and it is expected that the similarity of the positive sample is as large as possible and the similarity of the negative sample is as small as possible. The loss calculation formula is as follows:
[0093]
[0094] Where λ is a marginal hyperparameter that controls the looseness of the sorting.
[0095] S55: Optimize and update the parameters of the knowledge graph entity linking model based on the model loss, and increase the current number of training rounds by 1.
[0096] In step S6 of the embodiment, if the number of training rounds reaches a preset number of rounds, the training is completed and the trained model is saved.
[0097] In step S7 of the embodiment, the test data is input into the trained model, and the one with the greatest similarity is selected as the final link result according to the link strategy.
[0098] The graph entity linking method of the graph neural network and dynamic relationship interaction particularly considers the importance of different scale features in the entity linking process. By adopting multi-scale relationship processing and utilizing multimodal features, it effectively improves the knowledge graph entity linking effect and improves its application in scenarios with high reliability requirements and high error risks.
[0099] The above disclosure is merely one or more preferred embodiments of the present application and is not intended to limit the scope of the present application. A person skilled in the art will understand that all or part of the processes of the above embodiments and equivalent changes made in accordance with the claims of the present application are still within the scope of the present application.
Claims
1. A graph entity linking method based on graph neural network and dynamic relationship interaction, characterized in that: The following steps are involved: Construct a training dataset for completing the multimodal knowledge graph entity linking task; Extract multimodal features of mentions and entities in the training dataset to obtain input features; Construct a multimodal entity linking model based on multi-scale dynamic relationship interaction and initialize the model parameters; Input the input features into the network to obtain multi-scale output features; Update model parameters based on loss function optimization; Repeat the steps of inputting input features into the network, obtaining multi-scale output features, and optimizing and updating model parameters according to the loss function until the specified round is reached; The test data is input into the trained model, and the one with the greatest similarity is selected as the final link result according to the link strategy.
2. The graph entity linking method for graph neural network and dynamic relationship interaction according to claim 1 is characterized in that: Constructing a multimodal entity linking model based on multi-scale dynamic relationship interaction and initializing model parameters, the steps also include: The entity linking model includes a fine-grained relationship interaction module, a global attention fusion module, a feedforward neural network module, a Softmax normalization module, a similarity calculation module and a two-stage entity linking module based on a decision threshold; The fine-grained relationship interaction module is used to complete the relationship interaction between fine-grained features such as text or visual features, and extract the feature representation after feature interaction; The global attention fusion module is used to complete feature fusion and obtain fusion features; The feedforward neural network module is used to complete the feature space transformation of the hidden layer features; The Softmax normalization module is used to normalize the hidden layer output; The similarity calculation module is used to calculate the similarity between output features of different scales; The two-stage entity linking module based on decision threshold is used to complete the final entity linking task goal.
3. The graph entity linking method for graph neural network and dynamic relationship interaction according to claim 1 is characterized in that: Extracting multimodal features of mentions and entities in the training dataset to obtain input features, the step further includes: For text data, we use the pre-trained BERT model to extract the corresponding features. The model can be obtained from the open source address to obtain the mentioned text feature representation. and text feature representation of candidate entities ; For image data, the ResNet network is used to extract features from the image. The model can be obtained from the open source address to obtain the mentioned visual feature representation and visual feature representation of candidate entities .
4. The graph entity linking method for graph neural network and dynamic relationship interaction according to claim 1 is characterized in that: Inputting the input features into the network to obtain multi-scale output features, the steps further include: Initialize the number of training rounds for model training; Extract batches from the input features. Each set of data includes the textual and visual features of the mentions and the textual and visual features of the corresponding candidate entities. All features are then constructed into a sampling subgraph. The vertices in the subgraph are different features, and the edges are the interactions between the features of each modality. Specifically, the subgraph includes the relationship between the mentioned text and the entity text, the relationship between the mentioned image and the entity text, the relationship between the mentioned text and the entity image, and the relationship between the mentioned image and the entity image. The sampled subgraph is input into the fine-grained relational interaction network module of the model to complete the relational interaction propagation. For example, the text features of the mentions receive the text and image features from the corresponding candidate entities during the interaction to obtain the features after the relational interaction. Perform dimension expansion and broadcast replication on the mention features in the interactive output features to make them consistent with the candidate entity feature dimensions; Calculate the similarity between the mentions in the interactive output features and all candidate entity features, and get 4 sets of similarities, which are 、 、 and ; The original text features, original image features, text output features after interaction, and image output features mentioned are input into the global attention fusion module to learn the importance of cross-modal features and obtain fused features; For candidate entities, the original text features, original image features, text output features after interaction, and image output features are input into the global attention fusion module to learn the importance of cross-modal features and obtain the candidate entity fusion features; Perform dimension expansion and broadcast replication on the mentioned fusion features to make them consistent with the candidate entity feature dimensions; The similarity between the mention and the fused features of each candidate entity is calculated to obtain the fused feature similarity.
5. The graph entity linking method for graph neural network and dynamic relationship interaction according to claim 1 is characterized in that: Optimizing and updating the model parameters according to the loss function, the steps further include: Compare the fusion feature similarity between the mention and all candidate entities, and calculate the difference between the highest and second highest fusion feature similarity values; If the difference is greater than the threshold , then take the candidate entity with the highest similarity as the result, then compare the entity link result with the corresponding correct answer to complete the model loss calculation; If the difference is greater than or equal to the threshold , then compare the similarities of the four groups of interactive features and take the candidate entity with the largest sum of similarities as the entity linking result; Optimize and update the parameters of the knowledge graph entity linking model based on the model loss, and increase the current number of training rounds by 1.