Multi-modal knowledge graph entity alignment method based on combination of information screening and LLM fine tuning

Abstract

The invention discloses a multi-modal knowledge graph entity alignment method based on information screening in combination with LLM fine tuning, and belongs to the technical field of knowledge graph reasoning. An existing method has defects in learning different types of information between entities and neglects the negative influence of complementary information between the entities in different knowledge maps on entity alignment, and the method comprises the following steps: firstly, providing an information screening learning framework based on text information types; and learning a semantic information relation between different text information types in the knowledge graph. Then, a prompt for a multi-modal knowledge graph entity alignment task is designed, and an entity alignment result based on representation learning is optimized. Experimental results on the multi-modal knowledge graph data sets FB15K-DB15K and FB15K-YAGO15K show that the multi-modal knowledge graph entity alignment method based on the combination of information screening and LLM fine tuning is effective.

Description

Technical Field

[0001] The present invention relates to a multimodal knowledge graph entity alignment method based on information screening combined with LLM fine-tuning, and belongs to the technical field of knowledge graph reasoning. Background Art

[0002] Knowledge graph (KG), as a structured knowledge representation method, is widely used in recommendation systems, intelligent question answering, semantic search and other fields. The basic form of KG is usually composed of triples (h, r, t), in which entities are connected by relationships. However, traditional knowledge graphs often face the problem of incompleteness, which leads to certain limitations in practical applications. With the development of multimodal technology, the traditional text-dominated KG has gradually expanded to multimodal knowledge graphs (MMKG), which enhance the ability to express the complex real world by introducing multiple information sources such as images. At present, common multimodal knowledge graphs include cross-language knowledge graphs (such as DBP15K) and multimodal knowledge graphs (such as MMKG). Despite this, a single knowledge graph is still difficult to meet all application requirements, especially when dealing with the integration of cross-domain knowledge. In order to solve these problems, entity alignment technology is widely used in the fusion of knowledge graphs. Especially in multimodal environments, how to effectively align information of different modalities has become an important research direction.

[0003] Liu et al., Lin et al., Chen et al. use GNN to encode the structure of entities in multimodal knowledge graphs, encode images and texts with pre-trained models, and finally calculate the similarity between entities by combining modal fusion. Recent work on entity alignment in multimodal knowledge graphs has begun to focus on the comprehensive use of the encoding capabilities of embedding-based methods and the semantic understanding and reasoning capabilities of LLM. For example, Zhang et al. and Chen et al. use LLM's rich knowledge to enhance information for tasks, and Zhang et al. use LLM to construct a predicate-adjacency matrix to achieve relationship alignment. Chen et al. use LLM to annotate entities for unsupervised entity alignment. Jiang et al. first use Simple-HHEA to generate alignment candidates, and then use LLM's reasoning ability to score the similarity between each candidate entity and the target entity based on different information, and finally combine all scores to evaluate the similarity between the candidate entity and the target entity. Yang et al. modeled the EA task as a binary classification task, finding alignment entities by reasoning whether two entities from different knowledge graphs are the same entity. At present, there are still some shortcomings in the multimodal knowledge graph alignment method: 1. It does not consider the role of complementary information in entity alignment, that is, it does not consider that the overall semantic similarity of aligned entities will decrease when complementary information is richer than similar information. 2. It does not consider the importance of information similarity between the modalities of aligned entities, and cannot fully explore the contribution relationship of modal information to entity alignment.

[0004] In response to the above problems, the present invention proposes a multimodal knowledge graph entity alignment method based on information screening combined with LLM fine-tuning, which aims to filter out attribute types and relationship type information that are not important to the task, and optimize the entity alignment results by fine-tuning LLM. First, an information screening strategy based on the semantic similarity of attribute types and relationship types is proposed. The similarity of information types between graphs is calculated by semantically representing attribute types and relationship types, and information types that appear in both knowledge graphs or have strong semantic similarity are retained, and weakly associated complementary information types are pruned to achieve the screening of complementary information. Finally, a training set is constructed by negatively sampling the candidate entity set and disrupting the order of the candidate entities, and the candidate entities are segmented and multi-round inference is performed to minimize the length of the input text while finding the aligned entities that are ranked lower from the candidate entities.

[0005] Based on the above analysis, this paper proposes a multimodal knowledge graph entity alignment method based on information screening combined with LLM fine-tuning. Experiments were conducted on the public multimodal knowledge graph entity alignment dataset (MMKG), and the experimental results show the effectiveness of the proposed model. Summary of the invention

[0006] In order to solve the above-mentioned problems, the present invention provides a multimodal knowledge graph entity alignment method based on information screening combined with LLM fine-tuning.

[0007] The technical solution of the present invention is: a multimodal knowledge graph entity alignment method based on information screening combined with LLM fine-tuning, the method comprising:

[0008] Step 1: Use Roberta to embed the relationship types and attribute types in the two knowledge graphs, calculate the embedding scores of the relationship types and attribute types between the knowledge graphs, and use Qwen2-vl to convert the pictures of the entities in the two knowledge graphs into picture description captions;

[0009] Step 2: Use the existing embedding-based entity alignment method to train on the two knowledge graphs, learn the entity representations on the two knowledge graphs, and send the entities, relations, and pictures of the knowledge graphs to the trained model to obtain the total semantic embedding representation matrix of the entities;

[0010] Step 3: Calculate the similarity between entities based on the semantic embedding representation of the entities to obtain a ranked candidate entity set, and perform negative sampling on the candidate entity set;

[0011] Step 4: Obtain the semantic embedding representation similarity matrix of knowledge graph attributes and relationships from Step 1, summarize the similarity scores from the perspectives of the two knowledge graphs, and filter the attributes and relationships according to the similarity scores to obtain a new set of attribute types and relationship types;

[0012] Step 5. Design a prompt template according to the task, obtain the candidate entity set of the entity from Step 3 and the attribute type and relationship type set obtained from Step 4, and put the information of the target entity and candidate entity into the template.

[0013] Step 6. Fine-tune GLM4 with LoRA using the candidate entity sets of positive and negative sampling.

[0014] Furthermore, the Step 1 specifically includes the following:

[0015] Step 1.1, send the attribute type and relationship type sets A and R into the fine-tuned BERT to obtain the total semantic embedding representation matrix of entities and relationships:

[0016]

[0017] Step 1.2: Send the entity image I to the fine-tuned CLIP to obtain the total semantic embedding representation matrix of entities and relations:

[0018] caption ent = CLIP(I ent ) (3)

[0019] Step 1.3. Calculate the embedding scores between attribute types within an attribute type and between relationship types within a relationship type according to the following formula:

[0020]

[0021] in and Represents the source knowledge graph attribute type set A respectively s The i-th element in and the target knowledge graph attribute type set A t The jth element in ; and Represents the source knowledge graph attribute type set R s The i-th element in and the target knowledge graph attribute type set R t The jth element in I ent A picture representing the entity ent, caption ent Indicates I ent The function f(·) is the cosine similarity, and D(·) represents the distance score between samples.

[0022] Furthermore, the Step 2 specifically includes the following:

[0023] Step 2.1. Train on two knowledge graphs according to the existing embedding-based entity alignment method and parameters, divide the data into a training set Train and a test set Test, and use the training set Train to perform supervised training based on the existing method to obtain the trained model Premodal-EA.

[0024] Step 2.2, send the entity's attribute triplet AT, relationship triplet RT and image I into the trained model Premodal-EA to obtain the entity's semantic embedding representation matrix:

[0025] ent_emb=Premodal-EA(AT,RT,I) (6)

[0026] AT is represented as (ent, A, value), where ent is the entity, A is the attribute type, and value is the attribute value. RT is represented as (head_ent, R, tail_ent), where head_ent and tail_ent represent the head entity and tail entity of the relationship respectively, and R represents the type of the relationship. ent_emb represents the semantic embedding representation of the entity ent.

[0027] Furthermore, the Step 3 specifically includes the following:

[0028] Cross-domain similarity local scaling (CSLS) is used to calculate the similarity between entities:

[0029]

[0030] in Represent the i-th entity of the source knowledge graph and the j-th entity of the target knowledge graph respectively; Represent the semantic embedding representation of the i-th entity in the source knowledge graph and the target knowledge respectively

[0031] The semantic embedding representation of the jth entity in the graph. The function D(·) represents the similarity score between two samples.

[0032] Furthermore, the Step 4 specifically includes the following:

[0033] Step 4.1: Summarize the similarity scores of information between knowledge graphs from the perspectives of attribute type and relationship type:

[0034]

[0035] Step 4.2: Filter out attribute types and relationship types whose similarity scores are lower than the threshold from attribute types and relationship types to form a new attribute type set and relationship type set:

[0036]

[0037] in They represent the mth attribute type in the source knowledge graph and the nth attribute type in the target knowledge graph respectively. Represent the mth relationship type in the source knowledge graph and the nth relationship type in the target knowledge graph respectively. Th , R Th They represent the threshold of attribute type and the threshold of relationship type respectively; the function sim(a,B) represents the calculation of cosine similarity between vector a and each row vector in matrix B. Set=add(X) represents adding element X to Set.

[0038] Furthermore, the Step 5 specifically includes the following:

[0039] Step 5.1. Design a prompt template for the entity alignment task. For the one-to-one alignment of entities in entity alignment, the task is modeled as a multiple-choice question. For an entity and entity information, a candidate entity set and its information are given, and the aligned entity is inferred through question-answering. In order to avoid hallucinations caused by the absence of aligned entities in the candidate entity set, an entity id "999999" that does not exist in the knowledge graph is added to the options as an outlier.

[0040] Furthermore, the Step 6 specifically includes the following:

[0041] Step 6.1, when constructing the training set for fine-tuning LoRA, in order to alleviate the excessive learning of LLM's position information of aligned entities in candidate entities during fine-tuning, the candidate entity set is shuffled in order.

[0042] Step 6.2. At the same time, during fine-tuning, the aligned entities are randomly replaced for the same candidate entity, and this candidate entity is used as a negative sample to improve the generalization ability of the method.

[0043] The present invention also provides a multimodal knowledge graph entity alignment system based on information screening combined with LLM fine-tuning, including: a module for executing the multimodal knowledge graph entity alignment method based on information screening combined with LLM fine-tuning.

[0044] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the multimodal knowledge graph entity alignment method based on information screening combined with LLM fine-tuning is implemented.

[0045] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the multimodal knowledge graph entity alignment method based on information screening combined with LLM fine-tuning is implemented.

[0046] The beneficial effects of the present invention are:

[0047] 1. The present invention aims to filter out the attribute type and relationship type information that are not important to the task, and optimize the entity alignment results by fine-tuning LLM.

[0048] 2. The present invention processes the information of each modality of the entity using a pre-trained model, obtains a threshold based on the similarity of the semantic representation of the attribute type and the relationship type, and retains the attribute types and relationship types that are greater than the threshold; then, a candidate set of the entity is obtained through an embedding-based method; finally, a negative sampling is performed on the candidate set of the entity to construct a training set, and an entity alignment task is constructed to prompt the alignment task to be modeled as a multiple-choice task to evaluate the similarity between the candidate entity and the target entity based on the overall information of the entity.

[0049] 3. The present invention solves to a certain extent the problem that the overall semantic similarity of aligned entities will be reduced when complementary information is richer than similar information, which was not considered in previous related research work; after filtering out the information with weak correlation between graphs, fine-tuning LLM can comprehensively evaluate the similarity between entities through semantic understanding; the experimental results of the comparison model show that the method of the present invention can improve the performance of multimodal knowledge graph entity alignment to a certain extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 This is the framework of the multimodal knowledge graph entity alignment method based on information screening combined with LLM fine-tuning in the present invention. DETAILED DESCRIPTION

[0051] Example 1: The present invention provides a multimodal knowledge graph entity alignment method based on information screening combined with LLM fine-tuning. The following method proposed in this implementation is implemented on the multimodal knowledge graph entity alignment dataset MMKG;

[0052] The present invention uses the public multimodal knowledge graph entity alignment dataset MMKG to conduct experiments. MMKG contains three multimodal knowledge graphs FB15K, DB15K, and YG15K, which are divided into two aligned knowledge graphs FBDB15K and FBYG15K. FB15K is a subset extracted from Freebase, containing 14,951 entities and 1,345 relationship types, with a total of 592,213 triples. DB15K is a dataset aligned with FB15K based on DBpedia, containing 14,777 entities and 279 relationship types, with a total of 99,028 triples. YG15K is a dataset aligned with FB15K based on YAGO, containing 15,404 entities and 32 relationship types, with a total of 122,886 triples. The statistical information of the dataset is shown in Table 1:

[0053] Table 1 Dataset statistics

[0054]

[0055] like Figure 1 As shown, a multimodal knowledge graph entity alignment method based on information screening combined with LLM fine-tuning includes:

[0056] Step 1. Use Roberta to embed the relationship types and attribute types in the two knowledge graphs, calculate the embedding scores of the relationship types and attribute types between the knowledge graphs, and use Qwen2-vl to convert the images of the entities in the two knowledge graphs into image description captions:

[0057] Step 1.1, send the attribute type and relationship type sets A and R into the fine-tuned BERT to obtain the total semantic embedding representation matrix of entities and relationships:

[0058]

[0059] Step 1.2: Send the entity image I to the fine-tuned CLIP to obtain the total semantic embedding representation matrix of entities and relations:

[0060] caption ent = CLIP(I ent ) (3)

[0061] Step 1.3. Calculate the embedding scores between attribute types within an attribute type and between relationship types within a relationship type according to the following formula:

[0062]

[0063] in and Represents the source knowledge graph attribute type set A respectively s The i-th element in and the target knowledge graph attribute type set A t The jth element in ; and Represents the source knowledge graph attribute type set R s The i-th element in and the target knowledge graph attribute type set R t The jth element in I ent A picture representing the entity ent, caption ent Indicates I ent The function f(·) is the cosine similarity, and D(·) represents the distance score between samples.

[0064] Step 2: Use the existing embedding-based entity alignment method to train on two knowledge graphs, learn the entity representations on the two knowledge graphs, and send the entities, relations, and images of the knowledge graphs into the trained model to obtain the total semantic embedding representation matrix of the entity:

[0065] Step 2.1. Train on two knowledge graphs according to the existing embedding-based entity alignment method and parameters, divide the data into a training set Train and a test set Test, and use the training set Train to perform supervised training based on the existing method to obtain the trained model Premodal-EA.

[0066] Step 2.2, send the entity's attribute triplet AT, relationship triplet RT and image I into the trained model Premodal-EA to obtain the entity's semantic embedding representation matrix:

[0067] ent_emb=Premodal-EA(AT,RT,I) (6)

[0068] AT is represented as (ent, A, value), where ent is the entity, A is the attribute type, and value is the attribute value. RT is represented as (head_ent, R, tail_ent), where head_ent and tail_ent represent the head entity and tail entity of the relationship respectively, and R represents the type of the relationship. ent_emb represents the semantic embedding representation of the entity ent.

[0069] Step 3: Calculate the similarity between entities based on the semantic embedding representation of the entities to obtain a ranked set of candidate entities, and perform negative sampling on the candidate entity set:

[0070] Step 3.1, use cross-domain similarity local scaling (CSLS) to calculate the similarity between entities:

[0071]

[0072] in Represent the i-th entity of the source knowledge graph and the j-th entity of the target knowledge graph respectively; They represent the semantic embedding representation of the i-th entity in the source knowledge graph and the semantic embedding representation of the j-th entity in the target knowledge graph respectively. The function D(·) represents the similarity score between two samples.

[0073] Step 4: Obtain the semantic embedding representation similarity matrix of knowledge graph attributes and relationships from Step 1, summarize the similarity scores from the perspectives of the two knowledge graphs, and filter the attributes and relationships according to the similarity scores to obtain a new set of attribute types and relationship types:

[0074] Step 4.1: Summarize the similarity scores of information between knowledge graphs from the perspectives of attribute type and relationship type:

[0075]

[0076] Step 4.2: Filter out attribute types and relationship types whose similarity scores are lower than the threshold from attribute types and relationship types to form a new attribute type set and relationship type set:

[0077]

[0078] in They represent the mth attribute type in the source knowledge graph and the nth attribute type in the target knowledge graph respectively. Represent the mth relationship type in the source knowledge graph and the nth relationship type in the target knowledge graph respectively. Th , R Th They represent the threshold of attribute type and the threshold of relationship type respectively; the function sim(a,B) represents the calculation of cosine similarity between vector a and each row vector in matrix B. Set=add(X) represents adding element X to Set.

[0079] Step 5. Design a prompt template according to the task, obtain the candidate entity set of the entity from Step 3 and the attribute type and relationship type set obtained from Step 4, and put the information of the target entity and candidate entity into the template:

[0080] A prompt template is designed for the entity alignment task. The task is modeled as a multiple-choice question for the one-to-one alignment of entities in entity alignment. For an entity and entity information, a candidate entity set and its information are given, and the aligned entity is inferred through question-answering. In order to avoid hallucinations caused by the absence of aligned entities in the candidate entity set, an entity id "999999" that does not exist in the knowledge graph is added to the options as an outlier. The prompt template can be shown in Table 2:

[0081] Table 2 is a prompt template

[0082]

[0083] The content in Table 2 is translated as follows:

[0084] Task: Perform entity alignment by determining if there are aligned entities between candidate entities for a given target entity. When evaluating similarity, focus on information with similar attributes / relationship types, ignore information with different types, and focus on content without "no image" in the title. If the similarity of the type-similar information of two entities is high, they will be aligned. If aligned entities are found, output their identifiers. If no aligned entities are found, return 999999. The output should only include the alignment results.

[0085] Input description: Each entity is represented by attributes and relationship information.

[0086] Attribute information: Format: entity id: [(attribute type, attribute value)]

[0087] Example: 152:[('Birthdate','1999-10-23')] means that entity 152 has a 'Birthdate' attribute whose value is 'October 23, 1999'.

[0088] Relationship information: The format is entity id:{relationship type: number of adjacent entities}.

[0089] Example: 152:{'residence':4} means that entity 152 has 4 neighboring entities associated with the relationship type 'residence'.

[0090] Step 6. Fine-tune GLM4 with LoRA using the candidate entity set of positive and negative sampling:

[0091] Step 6.1. When constructing the training set for fine-tuning LoRA, in order to alleviate the excessive learning of the position information of the aligned entities in the candidate entities during fine-tuning, the candidate entity set is shuffled in order.

[0092] Step 6.2. At the same time, during fine-tuning, the aligned entities are randomly replaced for the same candidate entity, and this candidate entity is used as a negative sample to improve the generalization ability of the method.

[0093] The present invention also provides a multimodal knowledge graph entity alignment system based on information screening combined with LLM fine-tuning, including: a module for executing the multimodal knowledge graph entity alignment method based on information screening combined with LLM fine-tuning.

[0094] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the multimodal knowledge graph entity alignment method based on information screening combined with LLM fine-tuning is implemented.

[0095] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the multimodal knowledge graph entity alignment method based on information screening combined with LLM fine-tuning is implemented.

[0096] In order to verify the effectiveness of the model proposed in this invention, the method of this invention is compared with the following baseline methods of different labeling frameworks, as follows:

[0097] EVA adopts graph neural network (GNN) to capture the semantic relationship between entities and relations, and discovers aligned entities by gradually adding their neighbor entities to the candidate set during training.

[0098] MSNEA adopts TransE to capture the embedding representation of entities and relations, and uses images as nodes to guide the learning of relations and attributes.

[0099] MCLEA adopts Graph Attention Network (GAT) to learn the semantic relationship of entities, uses contrastive learning to align the unimodal representations of entities, and increases the interaction between modalities through KL (Kullback-Leibler Divergence).

[0100] MEAformer uses the attention mechanism to learn the weights between the various modalities of an entity to achieve entity-level modality fusion in order to obtain a fine-grained fusion representation of the entity.

[0101] PMF-MMEA adopts progressive freezing of features irrelevant to entity alignment to make full use of features in entities that are beneficial to alignment.

[0102] The proposed method is effectively compared with the baseline model on the FBDB15K and FBYG15K datasets of MMKG, and the results are shown in Table 3. (1) In general, the proposed method achieves the best results in almost all indicators on the two aligned knowledge graphs, which illustrates the effectiveness of the proposed method.

[0103] Table 3 shows the experimental results on FBDB15K and FBYG15K data.

[0104]

[0105] In order to verify the effectiveness of each part of the knowledge graph reasoning method proposed in this invention, the following ablation experiment is designed for comparison. The experimental results are shown in Table 4:

[0106] w / o Fliter: remove the information filtering module separately from the model of the present invention.

[0107] w / o neg: remove negative sampling of candidate entities separately from the model of the present invention.

[0108] Table 4 shows the ablation experiment results.

[0109]

[0110] The ablation experiment results show that removing the information screening module and directly using all the entity information will cause the performance to drop by about 2%. Removing negative sampling alone The above analysis shows the effectiveness of the method proposed in the present invention.

[0111] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A multimodal knowledge graph entity alignment method based on information screening combined with LLM fine-tuning, characterized by: The method comprises: Step 1: Use the Roberta model to embed the relationship types and attribute types in the two knowledge graphs, calculate the embedding scores of the relationship types and attribute types between the knowledge graphs, and use the Qwen2-vl model to convert the pictures of the entities in the two knowledge graphs into picture description captions; Step 2: Use the existing embedding-based entity alignment method to train on the two knowledge graphs, learn the entity representations on the two knowledge graphs, and send the entities, relations, and pictures of the knowledge graphs into the trained model to obtain the overall semantic embedding representation matrix of the entities; Step 3: Calculate the similarity between entities based on the semantic embedding representation of the entities to obtain a ranked candidate entity set, and perform negative sampling on the candidate entity set; Step 4: Obtain the semantic embedding representation similarity matrix of the knowledge graph attribute type and relationship type from Step 1, summarize the similarity scores from the perspectives of the two knowledge graphs, and filter the attributes and relationships according to the similarity scores to obtain a new set of attribute types and relationship types; Step 5. Design a prompt template according to the task, obtain the candidate entity set of the entity from Step 3 and the attribute type and relationship type set obtained from Step 4, and put the information of the target entity and candidate entity into the template; Step 6. Fine-tune GLM4 with LoRA using the candidate entity sets of positive and negative sampling.

2. According to claim 1, the multimodal knowledge graph entity alignment method based on information screening combined with LLM fine-tuning is characterized by: The Step 1 specifically includes the following: Step 1.1, send the attribute type and relationship type sets A and R into the fine-tuned BERT to obtain the total semantic embedding representation matrix of entities and relationships: Step 1.2: Send the entity image I to the fine-tuned CLIP to obtain the total semantic embedding representation matrix of entities and relations: caption ent =CLIP(I ent ) (3) Step 1.

3. Calculate the embedding scores between attribute types within an attribute type and between relationship types within a relationship type according to the following formula: in and Represents the source knowledge graph attribute type set A respectively s The i-th element in and the target knowledge graph attribute type set A t The jth element in ; and Represents the source knowledge graph attribute type set R s The i-th element in and the target knowledge graph attribute type set R 2 The jth element in I ent A picture representing the entity ent, caption ent Indicates I ent , the function f(·) is the cosine similarity, and D(·) represents the distance score between samples.

3. The multimodal knowledge graph entity alignment method based on information screening combined with LLM fine-tuning according to claim 1 is characterized by: The Step 2 specifically includes the following: Step 2.

1. Train on two knowledge graphs according to the existing embedding-based entity alignment method and parameters, divide the data into a training set Train and a test set Test, and use the training set Train to perform supervised training on the existing method to obtain the trained model Premodal-EA; Step 2.2, send the entity's attribute triplet AT, relationship triplet RT and image I into the trained model Premodal-EA to obtain the entity's semantic embedding representation matrix: ent_emb=Premodal-EA(AT,RT,I) (6) AT is expressed as (ent, A, value), where ent is the entity, A is the attribute type, and value is the attribute value; RT is expressed as (head_ent, R, tail_ent), where head_ent and tail_ent represent the head entity and tail entity of the relationship respectively, and R represents the type of the relationship; ent_emb represents the semantic embedding representation of the entity ent.

4. The multimodal knowledge graph entity alignment method based on information screening combined with LLM fine-tuning according to claim 1 is characterized by: In Step 3, cross-domain similarity local scaling (CSLS) is used to calculate the similarity between entities: in Represent the i-th entity of the source knowledge graph and the j-th entity of the target knowledge graph respectively; They represent the semantic embedding representation of the i-th entity in the source knowledge graph and the semantic embedding representation of the j-th entity in the target knowledge graph, respectively, and the function D(·) represents the similarity score between the two samples.

5. The multimodal knowledge graph entity alignment method based on information screening combined with LLM fine-tuning according to claim 1 is characterized by: In Step 4: Step 4.1: Summarize the similarity scores of information between knowledge graphs from the perspectives of attribute type and relationship type: Step 4.2: Filter out attribute types and relationship types whose similarity scores are lower than the threshold from attribute types and relationship types to form a new attribute type set and relationship type set: in They represent the mth attribute type in the source knowledge graph and the nth attribute type in the target knowledge graph, respectively. Represents the mth relation type in the source knowledge graph and the nth relation type in the target knowledge graph, respectively. Th , R Th Respectively represent the threshold of attribute type and the threshold of relationship type; the function sim(a,B) represents the calculation of cosine similarity between vector a and each row vector in matrix B, Set X =add(X) means adding element X to the Set.

6. The multimodal knowledge graph entity alignment method based on information screening combined with LLM fine-tuning according to claim 1 is characterized by: The Step 5 specifically includes the following: A prompt template is designed for the entity alignment task. The task is modeled as a multiple-choice question for the one-to-one situation of aligned entities in entity alignment. For an entity and entity information, a candidate entity set and its information are given, and the aligned entity is inferred in the form of question and answer. In order to avoid the illusion caused by the absence of aligned entities in the candidate entity set, an entity id "999999" that does not exist in the knowledge graph is added to the options as an outlier.

7. The multimodal knowledge graph entity alignment method based on information screening combined with LLM fine-tuning according to claim 1 is characterized by: The Step 6 specifically includes the following: Step 6.1, when constructing the training set for fine-tuning LoRA, in order to alleviate the excessive learning of the position information of the aligned entities in the candidate entities during fine-tuning, the candidate entity set is shuffled in order; Step 6.

2. At the same time, during fine-tuning, the aligned entities are randomly replaced for the same candidate entity, and this candidate entity is used as a negative sample to improve the generalization ability of the method.

8. A multimodal knowledge graph entity alignment system based on information screening combined with LLM fine-tuning, characterized in that: include: A module for executing a multimodal knowledge graph entity alignment method based on information screening combined with LLM fine-tuning as described in any one of claims 1 to 7.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, it implements a multimodal knowledge graph entity alignment method based on information screening combined with LLM fine-tuning as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements a multimodal knowledge graph entity alignment method based on information screening combined with LLM fine-tuning as described in any one of claims 1 to 7.

Images