Entity alignment method based on cross-modal interaction and large language model multi-level voting

Abstract

The application discloses an entity alignment method based on cross-modal interaction and large language model multi-level voting, which unifies relations, attributes and images into the same token embedding sequence in a text space, realizes deep interaction of multi-modal semantics by using a global semantic fusion path and a local cross-modal enhancement path, and optimizes entity vector representation by using contrast learning; meanwhile, the large language model is used to perform multi-level reasoning and voting on a candidate entity set to obtain a final entity alignment result. The application can effectively improve the accuracy of multi-modal knowledge graph entity alignment, solves the problems of mode fragmentation, interaction absence and low alignment accuracy of the existing multi-modal knowledge graph entity alignment method, and has important significance for the development of cross-language knowledge fusion, multi-modal recommendation, intelligent question answering and the like.

Description

Technical Field

[0001] This invention belongs to the field of natural language processing technology, and in particular relates to an entity alignment method based on cross-modal interaction and multi-level voting of a large language model. Background Technology

[0002] With the explosive growth of the Web, social media, and visual content, the same real-world entity is often scattered across multilingual, multi-domain, and multimodal knowledge graphs. Entity alignment aims to automatically identify semantically consistent entities across graphs and modalities, and is a fundamental technology for knowledge fusion, intelligent question answering, and recommendation systems. However, traditional entity alignment research faces three major challenges. First, there is a fragmentation of the heterogeneous semantic space. Textual and visual information are distributed in discrete symbolic spaces and continuous visual feature spaces, respectively. Feature concatenation or weighted fusion methods struggle to establish a unified trainable gradient propagation path between textual and visual information, thus limiting joint optimization of cross-modal features. Second, there is a lack of fine-grained interaction. Textual information is typically modeled using pooling or simple encoding, while visual information relies on pre-trained visual encoders for independent modeling. This makes it difficult to achieve deep semantic interaction between textual and visual information, resulting in insufficient modeling of local semantic associations. Finally, there is the illusion and position bias of large language models. Although large language models (such as GPT-4 and Qwen) perform excellently in text reasoning, when directly used for entity alignment, the lengthy candidate lists and lack of visual evidence trigger illusions and position biases. The error rate of single-round reasoning increases with the size of the candidate and cannot utilize external signals such as images for auxiliary alignment prediction.

[0003] Currently, to address the problems existing in traditional multimodal knowledge graph entity alignment, some research has attempted to improve the quality of entity vector representations by introducing multimodal information. Chinese invention patent CN116932770B discloses a "Multimodal Knowledge Graph Entity Alignment Method and System Based on Intermodal Interaction." This method first obtains multimodal data of the entities to be aligned from the knowledge graph, including structural information, visual information, relational information, and attribute information. It then fuses the features of each modality using a low-rank multimodal fusion method to model the interaction relationships between different modalities. Next, a cross-modal attention mechanism is introduced, enabling each single-modal feature to learn the intermodal interaction information from the fused multimodal features. These feature representations, enhanced by the cross-modal attention module, are then output through residual connections and normalization layers, followed by modal-weighted concatenation to generate the overall feature representation of the entity. To further optimize the entity vector representation, this method introduces a contrastive learning mechanism in the representation learning stage. By constructing positive and negative sample pairs, the feature representations of equivalent entities are brought closer together, while the feature representations of inequivalent entities are spaced further apart. Furthermore, this method weights negative samples within a batch, causing the model to focus more on difficult negative samples during training, thereby improving the model's discriminative ability. Finally, based on the learned overall entity feature representation, this method calculates the similarity between entities in the source and target knowledge graphs and selects the entity pair with the highest similarity as the final alignment result. However, this method has the following shortcomings: (1) In terms of text information modeling, the bag-of-words model is used to connect the feedforward network to encode relational information and attribute information respectively. Since the bag-of-words model only counts the frequency of word occurrence and ignores the contextual relationship between words, it is difficult to capture the deep semantic information in the text, thus limiting the expressive power of entity semantic representation.

[0004] (2) In terms of visual information modeling, this method uses the VGG16 convolutional neural network to extract image features. Although this method can obtain the visual representation of the image, its training objective is mainly for image classification tasks and lacks the ability to align image and text semantics. Therefore, it is difficult to establish an effective connection between visual features and text semantics, which limits the effect of multimodal information fusion.

[0005] (3) It mainly relies on low-rank multimodal fusion methods to fuse features of different modalities as a whole. Although it can model the interaction between modalities to a certain extent, it lacks fine-grained semantic interaction, so it is difficult to fully explore the complex semantic associations between different modalities.

[0006] (4) It relies entirely on representation learning models for entity matching without introducing the semantic reasoning capabilities of large language models. Therefore, it cannot take advantage of the semantic understanding and reasoning capabilities of large language models and it is difficult to further improve entity alignment performance in complex scenarios.

[0007] The aforementioned shortcomings essentially reflect the common limitations of current multimodal entity alignment schemes, such as inconsistent modal feature spaces, insufficient fine-grained semantic interaction, and a lack of high-order reasoning capabilities. Summary of the Invention

[0008] The present invention aims to solve the aforementioned technical problems existing in the prior art by providing an entity alignment method based on cross-modal interaction and multi-level voting of a large language model.

[0009] The technical solution of this invention is: an entity alignment method based on cross-modal interaction and multi-level voting of a large language model, which takes the source knowledge graph to be aligned and the target knowledge graph as input, and performs the following steps in sequence: Step 1. Multimodal feature extraction and word embedding sequence construction Extract relation triples and attribute triples of entities from the source and target knowledge graphs to be aligned, and perform textualization and word segmentation processing respectively to obtain relation word sequence and attribute word sequence; map the relation word sequence and attribute word sequence to a unified vector space to generate relation word embedding sequence and attribute word embedding sequence; obtain the entity association image, extract the visual features of the association image using a visual encoder, project the visual features to the same embedding dimension as the relation word embedding sequence and attribute word embedding sequence, and convert them into a fixed-length visual prefix word embedding sequence; Step 2. Generate multiple sets of entity vectors based on the trained dual-path cross-modal interaction model. The trained dual-path cross-modal interaction model processes relation lexical embedding sequences, attribute lexical embedding sequences, and visual prefix lexical embedding sequences, outputting four sets of entity vectors: trimodal joint vector representation, relation enhancement representation, attribute enhancement representation, and visual enhancement representation. Step 3. Multimodal feature fusion and multi-level voting entity alignment A comprehensive score matrix is ​​generated based on four sets of entity vectors. Candidate entity sets corresponding to the source entity are selected from the comprehensive score matrix. Then, in the order of attribute, relation, and vision, multi-round voting reasoning is carried out on the candidate entity sets. If the current level selects an aligned entity, the aligned entity is output as the alignment result of the source entity. For the source entity and its corresponding candidate entity set that do not produce an alignment result, the next level of voting reasoning is entered. After completing all levels of voting reasoning, the entity alignment result between the source knowledge graph and the target knowledge graph is obtained.

[0010] The preferred step 1 is as follows: Step 1.1 Concatenate the relation triples and attribute triples into relation text sequences and attribute text sequences respectively in lexicographical order. Then, use the BERT tokenizer to convert the relation text sequences and attribute text sequences into discrete tokens. After the embedding layer, map them into fixed-dimensional relation token embedding sequences and attribute token embedding sequences. Step 1.2 Use CLIP visual encoder to extract global visual features of entity association image, project the global visual features to the same embedding dimension as relation word embedding sequence and attribute word embedding sequence through multilayer perceptron, and convert them into visual prefix word embedding sequence of preset length.

[0011] The preferred method for constructing the trained dual-path cross-modal interaction model in step 2 is as follows: Step 2.1 Construct a dual-path cross-modal interaction model. This model includes a global semantic fusion path and a local cross-modal enhancement path set in parallel. The global semantic fusion path is equipped with a language model encoder. The local cross-modal enhancement path includes three sets of bidirectional interaction units. Each set of bidirectional interaction units includes two cross-attention modules with opposite directions, which are used to model the bidirectional semantic interaction between the two modalities. The output of each set of bidirectional interaction units is connected to the residual structure and the pooling layer. Step 2.2 Take the source knowledge graph and target knowledge graph in the training dataset, call steps 1.1 and 1.2 to obtain the relation word embedding sequence, attribute word embedding sequence and visual prefix word embedding sequence corresponding to the source graph and target graph respectively; Step 2.3 Global Semantic Fusion Path: The relation word embedding sequence, attribute word embedding sequence, and visual prefix word embedding sequence are concatenated and input into the language model encoder to achieve global semantic interaction between different modalities and output a trimodal joint vector representation. Step 2.4 Local Enhancement Path performs cross-attention interaction on each pair of relation lexical embedding sequence, attribute lexical embedding sequence, and visual prefix lexical embedding sequence. The interaction results are residually connected with the original lexical embedding sequence and pooled to output relation enhancement representation, attribute enhancement representation, and visual enhancement representation. Step 2.5 Based on the aligned seed pairs, bidirectional contrastive learning is performed on the three-modal joint vector representation, relation enhancement representation, attribute enhancement representation and visual enhancement representation respectively to obtain a four-branch contrastive loss. The four-branch contrastive loss is fused by dynamic weights to obtain the total loss. Combined with the labeled loss of entity seed pairs, backpropagation is performed to iteratively update the parameters of the dual-path cross-modal interaction model until the loss converges to a stable value, and the model training is completed.

[0012] The preferred step 3 is as follows: Step 3.1 Generate four similarity matrices by pairwise dot products of the three-modal joint vector representation, relation enhancement representation, attribute enhancement representation, and visual enhancement representation of the source knowledge graph and the target knowledge graph. Then, obtain a comprehensive score matrix by weighted summation using learnable weights. Based on the comprehensive score matrix, select a preset number of entities for each source entity to form a candidate entity set. Step 3.2 Calculate information gain based on the entity attributes of the source knowledge graph, and select several attributes with the highest information gain to construct key attribute triples; input the key attribute triples and candidate entity attribute triples into the large language model simultaneously, and conduct multiple rounds of voting reasoning on the candidate entity set; before each round of voting reasoning, randomly rearrange the input order of the candidate entity set for this round, and the large language model selects a candidate entity from the candidate entity set. After completing multiple rounds of voting, count the cumulative number of votes for the selected candidate entity in each round. If there is a candidate entity with the highest number of votes and not less than half, it is determined as the alignment result of the source entity; for source entities that have not produced an alignment result at this level, they, along with the corresponding candidate entity set, proceed to the next step for voting reasoning. Step 3.3 Based on the entity relations of the source knowledge graph, the importance assessment method of relation frequency-inverse entity frequency is used to obtain key relation triples. The key relation triples and candidate entity relation triples are simultaneously input into the large language model, and multiple rounds of voting reasoning are carried out on the candidate entity set. Before each round of voting reasoning, the input order of the candidate entity set is randomly rearranged, and the large language model selects a candidate entity from the candidate entity set. After multiple rounds of voting, the cumulative number of votes of the selected candidate entity in each round is counted. If there is a candidate entity with the highest number of votes and not less than half, it is determined as the alignment result of the source entity. For the source entity that has not produced an alignment result at this level, it, along with the corresponding candidate entity set, enters the next step for voting reasoning. Step 3.4 Simultaneously input the text descriptions of the source knowledge graph entity-associated images and the corresponding candidate entity-associated images into the large language model to conduct multiple rounds of voting reasoning on the candidate entity set. Before each round of voting reasoning, the input order of the candidate entity set for this round is randomly rearranged, and the large language model selects a candidate entity from the candidate entity set. After completing multiple rounds of voting, the cumulative number of votes for the selected candidate entity in each round is counted. If there is a candidate entity with the highest number of votes and not less than half, it is determined as the alignment result of the source entity. For source entities that do not produce an alignment result at this level, they are marked as entities that cannot be aligned.

[0013] This invention, through unified multimodal vector representation, cross-modal interactive modeling, and a multi-level reasoning mechanism for large language models, offers the following technical advantages compared to existing entity alignment technologies: (1) A unified multimodal representation mechanism for the embedding space reduces the semantic fragmentation problem caused by modal heterogeneity. This invention uses a unified textualization and visual prefix mapping module to map relationships, attributes, and images into a single-dimensional word embedding sequence, enabling multimodal information to be represented in a unified text space. This achieves standardized representation and alignment of multimodal features, effectively eliminating differences in feature distribution and scale misalignment between different modalities, and significantly improving the matching degree and consistency of multimodal feature fusion.

[0014] (2) The dual-path cross-modal interaction structure enhances the ability of multimodal semantic modeling. This invention designs a dual-path network structure with a global semantic fusion path and a local cross-modal enhancement path running in parallel. The global semantic fusion path utilizes BERT's self-attention mechanism to achieve global semantic interaction between vector representations of different modalities. The local cross-modal enhancement path uses a cross-attention mechanism to perform pairwise interactions between vector representations of different modalities, thereby enhancing fine-grained semantic associations. Through this dual-path design, it is possible to learn the overall semantic representation and further enhance the local cross-modal feature matching ability, thereby improving the discriminative ability of entity vector representations.

[0015] (3) Multi-branch contrastive learning and dynamic weight mechanism improve model robustness This invention employs a four-branch bidirectional contrastive learning mechanism, calculating the branch contrastive loss corresponding to relation vector representation, attribute vector representation, visual vector representation, and the joint vector representation of the three modalities. This allows each modality to independently learn discriminative entity vector representations. Simultaneously, this invention introduces dynamic weights based on softmax to dynamically rescale the contrastive losses of the four branches, automatically reducing the weights of modal branches with smaller contributions or higher noise levels, thereby achieving online modality pruning. This mechanism can adaptively adjust the importance of each modality without adding additional network structure, thus improving the model's robustness in situations with small samples or incomplete modalities.

[0016] (4) The multi-level large language model voting reasoning mechanism effectively reduces the large language model illusion problem. This invention constructs a fixed set of candidate entities and sequentially implements a three-level reasoning strategy for a large language model based on attributes, relationships, and visual information. This allows the large language model to make inferences and judgments using different modal evidence at each level. Specifically, the first level of reasoning and screening is performed using key attribute triples. If attribute information is insufficient, key relationship triples are introduced for the second level of reasoning. If the final result still cannot be determined, visual information is used for supplementary judgment at the third level. Simultaneously, during each level of reasoning, the order of the candidate entity set is randomly rearranged and multiple rounds of voting are conducted to determine the final result using a majority voting strategy. This effectively reduces the impact of positional bias and occasional hallucinations. Attached Figure Description

[0017] Figure 1This is an overall flowchart of an embodiment of the present invention.

[0018] Figure 2 This is a flowchart illustrating the construction process of the dual-path cross-modal interaction model trained according to an embodiment of the present invention. Detailed Implementation

[0019] To better understand this invention, the embodiments of this invention will explain some related knowledge and definitions used, as follows: (1) Knowledge graph: A knowledge graph is a semantic network that organizes knowledge in the form of a graph structure. It is usually composed of entities, relations and attributes. Entities represent objects in the real world, relations describe the semantic associations between entities, and attributes are used to characterize the feature information of entities.

[0020] (2) Entity: Node objects in the knowledge graph, which can be names, locations, organizations, etc.

[0021] (3) Entity alignment: The process of discovering entity pairs that refer to the same real object between different knowledge graphs and establishing equivalent links is the core step of cross-graph fusion.

[0022] (4) Cross-attention mechanism: a variant of attention where the query comes from one modality of data and the key-value comes from another modality of data. Fine-grained matching weights between the two heterogeneous features can be explicitly calculated to achieve local information interaction.

[0023] The following will detail an entity alignment method based on cross-modal interaction and multi-level voting of a large language model provided by an embodiment of the present invention. The method takes the source knowledge graph to be aligned and the target knowledge graph as input, and the specific process is as follows: Figure 1 As shown, proceed as follows: Step 1. Multimodal feature extraction and word embedding sequence construction Extract entities from the source knowledge graph and the target knowledge graph to be aligned. e The relation triples and attribute triples are processed into text and segmented into words respectively to obtain relation lexical sequences and attribute lexical sequences. These sequences are then mapped to a unified vector space to generate relation lexical embedding sequences and attribute lexical embedding sequences. The entity association image is obtained, and visual features are extracted using a visual encoder. These visual features are projected onto the same embedding dimension as the relation lexical embedding sequences and attribute lexical embedding sequences, and then converted into a fixed-length visual prefix lexical embedding sequence. The specific steps are as follows: Step 1.1 Concatenate the relation triples and attribute triples into a relational text sequence in lexicographical order. and attribute text sequence , means as follows: (1) (2) in, and This represents a mapping function that converts triples into text segments. This represents a string concatenation operation. and These represent relation triples and attribute triples, respectively. Representing entities e All relation triples in a knowledge graph The set constituted Representing entities e All attribute triples in a knowledge graph The set constituted Indicates a sorting operation; The relational text sequences are then segmented using the BERT tokenizer. and attribute text sequence The words are transformed into discrete tokens, which are then mapped to BERT's embedding layer. D 3D vector representation, thus obtaining the relational word embedding sequence. R and attribute lexical embedding sequence A ; Step 1.2: Extract entity association images using the CLIP visual encoder. image Global visual features ; (3) The global visual features are processed through a multilayer perceptron. Projection to relational word embedding sequence R and attribute lexical embedding sequence A Same embedding dimension ( D ), and convert it into a visual prefix lexical embedding sequence of preset length. P ; (4) in, l It is the visual prefix length. D It is the hidden layer dimension of BERT. , These are the weights and biases of the trained and converged visual projection layer, respectively.

[0024] Step 2. Generate four sets of entity vectors based on the trained dual-path cross-modal interaction model. The completed training of the dual-path cross-modal interaction model is as follows: Figure 2 The process shown is used to construct the following: Step 2.1 Build a dual-path cross-modal interaction model. The network includes a global semantic fusion path and a local cross-modal enhancement path set in parallel. The global semantic fusion path is equipped with a BERT encoder. The local cross-modal enhancement path includes three sets of bidirectional interaction units. Each set of bidirectional interaction units includes two cross-attention modules with opposite directions, which are used to model the bidirectional semantic interaction between the two modalities. The output of each set of bidirectional interaction units is connected to the residual structure and the pooling layer. Step 2.2: Retrieve the source knowledge graph and target knowledge graph from the training dataset, and call steps 1.1 and 1.2 to obtain the relation word embedding sequences corresponding to the source and target knowledge graphs, respectively. R Attribute lexical embedding sequence A and visual prefix word embedding sequence P ; Step 2.3 Embedding relational terms into the sequence R Attribute lexical embedding sequence A and visual prefix word embedding sequence P Concatenate them into a complete input sequence T; (5) in, Indicates the starting position of the sequence. This indicates a separator used to separate different parts. l It is the visual prefix length. It is the length of the relation word sequence R. It is the length of the attribute lexical sequence A, where "+3" represents 1. and 2 mark, D It is the hidden layer dimension of BERT; Then the input sequence T The input is fed into BERT, processed through BERT's multi-layer self-attention mechanism, and then linearly mapped using a multi-layer perceptron to obtain a unified-dimensional trimodal joint vector representation. ; (6) in, This represents a feedforward neural network. M This represents the hidden layer dimension of a multilayer perceptron; Step 2.4 Local Enhancement Path performs pairwise cross-attention interactions on the relation lexical embedding sequence, attribute lexical embedding sequence, and visual prefix lexical embedding sequence to achieve fine-grained cross-modal deep interaction and enhance the vector representation of each modality; specifically as follows: For any two modes , ，r、a and vLet the three modalities be relational, attribute-based, and visual, respectively, and let their corresponding embedding matrices be denoted as . and (that is, when) i = r hour (and so on), then the mode j Enhanced mode i The result is: (7) in, X i As a query matrix, X j It serves as both a key matrix and a value matrix. For modality i The number of lexical units.

[0025] Furthermore, for each mode i Combine it with two other modes j and z (in The interaction result is residually joined with its original embedding and then subjected to pooling to obtain the modality augmentation representation. : (8) in, This indicates a pooling operation. M It is the dimension of the vector representation obtained after the pooling operation; this operation yields the relation enhancement representation. Attribute enhancement representation and visual enhancement representation It preserves the original modal information while enhancing the complementarity between modes; Step 2.5 Based on the aligned seed pairs, represent the three-modal joint vector respectively. Relationship enhancement Attribute enhancement representation and visual enhancement representation Perform bidirectional contrastive learning to obtain a four-branch contrastive loss; Specifically, for each branch Calculate the two-way contrast loss : (9) (10) in, src Represents the source knowledge graph. tgt This represents the target knowledge graph. Indicates branch Zhongyuan Knowledge Graphi Vectors of entities, Indicates branch The first target knowledge graph j The vector of each entity, the i and j For the matching alignment seed pair, Indicates branch The entire batch of negative sample vectors, and m Not equal to i and j , The temperature coefficient is represented by "batch," which indicates the set of all entity samples in the current training batch. After performing four-branch bidirectional contrastive learning, four corresponding modal losses are obtained, and then the learnable branch weights are used. The softmax function generates the final weights for each branch. This process automatically reduces the weights of failed branches to zero, achieving online modal pruning. The final total loss is obtained by multiplying the dynamically acquired weights by the corresponding modal losses and summing the results. ; (11) (12) in, Represents the modal branch weights. Temperature represents the weight, controlling branch pruning. Indicates the first The final weight of the branch, Automatic pruning to 0 means online pruning, where the sum of the weights of all branches is 1. Indicates the first Modal loss of each branch, Indicates the total loss; Based on total loss Perform backpropagation to iteratively update the parameters of the two-path cross-modal interaction model until the loss converges to a stable value, thus completing model training; Step 2.6 Embedding relational terms into the sequence R、 Attribute Lexicon Embedding Sequence A and visual prefix word embedding sequence P The trained dual-path cross-modal interaction model yields four sets of entity vectors, including relation vector representations. Attribute vector representation Visual vector representation Joint vector representation of three modes .

[0026] Step 3. Multimodal feature fusion and multi-level voting entity alignment Step 3.1 Generate four similarity matrices by pairwise dot products of the trimodal joint vector representations, relation-enhanced representations, attribute-enhanced representations, and visual-enhanced representations of the source and target knowledge graphs. These matrices are then weighted and summed using learnable weights to obtain a comprehensive score matrix. Based on this comprehensive score matrix, a Top-Ranked Entity is selected for each source entity. k The candidate entities constitute a candidate entity set; Step 3.2 Calculate the information gain based on the entity attributes of the source knowledge graph, and select several attributes with the highest information gain to construct key attribute triples; Key attribute triples are obtained by using an information selection method based on information gain. Information gain value can effectively measure the ability of an attribute to distinguish entities and can quantify the amount of information that an attribute can bring to the entity alignment task. The higher the value, the more the attribute can help distinguish between matching and non-matching entities. The matching relationship between the source entity and the candidate entity is used as a label (1 = match, 0 = no match), and for each attribute... Calculate its information gain value : (13) in, It is the entropy of the label (a measure of the uncertainty of the label). Is a given attribute Conditional entropy of post-label (a measure of known conditions) (remaining uncertainty after labeling), then press Sort the values ​​from highest to lowest and take the top- k Each attribute is used to construct a key attribute triple; The key attribute triples and candidate entity attribute triples are simultaneously input into a large language model (such as a general-purpose large language model like GPT-4) to perform processing on the candidate entity set. n Round-by-round voting reasoning: Before each round of voting reasoning, the input order of the candidate entity set for this round is randomly rearranged. The large language model selects a candidate entity from the candidate entity set. After multiple rounds of voting, the cumulative number of votes for the selected candidate entity in each round is counted. If there is a candidate entity with the highest number of votes and not less than half, it is determined as the alignment result of the source entity. For the source entity that has not produced an alignment result at this level, it, along with the corresponding candidate entity set, enters the next step for voting reasoning. Step 3.3 Based on the entity relations of the source knowledge graph, the key relation triples are obtained by using the relation frequency-inverse entity frequency importance assessment method; Relation frequency is a measure of the frequency of a relation. r With a certain entity e The tightness of the binding ( r Appeared e The higher the frequency in the triplet, the better. eThe more important the representation, the better; inverse entity frequency is a measure of a relationship. r global scarcity ( r The fewer associated entities, the higher the distinguishability and the greater the frequency of inverse entities; the specific operation is as follows: (14) in, Indicates the number of times a relation appears in an entity. Indicates how many entities own the relationship. This represents the total number of entities, for each entity Each relationship Calculate the score , The higher, the better. right The stronger the recognizability; according to Sort from highest to lowest, and take the top- k These relationships constitute a key relation triplet; Key relation triples and candidate entity relation triples are simultaneously input into the large language model to perform processing on the candidate entity set. n Round-by-round voting reasoning: Before each round of voting reasoning, the input order of the candidate entity set for this round is randomly rearranged. The large language model selects a candidate entity from the candidate entity set. After multiple rounds of voting, the cumulative number of votes for the selected candidate entity in each round is counted. If there is a candidate entity with the highest number of votes and not less than half, it is determined as the alignment result of the source entity. For the source entity that has not produced an alignment result at this level, it, along with the corresponding candidate entity set, enters the next step for voting reasoning. Step 3.4: Simultaneously input the text descriptions of the source knowledge graph entity-associated images and the corresponding candidate entity-associated images into the large language model. The text descriptions of the associated images are pre-obtained using LLaVA image understanding. The system will automatically index the corresponding entity image text descriptions and input them into the large language model to perform analysis on the candidate entity set. n Round-by-round voting reasoning: Before each round of voting reasoning, the input order of the candidate entity set for this round is randomly rearranged. The large language model selects a candidate entity from the candidate entity set. After multiple rounds of voting, the cumulative number of votes for the selected candidate entity in each round is counted. If there is a candidate entity with the highest number of votes and not less than half, it is determined as the alignment result of the source entity. For the source entity that does not produce an alignment result at this level, it is marked as an entity that cannot be aligned.

[0027] To further illustrate the execution process of the three-level multi-round voting mechanism of the large language model of this invention, the following descriptions will take attribute-level voting, relation-level voting, and visual-level voting as examples. First, let's take attribute-level large language model voting as an example. Assume the source entity is "Huangshan". The system recalls four candidate target entities for this source entity based on the comprehensive similarity matrix, denoted as candidate (a), candidate (b), candidate (c), and candidate (d). The system inputs the key attribute triples of the source entity and the corresponding attribute triples of the four candidate target entities into the large language model, requiring the large language model to select a uniquely matching target entity from the candidates solely based on attribute information. Among them, the key attribute triple of the source entity "Huangshan" includes: <Huangshan, geographical location, Huangshan City, Anhui Province>, <Huangshan, landscape type, famous mountain with granite peak forest landform>, <Huangshan, scenic area level, World Cultural and Natural Heritage>. The attribute information of the four candidate target entities is as follows: Candidate (a) is "Huangshan", and its attribute triple includes: <Huangshan, province, Anhui Province>, <Huangshan, landform type, granite peak forest landform>, <Huangshan, scenic area level, World Cultural and Natural Heritage>. Candidate (b) is "Huangguoshu Waterfall", and its attribute triple includes: <Huangguoshu Waterfall, geographical location, Anshun City, Guizhou Province>, <Huangguoshu Waterfall, landscape type, karst waterfall landscape>, <Huangguoshu Waterfall, waterfall height, 77.8 meters>. Option (c) is "Jiuhua Mountain", and its attribute triplets include: <Jiuhua Mountain, province, Anhui Province>, <Jiuhua Mountain, religious attribute, famous Buddhist mountain>, <Jiuhua Mountain, scenic area level, World Cultural Heritage>.

[0028] The candidate (d) is "Lushan", and its attribute triple includes: <Lushan, main peak elevation, 1474 meters>, <Lushan, landscape type, metamorphic rock landform>, <Lushan, historical status, World Cultural Heritage>. In one round of attribute-level voting, the large language model performs matching judgments based on the input attribute triples and outputs a candidate number (a). This output indicates that in the current round of attribute-level voting, the target entity "Huangshan" corresponding to candidate (a) is selected as a candidate entity set that matches the source entity "Huangshan". In actual execution, attribute-level voting is not executed only once, but rather a pre-defined process is performed. n Rounds of voting. Before each round of voting, the system randomly rearranges the display order of candidate target entities and records the candidate entities selected by the large language model in that round. Complete. n After each round of voting, the system counts the number of votes for each candidate entity. If a candidate entity receives the most votes and not less than half, it is selected as the final aligned entity. If the attribute-level voting does not produce a candidate entity that meets the majority voting condition, the system proceeds to the relation-level voting.

[0029] Secondly, let's take the voting method of a relational large language model as an example. Assume the source entity is "Slender West Lake", and the system maintains the aforementioned Top... -k The candidate entity set remains unchanged, and the key relation triples of the source entity and the corresponding relation triples of the four candidate target entities are input into the large language model. The large language model performs matching judgments based solely on relation information and selects the unique matching target entity from the candidates. Among them, the key relationship triple of the source entity "Slender West Lake" includes: <Slender West Lake, adjacent to, Five Pavilion Bridge>, <Slender West Lake, city, Yangzhou, Jiangsu Province>, <Slender West Lake, associated culture, Yangzhou gardens>. The relationship information of the four candidate target entities is as follows: Candidate (a) is “Slender West Lake”, and its relational triplets include: <Slender West Lake, including scenic spots, Five Pavilion Bridge>, <Slender West Lake, located in, Yangzhou City, Jiangsu Province>, <Slender West Lake, associated with culture, classical gardens of Yangzhou>.

[0030] Candidate (b) is "West Lake", and its relational triplets include: <West Lake, famous scenic spot, Leifeng Pagoda>, <West Lake, located in Hangzhou, Zhejiang Province>, <West Lake, cultural label, world cultural landscape>. The candidate (c) is "East Lake", and its relational triples include: <East Lake is adjacent to Moshan Scenic Area>, <East Lake is located in Wuhan City, Hubei Province>, <East Lake is associated with the culture of Chu>. The candidate (d) is "Nanhu". Its relational triplets include: <Nanhu, adjacent scenic spot, Yuehe Old Street>, <Nanhu, location, Jiaxing City, Zhejiang Province>, <Nanhu, historical significance, red revolutionary memorial site>.

[0031] In a round of relation-level voting, the large language model performs matching judgments based on the input relation triples and outputs candidate number (a). This output indicates that in the current round of relation-level voting, the target entity "Slender West Lake" corresponding to candidate (a) is selected as the candidate entity that matches the source entity "Slender West Lake". In actual execution, relation-level voting also performs the preset... n Round voting. Before each round of voting, the system randomly rearranges the display order of candidate target entities and counts the candidate numbers output by the large language model. Complete. n After the first round of voting, if a candidate entity receives the most votes and not less than half, then that candidate entity is output as the final alignment result; if the relation-level voting still does not produce a candidate entity that meets the majority voting condition, then the visual-level voting is initiated. Next, let's take the visual-level large language model voting as an example. Assuming the source entity is "Leshan Giant Buddha," the system inputs the image and text descriptions of the source entity and four candidate target entities into the large language model, requiring the model to select the uniquely matching target entity from the candidates based solely on the visual description information. The image and text descriptions can be pre-generated by the visual language model based on the entity image, or they can be generated and saved by the system based on the entity image features. The image text description of the source entity "Leshan Giant Buddha" is: a seated Maitreya Buddha carved into the mountain cliff, with a total height of 71 meters. It is located at the confluence of the Minjiang River, Qingyi River and Dadu River. The Buddha's body is integrated with the mountain and has no independent base. It is a stone-carved Buddha statue from the Tang Dynasty. The image text descriptions of the four candidate target entities are as follows: Option (a) is “Leshan Giant Buddha”, whose image text description is: a giant seated Maitreya Buddha carved into the mountain cliff, 71 meters high, located at the confluence of the Minjiang River, Qingyi River and Dadu River. The Buddha’s body is integrated with the mountain and has no independent base. It is a cliff carving from the Tang Dynasty and is the world’s largest stone Buddha statue. Candidate (b) is “Stone Buddha of Lingyin Temple”, whose image text description is: a stone Buddha in the courtyard of the temple, 19.6 meters high, surrounded by temple buildings. Option (c) is “Buddha statues in Longmen Grottoes”, and its image text description is: Buddha statues carved into the grotto wall, with a total height of 17.14 meters, which are part of the grotto complex and consist of multiple Buddha statues side by side. Option (d) is “Buddha statue in Yungang Grottoes”, and its image text description is: an open-air Buddha carved into the cliff, 13.7 meters high, which is the main statue of the grotto complex. The Buddha’s body is connected to the rock, a stone Buddha statue from the Northern Wei Dynasty, surrounded by numerous niches. In a round of visual-level voting, the large language model performs visual semantic matching based on the input image text description and outputs candidate number (a). This output indicates that in the current round of visual-level voting, the target entity "Leshan Giant Buddha" corresponding to candidate (a) is selected as the candidate entity that matches the source entity "Leshan Giant Buddha". In actual execution, visual-level voting also performs the preset... n Rounds of voting. Before each round of voting, the system randomly rearranges the display order of candidate target entities and records the candidate entities selected by the large language model. (Complete) n After a round of voting, if a candidate entity receives the highest number of votes and not less than half, then that candidate entity is output as the final alignment result; if no candidate entity meets the majority voting condition after visual-level voting, then the alignment of that source entity is abandoned and it is marked as an entity that cannot be aligned. Through the aforementioned three-level, multi-round voting mechanism at the attribute, relation, and visual levels, this invention can progressively filter candidate entity sets using different modal evidence. When attribute information is sufficient to determine the alignment result, the system directly outputs the attribute-level voting result; when attribute information is insufficient, the system further utilizes relation information for supplementary judgment; when both attribute and relation information are insufficient, the system uses visual information for final judgment. Simultaneously, by randomly rearranging the order of candidate entity sets within each level and performing multiple rounds of voting, the impact of the candidate entity set arrangement order on the output of the large language model can be reduced, improving the stability and reliability of the entity alignment results.

Claims

1. An entity alignment method based on cross-modal interaction and multi-level voting of a large language model, taking the source knowledge graph and target knowledge graph to be aligned as input, characterized in that... Perform the following steps in sequence: Step 1. Multimodal feature extraction and word embedding sequence construction Extract relation triples and attribute triples of entities from the source and target knowledge graphs to be aligned, and perform textualization and word segmentation processing respectively to obtain relation lexical sequences and attribute lexical sequences; map the relation lexical sequences and attribute lexical sequences to a unified vector space to generate relation lexical embedding sequences and attribute lexical embedding sequences; obtain the entity association image, extract the visual features of the association image using a visual encoder, project the visual features to the same embedding dimension as the relation lexical embedding sequences and attribute lexical embedding sequences, and convert them into fixed-length visual prefix lexical embedding sequences; Step 2. Generate multiple sets of entity vectors based on the trained dual-path cross-modal interaction model. The trained dual-path cross-modal interaction model processes relation lexical embedding sequences, attribute lexical embedding sequences, and visual prefix lexical embedding sequences, outputting four sets of entity vectors: trimodal joint vector representation, relation enhancement representation, attribute enhancement representation, and visual enhancement representation. Step 3. Multimodal feature fusion and multi-level voting entity alignment A comprehensive score matrix is ​​generated based on four sets of entity vectors. Candidate entity sets corresponding to the source entity are selected from the comprehensive score matrix. Then, in the order of attribute, relationship, and visual, multi-round voting reasoning is carried out on the candidate entity sets. If the current level selects an aligned entity, the aligned entity is output as the alignment result of the source entity. For the source entity and its corresponding candidate entity set that do not produce an alignment result, the next level of voting reasoning is entered. After completing all levels of voting reasoning, the entity alignment results between the source knowledge graph and the target knowledge graph are obtained.

2. The entity alignment method based on cross-modal interaction and multi-level voting of a large language model according to claim 1, characterized in that... Step 1 is as follows: Step 1.1 Concatenate the relation triples and attribute triples into relation text sequences and attribute text sequences respectively in lexicographical order. Then, use the BERT tokenizer to convert the relation text sequences and attribute text sequences into discrete tokens. After the embedding layer, map them into fixed-dimensional relation token embedding sequences and attribute token embedding sequences. Step 1.2 Use CLIP visual encoder to extract global visual features of entity association image, project the global visual features to the same embedding dimension as relation word embedding sequence and attribute word embedding sequence through multilayer perceptron, and convert them into visual prefix word embedding sequence of preset length.

3. The entity alignment method based on cross-modal interaction and multi-level voting of a large language model according to claim 2, characterized in that... The trained dual-path cross-modal interaction model described in step 2 is constructed according to the following steps: Step 2.1 Construct a dual-path cross-modal interaction model. This model includes a global semantic fusion path and a local cross-modal enhancement path set in parallel. The global semantic fusion path is equipped with a language model encoder. The local cross-modal enhancement path includes three sets of bidirectional interaction units. Each set of bidirectional interaction units includes two cross-attention modules with opposite directions, which are used to model the bidirectional semantic interaction between the two modalities. The output of each set of bidirectional interaction units is connected to the residual structure and the pooling layer. Step 2.2 Take the source knowledge graph and target knowledge graph in the training dataset, call steps 1.1 and 1.2 to obtain the relation word embedding sequence, attribute word embedding sequence and visual prefix word embedding sequence corresponding to the source graph and target graph respectively; Step 2.3 Global Semantic Fusion Path: The relation word embedding sequence, attribute word embedding sequence, and visual prefix word embedding sequence are concatenated and input into the language model encoder to achieve global semantic interaction between different modalities and output a trimodal joint vector representation. Step 2.4 Local Enhancement Path performs cross-attention interaction on each pair of relation lexical embedding sequence, attribute lexical embedding sequence, and visual prefix lexical embedding sequence. The interaction results are residually connected with the original lexical embedding sequence and pooled to output relation enhancement representation, attribute enhancement representation, and visual enhancement representation. Step 2.5 Based on the aligned seed pairs, bidirectional contrastive learning is performed on the three-modal joint vector representation, relation enhancement representation, attribute enhancement representation and visual enhancement representation respectively to obtain a four-branch contrastive loss. The four-branch contrastive loss is fused by dynamic weights to obtain the total loss. Combined with the labeled loss of entity seed pairs, backpropagation is performed to iteratively update the parameters of the dual-path cross-modal interaction model until the loss converges to a stable value, and the model training is completed.

4. The entity alignment method based on cross-modal interaction and multi-level voting of a large language model according to claim 3, characterized in that... Step 3 is as follows: Step 3.1 Generate four similarity matrices by pairwise dot products of the three-modal joint vector representation, relation enhancement representation, attribute enhancement representation, and visual enhancement representation of the source knowledge graph and the target knowledge graph. Then, obtain a comprehensive score matrix by weighted summation using learnable weights. Based on the comprehensive score matrix, select a preset number of entities for each source entity to form a candidate entity set. Step 3.2 Calculate information gain based on the entity attributes of the source knowledge graph, and select several attributes with the highest information gain to construct key attribute triples; input the key attribute triples and candidate entity attribute triples into the large language model simultaneously, and conduct multiple rounds of voting reasoning on the candidate entity set; before each round of voting reasoning, randomly rearrange the input order of the candidate entity set for this round, and select a candidate entity from the candidate entity set by the large language model. After completing multiple rounds of voting, count the cumulative number of votes for the selected candidate entity in each round. If there is a candidate entity with the highest number of votes and not less than half, then determine it as the alignment result of the source entity. For source entities that have not produced alignment results at this level, they, along with the corresponding candidate entity set, proceed to the next step for voting reasoning. Step 3.3 Based on the entity relations of the source knowledge graph, the importance evaluation method of relation frequency-inverse entity frequency is used to obtain key relation triples. The key relation triples and candidate entity relation triples are simultaneously input into the large language model, and multiple rounds of voting reasoning are carried out on the candidate entity set. Before each round of voting reasoning, the input order of the candidate entity set in this round is randomly rearranged, and the large language model selects a candidate entity from the candidate entity set. After completing multiple rounds of voting, the cumulative number of votes of the selected candidate entity in each round is counted. If there is a candidate entity with the highest number of votes and not less than half, it is determined as the alignment result of the source entity. For source entities that have not produced alignment results at this level, they, along with the corresponding candidate entity set, proceed to the next step for voting reasoning. Step 3.4 Simultaneously input the text descriptions of the source knowledge graph entity-associated images and the corresponding candidate entity-associated images into the large language model, and conduct multiple rounds of voting reasoning on the candidate entity set; before each round of voting reasoning, the input order of the candidate entity set for this round is randomly rearranged, and the large language model selects a candidate entity from the candidate entity set. After completing multiple rounds of voting, the cumulative number of votes for the selected candidate entity in each round is counted. If there is a candidate entity with the highest number of votes and not less than half, it is determined as the alignment result of the source entity. For source entities that do not produce alignment results at this level, they are marked as unalignable entities.

Images