Entity alignment method, device, and storage medium

Abstract

The application discloses an entity alignment method and device and a storage medium, relates to the technical field of knowledge graphs, and comprises the following steps: encoding text information and visual information in an entity to be aligned in a source knowledge graph, and generating corresponding text representation and visual representation; generating target visual enhanced representation of the entity to be aligned by taking the text representation as a guide; determining a target gating value corresponding to the target visual enhanced representation based on a gating network; generating target fusion representation of the entity to be aligned according to the target gating value, the text representation and the target visual enhanced representation; obtaining a target entity matching the target fusion representation from a target knowledge graph according to the target fusion representation, and generating a target alignment relationship corresponding to the entity to be aligned according to the target entity, so that the problem of misalignment caused by image loss is solved by combining the text as an anchor point and limiting the introduction intensity of image information based on the gating network.

Description

Technical Field

[0001] This application relates to the field of knowledge graph technology, and in particular to entity alignment methods, devices and storage media. Background Technology

[0002] Existing unsupervised multimodal entity alignment methods typically first encode the textual and structural information of entities to obtain textual representations, and simultaneously perform visual encoding on the associated images of entities to obtain visual representations. Then, a unified multimodal entity representation is formed through weighted fusion, attention fusion, or gated fusion. Based on this, cosine similarity or dot product similarity is used for cross-graph retrieval and ranking, and the most similar target entity is directly selected as the alignment result. However, multimodal data in real-world scenarios generally suffers from missing data and noise, especially visual modalities, which may be missing, inconsistent across domains, or semantically weakly related. Traditional tightly coupled fusion methods that can be readily applied can easily amplify unreliable visual evidence and lead to misalignment.

[0003] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0004] The main objective of this application is to provide a method, apparatus, and storage medium for entity alignment, aiming to solve the technical problem of misalignment caused by defects in images.

[0005] To achieve the above objectives, this application proposes a solid alignment method, the steps of which include: Encode the textual and visual information of the entities to be aligned in the source knowledge graph to generate corresponding textual and visual representations; The target visual enhancement representation corresponding to the visual representation is generated based on the text representation; The target gating value corresponding to the target visual enhancement representation is determined based on the gating network; Generate a target fusion representation of the entity to be aligned based on the target gating value, the text representation, and the target visual enhancement representation; Based on the target fusion representation, target entities matching the target fusion representation are obtained from the target knowledge graph, and target alignment relationships corresponding to the entities to be aligned are generated based on the target entities.

[0006] For example, the step of generating the target visual enhancement representation corresponding to the visual representation based on the text representation includes: The text representation is used as the query vector, and the visual representation is used as the key vector and value vector, which are then input into the cross-attention fusion model. The output of the cross-attention fusion model is used as the target visual enhancement representation, which is a visual representation generated guided by the text representation.

[0007] For example, the step of determining the target gating value corresponding to the target visual enhancement representation based on the gating network includes: The text representation and the target visual enhancement representation are input as input information to the gating network; Obtain the output result generated by the gating network based on the input information, the output result including the target matching degree between the target visual augmentation representation and the text representation; The target gating value is generated based on the target matching degree, wherein the target gating value is positively correlated with the target matching degree.

[0008] For example, the step of generating the target fusion representation of the entity to be aligned based on the target gating value, the text representation, and the target visual enhancement representation includes: The target gate value is used as the target weight value corresponding to the target visual enhancement representation, and the target visual enhancement representation is weighted and fused. The weighted and fused visual representation of the target is fused with the text representation to generate the fused representation of the target.

[0009] For example, the step of obtaining the target entity matching the target fusion representation from the target knowledge graph based on the target fusion representation includes: The target fusion representation is input into the retrieval pre-training model so that the retrieval pre-training model can obtain at least one candidate entity from the target knowledge graph based on the target fusion representation; A candidate entity set is generated based on the candidate entities; The target entity corresponding to the entity to be aligned is determined based on the candidate entity set.

[0010] For example, the step of determining the target entity corresponding to the entity to be aligned based on the candidate entity set includes: The textual and visual information of the entity to be aligned and the candidate entities are input into a multimodal large model to obtain a preset number of alignment selection results; Obtain the number of selections for each candidate entity in the alignment selection results; Candidate entities selected more than a preset number of times are identified as the target entities corresponding to the entities to be aligned.

[0011] For example, after the step of generating the target alignment relationship corresponding to the entity to be aligned based on the target entity, the method further includes: The target alignment relationship is input into the retrieval pre-training model so that the retrieval pre-training model can perform an update operation based on the target alignment relationship; The updated retrieval pre-trained model is used to return the steps of inputting the target fusion representation into the retrieval pre-trained model.

[0012] For example, the step of the retrieval pre-trained model performing an update operation based on the target alignment relationship includes: The target entity in the target alignment relationship is determined as the target positive matching entity of the entity to be aligned; Identify the remaining candidate entities in the candidate entity set, excluding the positive matching entity, and obtain the similarity between the remaining candidate entities and the entity to be aligned; Entities with a similarity greater than or equal to a preset similarity are identified as target negative samples; The target contrast learning loss value is generated by forming a triplet based on the entity to be aligned, the target positive matching entity, and the target negative sample. The parameters of the retrieval pre-trained model are updated using gradients based on the target contrastive learning loss value.

[0013] In addition, to achieve the above objectives, this application also proposes an entity alignment device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the entity alignment method as described above.

[0014] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and which, when executed by a processor, implements the steps of the entity alignment method described above.

[0015] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the entity alignment method described above.

[0016] This application first generates a target visual augmentation representation corresponding to the visual representation using the text representation as a guide, and then uses highly reliable text information as a benchmark to retrieve image information. Subsequently, a target gate value corresponding to the target visual augmentation representation is determined based on a gating network to achieve dynamic introduction of image information. When the image information is unreliable, the introduction intensity is reduced, and when the image information is reliable, the introduction intensity is increased, avoiding entity misalignment caused by excessive introduction of unreliable image information. The embodiments of this application improve the robustness of entity alignment by using reliable text information as a guide and setting a gate value for image information. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the first embodiment of the entity alignment method of this application. Figure 2 This is a schematic diagram of the architecture of the entity alignment device involved in the first embodiment of the entity alignment method of this application; Figure 3 This is a schematic diagram of the process involved in the first embodiment of the entity alignment method of this application; Figure 4 This is a detailed flowchart of step S20 in the first embodiment of the entity alignment method of this application; Figure 5 This is a detailed flowchart of step S30 in the first embodiment of the entity alignment method of this application; Figure 6 This is a detailed flowchart of step S40 in the first embodiment of the entity alignment method of this application; Figure 7 This is a detailed flowchart of step S50 in the second embodiment of the entity alignment method of this application; Figure 8 This is a detailed flowchart of step S53 in the third embodiment of the entity alignment method of this application; Figure 9 This is a flowchart illustrating the fourth embodiment of the entity alignment method of this application; Figure 10 This is a schematic diagram of the device structure of the hardware operating environment involved in the entity alignment method in the embodiments of this application.

[0020] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0021] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0022] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0023] In this embodiment, for ease of description, the entity alignment device will be used as the execution subject in the following description.

[0024] When performing multimodal alignment of entities, visual modalities are unreliable, such as missing data, noise interference, weak semantic correlation, and cross-domain inconsistencies. If entities are aligned based on unreliable visual modalities, it will excessively affect the final entity representation, introduce "visual negative transfer," and cause the system to make incorrect alignment judgments.

[0025] This application provides a solution that, after generating text and visual representations of the entities to be aligned using an encoder, first calculates a visually enhanced representation corresponding to the visual representation based on a cross-attention mechanism. Specifically, when generating the visually enhanced representation using the cross-attention mechanism, reliable text information is used as a guide to generate the visually enhanced representation. Then, a gate value corresponding to the visually enhanced representation is obtained, and the introduction strength of the visually enhanced representation is determined based on the gate value. This prevents the over-introduction of visually enhanced representations that are semantically inconsistent with the text representation, which would lead to inaccurate fusion representations and consequently, inaccurate target entities generated with incorrect fusion representations, affecting the accuracy of entity alignment. This application's embodiment improves alignment accuracy by combining text information as a guide with setting a gate value to dynamically introduce image information.

[0026] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device or entity alignment device capable of performing the above functions. The following description uses an entity alignment device as an example to illustrate this embodiment and the subsequent embodiments.

[0027] First Embodiment Based on this, embodiments of this application provide an entity alignment method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the entity alignment method of this application.

[0028] In this embodiment, the entity alignment method includes steps S10 to S50: Step S10: Encode the text and visual information of the entities to be aligned in the source knowledge graph to generate corresponding text and visual representations; Step S20: Using the text representation as a guide, generate the target visual enhancement representation corresponding to the visual representation; Step S30: Determine the target gating value corresponding to the target visual enhancement representation based on the gating network; Step S40: Generate the target fusion representation of the entity to be aligned based on the target gating value, the text representation, and the target visual enhancement representation; Step S50: Obtain target entities that match the target fusion representation from the target knowledge graph based on the target fusion representation, and generate target alignment relationships corresponding to the entities to be aligned based on the target entities.

[0029] In this embodiment, the entity alignment method is applied to an entity alignment apparatus. (Refer to...) Figure 2 The entity alignment device includes a retrieval pre-trained model 10 and a multimodal large model 20. The retrieval pre-trained model 10 is used to generate a candidate entity set containing a preset number (N) of candidate entities, so as to input the preset number of candidate entities into the multimodal large model 20. The candidate entity set includes at least one candidate entity.

[0030] In some alternative implementations, the pre-trained model 10 is a lightweight training model. The preset number can be user-defined or determined in real time by the system based on the computing power scale of the multimodal large model.

[0031] Understandably, by limiting the number of candidate entities input to the multimodal large model, the inference overhead of the multimodal large model 20 is reduced, while the risk of the multimodal large model 20 generating illusions or unstable judgments is also reduced. The multimodal large model 20 is used to match the corresponding target entity for the entity to be aligned from the input set of candidate entities.

[0032] It should be noted that the multimodal large model 20 includes reasoning and voting modules. The multimodal large model 20 matches the entity to be aligned with the corresponding target entity by repeatedly performing reasoning R times on the same candidate set for the same entity to be aligned, obtaining R candidate selection outputs. The votes of the above outputs are counted, and the candidate entity with the highest number of votes is selected as the target entity of the entity to be aligned.

[0033] When the vote count does not result in a majority, no pseudo-alignment label is generated and the sample is discarded to prevent incorrect results from being used as target entities. When the vote count results in a majority, the target entity relationship between the entity to be aligned and the target entity is output, and a pseudo-alignment label is generated. The target entity relationship is then generated and returned to the retrieval pre-training model 10 for comparative self-training updates.

[0034] Optionally, in the case where the image of the entity to be aligned or the candidate entity is missing, a pure white placeholder image can be used to replace the image information in the input to maintain the consistency of the input form, and the prompt instructions can be used to make it clear that the placeholder image is only used for placeholder and does not contain effective semantic information, thereby avoiding interference from the missing image to the reasoning.

[0035] In this embodiment, a lightweight retrieval pre-trained model 10 is constructed to pre-select candidate entities for inference and voting by the multimodal large model 20, thereby selecting the target entity corresponding to the entity to be aligned. By limiting the number of candidate entities input to the multimodal large model 20, time-consuming and labor-intensive massive searches are avoided, greatly reducing the inference cost of the multimodal large model 20. In addition, the multimodal large model 20 performs multiple inferences, generating corresponding results each time, and voting on the output results. This iterative inference and voting mechanism avoids the accumulation and amplification of errors in the loop, improving the stability of the multimodal large model 20.

[0036] As an optional implementation, the retrieval pre-training model 10 internally includes a language pre-training model 101, an image pre-training model 102, a cross-attention fusion model 103, and a threshold network 104. The language pre-training model 101 is used to receive the text information of the entity to be aligned, encode the text information to generate a text representation of the text information in the entity to be aligned. The image pre-training model 102 is used to receive the image information of the entity to be aligned, encode the image information to generate a visual representation of the image information in the entity to be aligned.

[0037] For example, the language pre-trained model 101 generates text representations in the following way:

[0038] Specifically, the text sequence Se of the entity e to be aligned is obtained, input into the language pre-trained model 101 (L-PTM) to obtain the text embedding, and then normalized to generate the text representation h. t The aforementioned text sequence consists of attribute and relation triplet information.

[0039] Optionally, the image pre-trained model 102 generates visual representations in the following way:

[0040] In this process, the visual information Ie of the entity to be aligned is input into the image pre-training model 102. The image pre-training model 102 (I-PTM) extracts visual features based on the visual information, projects them linearly onto the same dimension as the text representation, and then normalizes them to obtain the visual representation h. v .

[0041] As an optional implementation, the cross-attention fusion model 103 is used to perform cross-attention computation on the text and visual representations of the entities to be aligned. Exemplarily, the cross-attention computation is performed by generating a visually enhanced representation corresponding to the visual representation, guided by the text representation. Specifically, the text representation is used as a query vector, and the visual representation is used as a key vector and value vector, which are input into the cross-attention fusion model 103 to generate the visually enhanced representation.

[0042] Understandably, visual modalities are inherently unreliable. For example, one issue is missing data: many entities in a knowledge graph may not be associated with images, or image links may be broken; another is noise interference: images may be blurry, low-resolution, or contain irrelevant background information; yet another is weak semantic relevance: the images associated with an entity may not directly represent its core semantics (e.g., the entity "Apple Inc." may be incorrectly associated with an image of an apple); and yet another is cross-domain inconsistency: knowledge graphs from different sources may use vastly different image styles, perspectives, and content for the same entity.

[0043] It should be noted that treating textual and image information equally, or allowing unreliable image information to excessively influence the final entity representation, will introduce "visual negative transfer," causing the system to make incorrect alignment judgments. Based on this, this application proposes a "text-centered cross-attention fusion mechanism" to address the problem of misalignment caused by unreliable images.

[0044] For example, during cross-modal attention computation, text representation is used as the query vector, and visual representation is used as the key and value vectors. The text representation "queries" the visual representation vector (as the key and value), meaning the cross-attention fusion process revolves around textual semantics. The cross-attention fusion model 103 considers "Based on the text description, which information in the image is relevant?" to obtain the enhanced visual representation. It is understood that textual representations (such as entity attributes and relation descriptions) typically express the core semantics of an entity more directly and stably than images. For example, attributes such as "capital," "population," and "founded in" are explicit. Therefore, using textual information as an anchor provides a stable and interpretable semantic benchmark for the cross-attention fusion model 103. Even when visual information is chaotic, the cross-attention fusion model 103 can match based on this benchmark, avoiding being misled by misleading visual similarities, such as two different but similar-looking buildings. Furthermore, different knowledge graphs may use drastically different images to describe the same entity, such as daytime and nighttime images of the "Eiffel Tower." If image feature matching is performed directly, the similarity may be very low. However, a text-guided fusion mechanism will first focus on the semantics of the text (such as "the ironwork tower in Paris"), and then look for features in the image that match the text description (such as the structure of the tower). This weakens differences in style, lighting, etc., and focuses on semantic alignment.

[0045] Optionally, the method for generating a visually enhanced representation of the target across the attention fusion model 103 is as follows: In the initial retrieval, a weighted fusion method is used:

[0046] Where α is the initial weight, h t h and hv represent text and visual representations, respectively. final This is a fusion representation. The initial weight α can be user-defined or system-defined; for example, the initial weight α is selected from [0, 100%].

[0047] During the iteration phase, cross-attention computation centered on text is introduced (text is the query, and visuals are the key and value):

[0048] Among them, W Q W K and W V These are the weight matrices; h t h v These represent textual and visual representations, respectively; d is the dimension of the key vector, used for scaling to prevent the dot product from becoming too large, and h... cafAttention score for visual representation.

[0049] It should be noted that after determining the target visual enhancement representation using a text-guided cross-attention fusion mechanism, the gating network 104 generates a gating value corresponding to the visual enhancement representation, and the degree of introduction of the visual enhancement representation is determined based on the gating value. It can be understood that the clearer the image information, the higher the gating value, and the higher the weight assigned to the visual enhancement representation, thus increasing its visual contribution; conversely, the more blurred the image information, the lower the gating value, and the lower the weight assigned to the visual enhancement representation, thus decreasing its visual contribution.

[0050] In some optional implementations, the gating network 104 is used to determine the semantic consistency between the visual augmented representation and the text representation. The stronger the semantic consistency, the higher the matching degree between the visual augmented representation and the text representation, and the higher the gating value generated by the gating network 104. The weaker the semantic consistency, the lower the matching degree between the visual augmented representation and the text representation, and the lower the gating value generated by the gating network 104. The gating value is positively correlated with the semantic consistency, the gating value is positively correlated with the matching degree, and the matching degree is positively correlated with the semantic consistency.

[0051] Understandably, the gating network 104 automatically reduces the weight of visual representation when image noise is high, preventing undesirable image data from contaminating the fused representation. The retrieval pre-trained model 10 utilizes the gating network 104 to dynamically determine the reliability of the current image information based on the consistency between text and image semantics. Specifically, when the image is clear and relevant, the gating value is increased, allowing more visual enhancement information to be incorporated into the text representation. When the image is missing, noisy, or irrelevant, the gating value is decreased, suppressing or even eliminating the influence of unreliable visual information, causing the fused representation to return to a state dominated by text semantics.

[0052] For example, the gating network 104 generates the gating value g corresponding to the visual augmentation representation in the following way:

[0053] Where g is the gate value, h t For text representation, h caf For attention scores of visual representation, Wg and b g Let σ be the trainable parameters of the gated network 104, and let σ represent the sigmoid function.

[0054] Optionally, after generating the text representation, visual enhancement representation, and gating value, the pre-trained model 10 is retrieved based on the text representation h. t Attention score h corresponding to visual representation caf Generate a fusion representation h with the gate value g. finalThe method is as follows:

[0055] Optionally, the retrieval pre-trained model 10 further includes a candidate retrieval module 105, which is used to determine at least one candidate entity in the target knowledge graph that matches the entity to be aligned, based on the fusion representation.

[0056] As an optional implementation, the method for determining at least one candidate entity that matches the entity to be aligned is to calculate the similarity between the entity to be aligned and the entity in the target knowledge graph, and to take the entity with a similarity greater than or equal to a preset threshold as the candidate entity that matches the entity to be aligned. The similarity can be calculated based on cosine similarity.

[0057]

[0058] in, h （s） final , h （t） final The fusion representation of different entities, sim(e) s e t ) represents the calculated cosine similarity.

[0059] It should be noted that the retrieval pre-training model 10 outputs the candidate entity set based on a lightweight model, which reduces the workload required for the multimodal large model 20 to infer the candidate entity set and greatly reduces the inference cost. At the same time, based on the characteristic that text information is usually more accurate and structured than images and is a more reliable judgment benchmark, the retrieval pre-training model 10 proposed in this application realizes automatic filtering of bad visual information through adaptive gating to prevent being misled. Even if the visual modality fails completely, the system can still work normally based on text, which improves the robustness and stability of entity alignment.

[0060] In some optional implementations, the entity alignment device proposed in this application also has a self-updating function. For example, the pseudo-alignment result (e) inferred via the multimodal large model 20 and confirmed by majority voting is used to update the alignment function. s ,e pred ) as the training supervision signal, where e pred The positive matching entity e, identified as the entity to be aligned pos It should be noted that the positive matching entity is the target entity of the entity to be aligned; in order to construct discriminative training samples, the candidate entity set C of the entities to be aligned is then selected. cand(es) Negative sample selection is performed within the candidate entity set by removing positive samples (e). posThen, based on the similarity metric in the current PTM (pre-trained model) representation space, select the entity e to be aligned. s The candidate entity with the highest similarity is selected as the most difficult negative sample e. neg This ensures that the training process focuses on constraining the most easily confused erroneous matches. Understandably, the pseudo-alignment results confirmed by voting are used as a supervision signal. By comparing and contrasting these results with the pre-trained model, the model is updated to generate a higher-quality candidate set in the next round, thus achieving closed-loop iterative improvement and reducing the instability of unsupervised bootstrapping.

[0061] For example, the calculation method for generating the target entity based on the voting reasoning mechanism is as follows:

[0062] Among them, e pred For the target entity, II(v) r =c) represents the probability of selecting a candidate entity from the candidate entity set, argmax （c Ccand） It is used to characterize the target entity selected from the candidate entity set that best matches the entity to be aligned.

[0063] In yet another embodiment, after determining the target entity based on a voting reasoning mechanism, the entity e to be aligned is selected. s The candidate entity with the highest similarity is selected as the most difficult negative sample e. neg The calculation method is as follows:

[0064] Optionally, after obtaining the triple (entity to be aligned, target entity, hardest negative sample) (e s ,e pos ,e neg After that, the retrieval pre-trained model 10 is updated using two types of contrastive learning objectives.

[0065] Optionally, the contrastive learning objective includes a pairing interval loss, used to constrain the similarity of positive samples to be significantly higher than that of negative samples and to satisfy a preset interval, utilizing the pairing interval loss L. align The calculation method is as follows:

[0066] Where, sim(h) s h pos ) is used to characterize the similarity between the feature vector of the entity to be aligned and the positive sample vector, sim(h) s h neg () is used to characterize the similarity between feature vectors and negative sample vectors.

[0067] In another embodiment, the contrastive learning objective further includes an information noise contrastive loss (InfoNCE contrastive loss), used to improve the relative separability of positive samples within the candidate entity range and suppress other contrastive terms such as negative samples. The information noise contrastive loss L... nce The calculation method is as follows:

[0068] Here, τ represents the temperature coefficient, which is used to adjust the balance of the similarity distribution.

[0069] Finally, the above losses are combined into a total loss L. total The parameters of the retrieved pre-trained model 10 are updated via backpropagation, and optionally, the total loss L is calculated. total The method is as follows:

[0070] It should be noted that after updating the retrieval pre-trained model 10 based on backpropagation of the total loss, the execution step S20 is returned based on the updated detection pre-trained model 10. Specifically, the candidate entity set is regenerated with the updated retrieval pre-trained model 10, and the regenerated candidate entity set is input into the multimodal large model 20 for the multimodal large model 20 to re-perform inference voting to update the triple (entity to be aligned, target entity, hardest negative sample) (e s ,e pos ,e neg The total contrast loss is recalculated based on the updated triples, and the retrieval pre-trained model 10 is updated repeatedly based on the recalculated total contrast loss. This process continues until the number of iterations is greater than or equal to the preset number of iterations, thereby improving the retrieval and alignment performance of the retrieval pre-trained model 10.

[0071] In the embodiments of this application, the entity alignment device automatically constructs hard negative samples by ranking the similarity within the candidate entity set during the self-training phase, and adopts a joint optimization objective of marginal ranking loss and contrast loss to widen the gap between positive and negative samples and enhance the discriminative representation, thereby improving alignment accuracy and iterative training stability.

[0072] In yet another embodiment, reference is made to... Figure 3 , Figure 3A simplified flowchart illustrating the entity alignment operation using an entity alignment device is presented. A candidate entity set for the entities to be aligned is obtained by retrieving a pre-trained model based on the source and target knowledge graphs. This candidate entity set is then input into a multimodal large-scale model, which performs inference and voting operations based on the candidate entity set. Candidate entities with more than a preset number of votes are designated as target entities. Pseudo-alignment results and corresponding pseudo-labels are generated based on the target entities. Negative samples (the most difficult negative samples) are selected from the candidate entity set. A contrastive loss is calculated based on the pseudo-alignment results and the negative samples. This contrastive loss is then used to update the retrieval pre-trained model, forming a closed-loop iterative entity alignment operation.

[0073] In this embodiment, the entity alignment device described above is used to perform entity alignment operations. Specifically, entities to be aligned are obtained from a source knowledge graph, which corresponds to a target knowledge graph. The entity alignment device is used to obtain target entities from the target knowledge graph that match the entities to be aligned in the source knowledge graph, and the entities to be aligned and the target entities are mutually matched.

[0074] For example, after identifying the entities to be aligned in the source knowledge graph, the textual and visual information of the entities to be aligned are obtained. The textual information is information represented in text form, including entity, relation triples, and attribute triples. The visual information is information represented in image form, including images associated with the entities to be aligned. The textual and visual information are respectively input into a language pre-training module and an image pre-training model. The language pre-training model first generates a text sequence of the textual information, and then generates a text representation based on the text sequence after normalization. The image pre-training model pre-initializes linear projection and fusion parameters to linearly project the visual information into a space with the same dimension as the textual information, thus mapping the visual information to the same common metric space as the textual information. For example, dimension D=168. Specifically, the image pre-training model extracts visual features based on the visual information and linearly projects them into the same dimension as the textual information, then normalizes them to generate the visual representation, thereby achieving dimension alignment. In another embodiment, the text representation can be dimensionally aligned based on the dimension of the visual representation, or it can be based on linearly projecting both the visual representation and the text representation onto the same dimension.

[0075] It is understood that, given the common occurrence of missing and noisy visual representations, especially the potential for missing visual modalities, cross-domain inconsistencies, or weak semantic correlations, this application introduces a text-guided cross-attention fusion mechanism to compensate for the unreliability of visual representations. This mechanism uses reliable text representations to compensate for the unreliability of visual representations, thereby improving robustness in noisy and missing modal scenarios.

[0076] Optionally, refer to Figure 4 Step S20 includes: Step S21: The text representation is used as the query vector, and the visual representation is used as the key vector and value vector, which are then input into the cross-attention fusion model. Step S22: The result output by the cross-attention fusion model is used as the target visual enhancement representation, which is a visual representation generated guided by the text representation.

[0077] Optionally, the text representation is used as the query vector for cross-attention computation, and the visual representation is used as the key and value vectors for cross-attention computation. It should be noted that the "text guidance" in cross-attention fusion essentially uses the text semantics of the entity to be aligned as a query signal to dynamically filter and enhance the regional features in the image that align with the text semantics. Its core is not average fusion, but semantic-driven attention focusing, concentrating attention on the region corresponding to the text semantics and enhancing the image features of that region. The enhanced visual representation is then used as the visual enhancement representation; that is, the visual enhancement representation is an enhanced visual representation generated with the text representation as guidance.

[0078] In another embodiment, when image information contains noise, background, or incorrect associations, the visually enhanced visual representation still contains noise. Directly aligning entities with the noisy visual representation can easily lead to misalignment. Therefore, after obtaining the visually enhanced representation, this embodiment proposes an adaptive gating network. The gating network calculates the gating value of the visually enhanced representation. A higher semantic consistency between the visually enhanced representation and the text representation results in a higher gating value, assigning a higher weight to the visually enhanced representation, and thus a greater visual contribution. Conversely, a lower semantic consistency between the visually enhanced representation and the text representation results in a lower gating value, assigning a lower weight to the visually enhanced representation, and thus a smaller visual contribution, to avoid misalignment caused by erroneous graphic information. Optionally, refer to... Figure 5 Step S30 includes: Step S31: Input the text representation and the target visual enhancement representation as input information into the gating network; Step S32: Obtain the target gating value generated by the gating network based on the input information, wherein the target gating value is positively correlated with the degree of matching between the text representation and the target visual enhancement representation.

[0079] Optionally, the gating value output by the gating network is a scalar between 0 and 1. It is calculated by concatenating the attention scores corresponding to the text representation and the visually enhanced representation after cross-attention fusion, then linearly projecting it through a learnable weight matrix and bias term, and finally outputting the final gating value through the sigmoid activation function sigma.

[0080] In this embodiment, after generating the text representation and the target visual augmentation representation, both are input into a gating network. The gating network concatenates the text representation and the target visual augmentation representation, then performs a linear projection through a weight matrix and bias term, and finally outputs a target gating value through the sigmoid activation function sigma. It can be understood that this target gating value is used to characterize the semantic consistency between the target visual augmentation representation and the text representation. Higher semantic consistency between the visual augmentation representation and the text representation indicates a higher degree of matching between them, meaning the target visual augmentation representation is more reliable and should be assigned a higher weight, thus allowing more visual augmentation representations to be included in the calculation of the fused representation. Conversely, lower semantic consistency indicates a lower degree of matching between them, meaning the visual augmentation representation is less reliable and should be assigned a smaller weight.

[0081] In yet another alternative implementation, refer to Figure 6 After determining the target gate value, step S40 includes: Step S41: Use the target gate value as the target weight value corresponding to the target visual enhancement representation, and perform weighted fusion on the target visual enhancement representation; Step S42: Perform a fusion operation on the weighted fused target visual enhancement representation and the text representation to generate the target fused representation.

[0082] Optionally, after determining the target gating value, the target gating value is used as the weight corresponding to the target visual enhancement, and the target visual enhancement representation is weighted and fused based on the target gating value. Specifically, the target visual enhancement representation is multiplied by the target gating value, and the product result is added to the sum of the text representation as the fused representation of the entity to be aligned. It can be understood that this mechanism uses cross-attention computation with text vectors as queries and visual vectors as keys and values ​​to obtain a visual enhancement representation consistent with the text semantics, and dynamically adjusts the injection intensity of the visual representation by outputting the gating value through the gating network. When the image information is consistent with the text semantics and is available, the visual contribution is increased; when the image is missing or has high noise, the visual influence is suppressed, so that the final entity representation has the ability to adaptively control the reliability of visual evidence without the need for manual rules, thereby reducing the risk of visual negative transfer and improving alignment robustness.

[0083] Optionally, after generating the target fusion representation, target entities matching the target fusion representation are obtained from the target knowledge graph based on the target fusion representation, and target alignment relationships corresponding to the entities to be aligned are generated based on the target entities.

[0084] In this embodiment, a text-centric cross-attention fusion mechanism based on a gating network is introduced. Text representation is used to guide the enhancement of visual representation, generating a visually enhanced representation. The degree of enhancement is determined based on the gating value output by the gating network. The visually enhanced representation is then weighted and fused with the gating value. This weighted fused representation is then fused with the text representation to generate a target fused representation of the entity to be aligned. Finally, the target entity is determined from the target knowledge graph based on this target fused representation. By introducing a gating network and a text-centric cross-attention fusion mechanism, the impact of image noise on entity alignment is reduced, improving the accuracy and robustness of entity alignment.

[0085] Second Embodiment Based on the above embodiments, referring to Figure 7 Step S50 includes: Step S51: Input the target fusion representation into the retrieval pre-training model so that the retrieval pre-training model can obtain at least one candidate entity from the target knowledge graph based on the target fusion representation; Step S52: Generate a candidate entity set based on the candidate entities; Step S53: Determine the target entity corresponding to the entity to be aligned based on the candidate entity set.

[0086] Optionally, existing entity alignment operations using multimodal large models typically involve inputting both the source and target knowledge graphs into the multimodal large model. This results in a large volume of data processing required by the multimodal large model, leading to high overall costs. Therefore, this application's embodiment utilizes a lightweight retrieval pre-trained model to perform a preliminary retrieval of the target knowledge graph, generating a candidate entity set for the entities to be aligned. This compresses what could have been thousands of matching possibilities into a small, high-quality candidate entity list (e.g., 10), significantly reducing the difficulty and cost of subsequent steps.

[0087] As an optional implementation, after obtaining the target fusion representation, the retrieval pre-trained model retrieves candidate entities that match the target fusion representation from the target knowledge graph. Specifically, the retrieval pre-trained model retrieves candidate entities that match the target fusion representation by calculating the similarity between the entity to be aligned and the entity to be matched in the target knowledge graph based on the target fusion representation, and determining the entities to be matched with a similarity greater than or equal to a preset similarity threshold as candidate entities. In another embodiment, the entities to be matched can also be sorted according to the calculated similarity, and a preset number (N) of entities to be matched at the top of the sorted list can be used as candidate entities. For example, N can be 10, and the preset number can be user-defined, configured by the system, or determined by the system based on the number of matched candidate entities; this is not limited here. It should be noted that the similarity can be calculated by calculating cosine similarity or dot product similarity; this is not limited here.

[0088] Optionally, after generating a preset number of candidate entities, a candidate entity set is generated based on the candidate entities. The candidate entity set contains only the preset number of candidate entities, which greatly reduces the computational cost compared to using all entities in the target knowledge graph as candidate entities.

[0089] Optionally, when generating the candidate entity set, only the candidate entity set containing a preset number of candidate entities is input into the multimodal large model. The multimodal large model matches the entities to be aligned based on the candidate entity set to filter out the target entities that match the entities to be aligned, thus completing the entity alignment operation.

[0090] In this embodiment, a preset number of candidate entities are selected from the target knowledge graph using a lightweight retrieval pre-trained model, which limits the number of entities input to the multimodal large model and reduces the reasoning cost required for reasoning and voting in the multimodal large model.

[0091] Third Embodiment Based on the above embodiments, referring to Figure 8 Step S53 includes: Step S531: Input the target fusion representation of the entity to be aligned and the text and visual information of the candidate entities into the multimodal large model to obtain a preset number of alignment selection results; Step S532: Obtain the number of selections corresponding to each candidate entity in the alignment selection result; Step S533: The candidate entity whose selection count is greater than the preset number is determined as the target entity corresponding to the entity to be aligned.

[0092] In this embodiment of the application, to avoid uncertainty and illusion risks in the output results of the multimodal large model, the multimodal large model is controlled to perform a preset number (R) inferences on the candidate entity, and each inference generates a corresponding alignment selection result. For example, the preset number is greater than or equal to 2 times, and the preset number is preferably 5 times.

[0093] As an optional implementation method, the target fusion representation of the entity to be aligned, along with the textual and visual information of N candidate entities, are input into a multimodal large model. Specifically, the target fusion representation of the entity to be aligned, along with the textual and visual information of the candidate entities, is assembled into a structured prompt input multimodal large model according to a preset prompt template. This allows the multimodal large model to output alignment selection results within the candidate entity set. It should be noted that, in cases where images of the entity to be aligned or candidate entities are missing, a preset placeholder image is used in the prompt input to maintain consistency in the input format. The prompt instruction clearly states that the placeholder image is only for placement and does not contain valid semantic information, thereby avoiding interference from missing images on inference. For example, the preset placeholder image can be a plain white image.

[0094] Optionally, for the same entity to be aligned, the multimodal large model repeatedly performs inference a total of R times on the same candidate entity set, obtaining R alignment selection results. After obtaining a preset number of alignment selection results, the alignment selection results are counted to determine the selection count corresponding to each candidate entity. The candidate entity with a selection count greater than the preset count is taken as the target entity corresponding to the entity to be aligned. For example, if the candidate entities in this instance are A, B, C, and D, and after repeating the inference 5 times, their corresponding selection counts are 3, 1, 1, and 0, respectively, then candidate entity A is taken as the target entity. Optionally, if the vote count does not form a majority result, the inference result of this instance is discarded, and the process returns to step S20 to regenerate the candidate entity set.

[0095] In this embodiment of the application, after generating candidate entities containing a preset number of candidate entities using a retrieval pre-trained model, the same candidate entity set is repeatedly inferred multiple times using a multimodal large model, so as to reduce the uncertainty and illusion risk of the output results of the multimodal large model.

[0096] Fourth embodiment Based on the above embodiments, referring to Figure 9 After step S50, the method further includes: Step S60: Input the target alignment relationship into the retrieval pre-training model so that the retrieval pre-training model can perform an update operation based on the target alignment relationship; Step S70: Using the updated retrieval pre-trained model, return to the step of inputting the target fusion representation into the retrieval pre-trained model.

[0097] In this embodiment, the retrieval pre-training model has a self-updating function, improving its accuracy through iterative training. Specifically, the iterative training of the retrieval pre-training model involves inputting the target alignment relationship into the model for updating. The update operation includes the following steps: The target entity in the target alignment relationship is determined as the target positive matching entity of the entity to be aligned; Identify the remaining candidate entities in the candidate entity set, excluding the positive matching entity, and obtain the similarity between the remaining candidate entities and the entity to be aligned; Entities with a similarity greater than or equal to a preset similarity are identified as target negative samples; A target contrastive learning loss value is generated based on the triple consisting of the entity to be aligned, the target positive matching entity, and the target negative sample; The parameters of the retrieval pre-trained model are updated using gradients based on the target contrastive learning loss value.

[0098] As an optional implementation, after determining the target entity, the multimodal large model outputs the correspondence between the entity to be aligned and the target entity as a pseudo-alignment label, which is then used by the retrieval pre-trained model to perform comparative self-training updates. Specifically, after obtaining the target entity relationship, the pseudo-alignment result (entity to be aligned, target entity) (e) determined by the multimodal large model through reasoning and majority voting is first used. s ,e pred ) as training supervision signal, where the target entity e pred Let e ​​be the entity to be aligned. s Positive matching entity e pos Subsequently, to construct discriminative training samples, the pre-trained model performs negative sample selection from the candidate entity set of entities to be aligned, removing positively matching samples (e) from the candidate entity set through the negative sample selection operation. pos Then, based on the similarity metric in the current PTM representation space, select the entity e to be aligned. sThe candidate entity with the highest similarity is selected as the most difficult negative sample e. neg The most difficult negative sample is used as the target negative sample to ensure that the training process focuses on constraining the most easily confused incorrect matches.

[0099] It should be noted that after determining the target negative sample, a target triplet is generated based on the entity to be aligned, the positive matching entity, and the target negative sample. The target triplet is defined as (entity to be aligned, positive matching entity, target negative sample) (e s ,e pos ,e neg The target triplet is determined by ), and the contrast loss value is calculated based on the target triplet. The calculated contrast loss value is then used as the target contrast loss value.

[0100] In some optional implementations, the target contrast loss value includes a target pairing interval loss value. This target pairing interval loss value is used to constrain the similarity of positively matching samples to be significantly higher than the similarity of negative samples and to satisfy a preset interval. Specifically, it is used to determine whether the matching degree between the target positively matching entity and the target negative sample exceeds the preset similarity interval, and the similarity between the target positively matching entity and the entity to be aligned should be greater than the similarity between the target negative sample and the entity to be aligned. The target pairing interval loss value is calculated by determining a first similarity between the entity to be aligned and the target positively matching entity, and a second similarity between the entity to be aligned and the target negative sample, and then calculating the target pairing interval loss value based on the first and second similarities.

[0101] In another embodiment, the target contrast loss value may further include the target InfoNCE contrast loss (InfoNoise-Contrastive EstimationInfo contrast loss). The target InfoNoise contrast loss is used to improve the relative separability of positive samples within the candidate entity range and suppress other contrast terms.

[0102] It is understood that the target contrastive loss value may also include other types of contrastive loss values, such as symmetric cross-entropy loss values, which are not limited here.

[0103] In this embodiment of the application, the target contrast loss value includes the target pairing interval loss value and / or the target information noise contrast loss value. When the target contrast loss value includes the target pairing interval loss value and the target information noise contrast loss value, the target pairing interval loss value and the target information noise contrast loss value are combined, and the total loss value after combination is used as the target contrast loss value. Then, the retrieval pre-trained model is updated based on the target contrast loss value.

[0104] Optionally, based on the updated retrieval pre-trained model, execution step S20 is returned to regenerate the candidate set, realizing a closed-loop iteration of repeated retrieval—inference voting—self-training update, thereby gradually improving the retrieval and alignment effect.

[0105] In this embodiment of the application, the retrieval and alignment performance of the retrieval pre-training model is improved by iteratively training and updating the retrieval pre-training model using contrastive loss values.

[0106] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the entity alignment method of this application. Any simple transformations based on this technical concept are within the protection scope of this application.

[0107] This application provides an entity alignment device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the entity alignment method in Embodiment 1 above.

[0108] The following is for reference. Figure 10 The diagram illustrates a structural schematic suitable for implementing the entity alignment device in the embodiments of this application. The entity alignment device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 10 The entity alignment device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0109] like Figure 10As shown, the physical alignment device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the physical alignment device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the physical alignment device to communicate wirelessly or wiredly with other devices to exchange data. While the figures show physical alignment devices with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0110] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0111] The entity alignment device provided in this application, employing the entity alignment method in the above embodiments, can solve the technical problem of misalignment caused by low image information quality. Compared with the prior art, the beneficial effects of the entity alignment device provided in this application are the same as those of the entity alignment method provided in the above embodiments, and other technical features of this entity alignment device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0112] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0113] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0114] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the entity alignment method in the above embodiments.

[0115] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0116] The aforementioned computer-readable storage medium may be included in the entity alignment device; or it may exist independently and not assembled into the entity alignment device.

[0117] The aforementioned computer-readable storage medium carries one or more programs that, when executed by an entity alignment device, enable the entity alignment device to write computer program code for performing the operations of this application in one or more programming languages ​​or a combination thereof. These programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—and conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0118] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0119] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0120] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described entity alignment method, which can solve the technical problem of misalignment caused by low image information quality. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the entity alignment method provided in the above embodiments, and will not be repeated here.

[0121] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the entity alignment method described above.

[0122] The computer program product provided in this application can solve the technical problem of misalignment caused by low image information quality. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the entity alignment method provided in the above embodiments, and will not be repeated here.

[0123] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A method for aligning entities, characterized in that, The steps of the entity alignment method include: Encode the textual and visual information of the entities to be aligned in the source knowledge graph to generate corresponding textual and visual representations; The target visual enhancement representation corresponding to the visual representation is generated based on the text representation; The target gating value corresponding to the target visual enhancement representation is determined based on the gating network; Generate a target fusion representation of the entity to be aligned based on the target gating value, the text representation, and the target visual enhancement representation; Based on the target fusion representation, target entities matching the target fusion representation are obtained from the target knowledge graph, and target alignment relationships corresponding to the entities to be aligned are generated based on the target entities.

2. The entity alignment method as described in claim 1, characterized in that, The step of generating the target visual enhancement representation corresponding to the visual representation based on the text representation includes: The text representation is used as the query vector, and the visual representation is used as the key vector and value vector, which are then input into the cross-attention fusion model. The output of the cross-attention fusion model is used as the target visual enhancement representation, which is a visual representation generated guided by the text representation.

3. The entity alignment method as described in claim 1, characterized in that, The step of determining the target gating value corresponding to the target visual augmentation representation based on the gating network includes: The text representation and the target visual enhancement representation are input as input information to the gating network; Obtain the target gating value generated by the gating network based on the input information, wherein the target gating value is positively correlated with the degree of matching between the text representation and the target visual enhancement representation.

4. The entity alignment method as described in claim 1, characterized in that, The step of generating the target fusion representation of the entity to be aligned based on the target gating value, the text representation, and the target visual enhancement representation includes: The target gate value is used as the target weight value corresponding to the target visual enhancement representation, and the target visual enhancement representation is weighted and fused. The weighted and fused visual representation of the target is fused with the text representation to generate the fused representation of the target.

5. The entity alignment method as described in claim 1, characterized in that, The step of obtaining the target entity matching the target fusion representation from the target knowledge graph based on the target fusion representation includes: The target fusion representation is input into the retrieval pre-training model so that the retrieval pre-training model can obtain at least one candidate entity from the target knowledge graph based on the target fusion representation; A candidate entity set is generated based on the candidate entities; The target entity corresponding to the entity to be aligned is determined based on the candidate entity set.

6. The entity alignment method as described in claim 5, characterized in that, The step of determining the target entity corresponding to the entity to be aligned based on the candidate entity set includes: The target fusion representation of the entity to be aligned, along with the textual and visual information of the candidate entities, is input into a multimodal large model to obtain a preset number of alignment selection results. Obtain the number of selections for each candidate entity in the alignment selection results; Candidate entities selected more than a preset number of times are identified as the target entities corresponding to the entities to be aligned.

7. The entity alignment method according to any one of claims 1-6, characterized in that, After the step of generating the target alignment relationship corresponding to the entity to be aligned based on the target entity, the method further includes: The target alignment relationship is input into the retrieval pre-training model so that the retrieval pre-training model can perform an update operation based on the target alignment relationship; The updated retrieval pre-trained model is used to return the steps of inputting the target fusion representation into the retrieval pre-trained model.

8. The entity alignment method as described in claim 7, characterized in that, The steps of the retrieval pre-trained model performing the update operation based on the target alignment relationship include: The target entity in the target alignment relationship is determined as the target positive matching entity of the entity to be aligned; Identify the remaining candidate entities in the candidate entity set, excluding the positive matching entity, and obtain the similarity between the remaining candidate entities and the entity to be aligned; Entities with a similarity greater than or equal to a preset similarity are identified as target negative samples; A target contrastive learning loss value is generated based on the target triplet consisting of the entity to be aligned, the target positive matching entity, and the target negative sample. The parameters of the retrieval pre-trained model are updated using gradients based on the target contrastive learning loss value.

9. A solid alignment device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the entity alignment method as described in any one of claims 1 to 8.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the entity alignment method as described in any one of claims 1 to 8.

Images