Entity alignment model training method and device

Abstract

The invention relates to the technical field of natural language processing, and provides a training method and device of an entity alignment model, which are used for improving the performance of entity alignment. According to the model, multiple modal features are extracted from each entity in the entity sample pair, so that multivariate information of the entities is fully utilized, the feature expression ability of the entities is improved, and the accuracy of the entity alignment model is improved. On the other hand, aiming at the imbalance problem among multiple modal features, based on the multiple modal features of each entity, a first mixed feature used for multi-view comparative learning and a second mixed feature used for single-view comparative learning are generated respectively, and the alignment loss is calculated in combination with the original multiple modal features and the alignment label; therefore, the entity alignment model can dynamically learn differences in the modal features and between the modal features, the difference influence caused by different modal features is reduced, and the alignment precision of the entity alignment model is improved.

Description

Background Art

[0002] In the field of natural language processing, entity alignment algorithms are mainly divided into two categories: one is based on translation models, and the other is based on graph representation learning. Among them, the graph representation learning-based method benefits from the ability of graph neural networks to aggregate node neighborhood information to generate graph representation vectors, and has greater advantages in processing graph structure data and better modeling effects on relational entities.

[0003] Currently, an existing entity alignment method encodes different types of data distribution across knowledge graphs to obtain feature representation of multivariate data. However, since this method cannot capture the semantic information of the relationship between entities, the accuracy of entity alignment is still low. Summary of the invention

[0004] The embodiments of the present application provide a method and device for training an entity alignment model to improve the accuracy of entity alignment.

[0005] In a first aspect, an embodiment of the present application provides a method for training an entity alignment model, comprising:

[0006] Constructing a training sample set, wherein the alignment label of each entity sample pair in the training sample set is used to indicate whether two entities in the corresponding entity sample pair originating from different knowledge graphs are aligned;

[0007] According to the training sample set, the entity alignment model is trained for multiple rounds of iterations until the model converges, wherein each round of iteration training includes:

[0008] Extract multiple modal features of two entities in multiple entity sample pairs respectively;

[0009] Performing multi-view cross-attention calculations on the multiple modal features of each of the two entities in the multiple entity sample pairs to obtain a first mixed feature, and performing single-view merging on the multiple modal features of each of the two entities in the multiple entity sample pairs to obtain a second mixed feature;

[0010] According to the multiple modal features, the first mixed features and the second mixed features of each of the two entities in the multiple entity sample pairs, combined with the corresponding alignment labels, a target loss value is determined, and the model parameters are adjusted according to the target loss value.

[0011] In a second aspect, an embodiment of the present application provides a training device for an entity alignment model, comprising:

[0012] A sample construction unit, used to construct a training sample set, wherein the alignment label of each entity sample pair in the training sample set is used to indicate whether two entities in the corresponding entity sample pair originating from different knowledge graphs are aligned;

[0013] The iterative training unit is used to perform multiple rounds of iterative training on the entity alignment model according to the training sample set until the model converges, wherein each round of iterative training includes:

[0014] Extract multiple modal features of two entities in multiple entity sample pairs respectively;

[0015] Performing cross-attention calculations on the multiple modal features of each of the two entities in the multiple entity sample pairs to obtain a first mixed feature, and merging the multiple modal features of each of the two entities in the multiple entity sample pairs to obtain a second mixed feature;

[0016] According to the multiple modal features, the first mixed features and the second mixed features of each of the two entities in the multiple entity sample pairs, combined with the corresponding alignment labels, a target loss value is determined, and the model parameters are adjusted according to the target loss value.

[0017] Optionally, the multiple modal features include at least a graph structure embedding feature, and the iterative training unit is specifically used for:

[0018] For the current entity in any entity sample pair, perform the following operations:

[0019] A multi-head attention mechanism is used to extract node features of the current entity and adjacent entities of the current entity, and a multi-head attention mechanism is used to extract edge features between the current entity and the adjacent entities, and an attention coefficient is determined according to the node features and the edge features; the edge features represent the association relationship between the current entity and the adjacent entities;

[0020] The node features of the adjacent entities of the current entity are aggregated according to the attention coefficient to obtain the graph structure embedding features of the current entity in the knowledge graph.

[0021] Optionally, the multiple modal features include at least a richness embedding feature, and the iterative training unit is specifically used for:

[0022] For the current entity in any entity sample pair, perform the following operations:

[0023] Acquire the association relationship between the current entity and the adjacent entity and the attribute of the current entity, and count the association relationship and the additional information, wherein the attribute represents the additional information of the current entity;

[0024] The counting result is one-hot encoded to obtain the richness embedding feature of the current entity.

[0025] Optionally, the multiple modal features include at least a relation embedding feature, an attribute embedding feature, and a visual embedding feature, and the iterative training unit is specifically used for:

[0026] For the current entity in any entity sample pair, perform the following operations:

[0027] Using a bag-of-words model, according to the association between the current entity and adjacent entities in the knowledge graph, a relational embedding feature of the current entity is generated; and, using a bag-of-words model, according to the attributes of the current entity, an attribute embedding feature of the current entity is generated, wherein the attribute represents additional information of the current entity;

[0028] A pre-trained visual model is used to generate a visual embedding feature of the current entity according to pixel information of the current entity in the image.

[0029] Optionally, the iterative training unit is specifically used for:

[0030] For the current entity in any entity sample pair, perform the following operations:

[0031] splicing multiple modal features of the current entity to obtain intermediate features;

[0032] According to the three weight matrices of the multiple attention heads, the intermediate features are transformed respectively to obtain a query matrix, a key matrix and a value matrix across perspectives;

[0033] Determining a multi-view cross-attention coefficient according to the query matrix and the key matrix;

[0034] Determining a multi-view modal feature according to the multi-view cross-attention coefficient and the value matrix;

[0035] The intermediate features and the multi-view modal features are subjected to residual connection and standardization processing to obtain a first mixed feature of the current entity.

[0036] Optionally, the iterative training unit is specifically used for:

[0037] For the current entity in any entity sample pair, perform the following operations:

[0038] Determining respective weighting weights according to the information density corresponding to each of the multiple modal features of the current entity;

[0039] According to the respective weighted weights, the multiple modal features of the current entity are combined to obtain a second mixed feature.

[0040] Optionally, the iterative training unit is specifically used for:

[0041] Determining a modality intra-sub-loss value for single-view contrastive learning according to the respective multiple modal features and alignment labels of two entities in the multiple entity sample pairs;

[0042] Determining an inter-modal sub-loss value of multi-view contrastive learning according to the respective multiple modal features, the first mixed feature, and the alignment label of the two entities in the multiple entity sample pairs;

[0043] Determining a multimodal joint sub-loss value for single-view contrastive learning according to respective second mixed features and alignment labels of two entities in the plurality of entity sample pairs;

[0044] The target loss value is determined according to the intra-modal sub-loss value, the inter-modal sub-loss value and the multi-modal joint sub-loss value.

[0045] In a third aspect, an embodiment of the present application provides an electronic device, including a processor, a memory, and a communication interface, wherein the communication interface, the memory, and the processor are connected via a bus;

[0046] The communication interface is used to send and receive data;

[0047] The memory is used to store computer instructions;

[0048] The processor is used to read the computer instructions in the memory and execute the steps of any one of the entity alignment model training methods in the first aspect.

[0049] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which includes a computer program. When the computer program is run on an electronic device, the computer program is used to enable the electronic device to execute the steps of any one of the entity alignment model training methods in the first aspect above.

[0050] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program, and the computer program is stored in a computer-readable storage medium; when a processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, so that the electronic device performs the steps of any one of the entity alignment model training methods in the first aspect above.

[0051] The beneficial effects of the training method and device of the entity alignment model provided in this application are:

[0052] By extracting multiple modal features from each entity in the entity sample pair, the multivariate information of the entity is fully utilized, the feature expression ability of the entity is improved, and the accuracy of the entity alignment model is improved. On the other hand, in order to address the imbalance problem between multiple modal features, based on the multiple modal features of each entity, the first mixed feature for multi-view contrastive learning and the second mixed feature for single-view contrastive learning are generated respectively, and then the loss value is calculated by combining the original multiple modal features and alignment labels, so that the entity alignment model can dynamically learn the differences within and between modal features, reduce the difference caused by different modal features, and improve the alignment accuracy of the entity alignment model.

[0053] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0055] Figure 1 The following is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0056] Figure 2 An algorithm framework for the entity alignment model provided in the embodiments of the present application;

[0057] Figure 3 A flow chart of a training method for an entity alignment model provided in an embodiment of the present application;

[0058] Figure 4 A flow chart of the modal feature fusion method provided in the embodiment of the present application;

[0059] Figure 5 A schematic diagram of the temperature hyperparameter optimization process provided in an embodiment of the present application;

[0060] Figure 6 A schematic diagram of adjusting the attention coefficient provided in an embodiment of the present application;

[0061] Figure 7 A comparison diagram of different models provided in the embodiments of the present application;

[0062] Figure 8A diagram showing the model comparison results in a low-resource scenario provided in an embodiment of the present application;

[0063] Fig. 9 A device structure diagram of a physical alignment model provided in an embodiment of the present application;

[0064] Fig.10 A structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical solutions and advantages of this application clearer, this application will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0066] Based on the exemplary embodiments shown in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application. In addition, although the disclosure in this application is introduced according to one or several exemplary examples, it should be understood that each aspect of these disclosures can also constitute a complete technical solution separately.

[0067] It should be understood that the terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, for example, they can be implemented according to an order other than those given in the diagrams or descriptions of the embodiments of this application.

[0068] In addition, the terms "include" and "have" and any variations thereof are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to those components expressly listed but may include other components not expressly listed or inherent to such products or devices.

[0069] In order to clearly describe the embodiments of the present application, the nouns in the embodiments of the present application are explained below.

[0070] GCN (Graph Convolutional Networks): Contains multiple convolutional layers, each of which only processes first-order neighborhood information. By stacking several convolutional layers, multi-order neighborhood information can be transmitted.

[0071] GAT (Graph Attention Network): uses the attention mechanism to learn the relationship between nodes, thereby achieving tasks such as node classification and node-level feature learning on graph data.

[0072] PVM (Pre-trained Visual Model): refers to a large model trained with a large amount of image data, which can be used directly or after fine-tuning on new tasks.

[0073] NLP (Natural Language Processing) is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can achieve effective communication between people and computers using natural language. Natural language processing is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field will involve natural language, that is, the language people use in daily life, so it is closely related to the study of linguistics. Natural language processing technology usually includes text processing, semantic understanding, machine translation, robot question answering, knowledge graph and other technologies.

[0074] The preferred embodiments of the present application are described below in conjunction with the drawings in the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application. In addition, the embodiments and features in the embodiments of the present application may be combined with each other if there is no conflict.

[0075] Figure 1 The application scenario diagram provided by the embodiment of the present application is exemplarily shown, and the application scenario diagram includes two terminal devices 110 and a server 120.

[0076] In the embodiments of the present application, the terminal device 110 includes but is not limited to mobile phones, tablet computers, laptop computers, desktop computers, e-book readers, intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals and other devices; a voice interaction client may be installed on the terminal device, which may be software (such as a browser, game software, etc.), or a web page, a small program, etc. The server 120 is a background server corresponding to the software or web page, small program, etc., or a server specifically used for voice interaction, which is not specifically limited in the present application. The server 120 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0077] It should be noted that the training method of the entity alignment model in each embodiment of the present application can be executed by an electronic device, which can be a terminal device 110 or a server 120, that is, the method can be executed by the terminal device 110 or the server 120 alone, or can be executed by the terminal device 110 and the server 120 together. For example, when executed by the server 120 alone, the server 120 performs multiple rounds of iterative training on the entity alignment model based on the constructed training sample set until the model converges, wherein the alignment label of each entity sample pair in the training sample set is used to indicate whether the two entities from different knowledge graphs in the corresponding entity sample pair are aligned, and each round of iterative training includes: extracting multiple modal features of each of the two entities in the multiple entity sample pairs; performing multi-view cross-attention calculations on the multiple modal features of each of the two entities in the multiple entity sample pairs to obtain a first mixed feature, and performing single-view merging on the multiple modal features of each of the two entities in the multiple entity sample pairs to obtain a second mixed feature; determining the target loss value according to the multiple modal features of each of the two entities in the multiple entity sample pairs, the first mixed feature, and the second mixed feature, combined with the corresponding alignment label, and adjusting the model parameters according to the target loss value.

[0078] After the entity alignment model training is completed, when the terminal device 110 inputs a query text, the server 120 can parse the entities to be aligned in the query text based on the trained entity alignment model, and determine synonymous entities with the same semantics as the entities to be aligned, thereby responding to the query based on the synonymous entities.

[0079] In addition, when the method provided by the embodiment of the present application is executed by a terminal device alone, its execution method is similar to the process of the server executing alone, mainly replacing the server with a terminal device. In addition, when the method provided by the embodiment of the present application is executed by a terminal device and a server in cooperation, the steps that need to be reflected on the front-end interface can be executed by the terminal device, and some steps that require background calculations and do not need to be reflected on the front-end interface can be executed by the server.

[0080] In an optional implementation, the terminal device 110 and the server 120 may communicate with each other via a communication network.

[0081] In an optional implementation, the communication network is a wired network or a wireless network.

[0082] It should be noted that Figure 1 The figure is only an example. In fact, the number of terminal devices and servers is not limited and is not specifically limited in the embodiments of the present application.

[0083] In an embodiment of the present application, when there are multiple servers, the multiple servers can form a blockchain, and the servers are nodes on the blockchain; as in the training method of the entity alignment model disclosed in the embodiment of the present application, the entity data involved can be stored on the blockchain, for example, a knowledge graph, multiple modal features, etc.

[0084] In addition, the embodiments of the present application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving and other scenarios.

[0085] The following describes the control method of the virtual character provided by the exemplary embodiment of the present application in combination with the application scenarios described above and with reference to the accompanying drawings. It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principles of the present application, and the implementation methods of the present application are not limited in this respect.

[0086] See also Figure 2 As shown, the algorithm framework of the entity alignment model provided in the embodiment of the present application realizes the entity alignment algorithm based on multi-view feature fusion (EAMF). In this model, considering that the source knowledge graph and the target knowledge graph usually contain information of multiple modalities, in order to balance the differences and biases of information between modalities, different feature extraction methods are applied to different modal information for extraction, mainly including three feature extractors, the first is a graph convolutional neural network used to extract graph structure information, the second is a bag of words model (Bag of Words, BOW) used to extract entity relationships, attributes and richness information in the knowledge graph, and the third is a pre-trained visual model used to extract entity visual information, wherein, by introducing an encoding mechanism for richness information, the feature expression of cross-modal information is optimized, thereby further refining the feature description under each modality. In addition, a multi-view cross-attention mechanism is adopted to jointly encode multiple modal features to promote the deep fusion of different modal features. A single-view contrastive learning strategy is implemented within various modal features to extract clear intra-modal entity features, while a multi-view contrastive learning strategy is adopted between modal features to promote information exchange and complementarity between modal features. During contrastive learning, the uncertainty weighted loss function is used in combination with the self-restraint mechanism (SrM) to enhance the robustness of the algorithm, thereby enhancing the performance of the model in cross-modal entity representation learning.

[0087] The EAMF model provided in the embodiment of the present application incorporates the graph structure of edge information modeling in the multimodal feature extraction part, regards the knowledge graph as a heterogeneous graph, and adds richness information, thereby enhancing the ability to extract multivariate features and effectively addressing the problem of insufficient utilization of entity multivariate information in the entity alignment algorithm. Further, based on single-multi-view contrast learning, a self-constraint mechanism is used to jointly optimize the hybrid features, features within a single modality, and feature representations between multiple modalities, effectively alleviating the problem of unbalanced information differences between single and multimodal in existing entity alignment algorithms.

[0088] See also Figure 3 The training process of the entity alignment model provided in the embodiment of the present application mainly includes the following steps:

[0089] S301: Construct a training sample set.

[0090] Each node in the knowledge graph represents an entity, following the 1-to-1 alignment constraint criterion. For two entities from different knowledge graphs, the entity sample pair consisting of the two entities can be pre-labeled with alignment labels based on their semantic information. Specifically, if the semantics of the two entities in the entity sample pair are the same or similar, indicating that the two entities have been aligned, the entity sample pair is labeled with an aligned label. If the semantics of the two entities in the entity sample pair are different and dissimilar, indicating that the two entities are not aligned, the entity sample pair is labeled with an unaligned label. Among them, the entity sample pairs marked with aligned labels are used as positive samples for entity alignment model training, and the entity sample pairs marked with unaligned labels are used as negative samples for entity alignment model training.

[0091] Generally, in the field of NLP, in order to distinguish the knowledge graphs where two entities in an entity sample pair are located, one knowledge graph is recorded as the target knowledge graph and the other knowledge graph is recorded as the source knowledge graph. After labeling multiple entity sample pairs, the construction of the training sample set is completed.

[0092] S302: Perform multiple rounds of iterative training on the entity alignment model according to the training sample set until the model converges.

[0093] Among them, the conditions for model convergence include but are not limited to the loss value of the model being less than a preset threshold, the number of iterations reaching a preset number, etc.

[0094] Specifically, each round of iterative training includes the following steps:

[0095] S3021: Extract multiple modal features of two entities in multiple entity sample pairs respectively.

[0096] For multiple entity sample pairs in the same batch, different feature extractors are used to extract multiple modal features.

[0097] In one example, the multiple modal features include graph structure embedding features. Specifically, the extraction process of the graph structure embedding features includes:

[0098] First, for the current entity contained in any entity sample pair of multiple entity sample pairs in the same batch, a multi-head attention mechanism is used to extract the node features of the current entity and the adjacent entities of the current entity, and a multi-head attention mechanism is used to extract the edge features between the current entity and the adjacent entities, and the attention coefficient is determined according to the node features and edge features. Among them, the current entity is an entity from the target knowledge graph or an entity from the source knowledge graph, and the edge features represent the association relationship between the current entity and the adjacent entity.

[0099] Generally, a knowledge graph is composed of multiple nodes and edges connecting the nodes, where each node represents an entity and the edges connecting the nodes represent the relationship between entities. In the field of graph structure information processing, GNN is a widely used technology, so this embodiment selects GAT in GNN to model the structural information of the knowledge graph. However, the knowledge graph can essentially be regarded as a heterogeneous graph. The traditional GAT mainly models homogeneous graphs. If it is directly applied to the knowledge graph, it cannot fully express the diverse node and edge types in the graph.

[0100] In order to solve the problems of traditional GAT in extracting graph structure information, the embodiment of the present application designs an improved edge information enhanced graph attention network (Edge Information Enhanced GAT, eGAT). The design of eGAT takes into account the importance of edge information in the knowledge graph and optimizes the algorithm by giving equal weight to each edge representing the association relationship. Among them, eGAT uses a multi-head attention mechanism to capture the features of nodes in the knowledge graph and the adjacent nodes connected to the nodes. At the same time, a multi-head attention mechanism is used to capture the edge features of the association relationship between nodes and adjacent nodes, and the node features and edge features are modeled in a global connection manner.

[0101] Specifically, the calculation formula for the attention coefficient between nodes is:

[0102]

[0103] in, is the weight matrix of the K linear mapping heads, is the weight matrix of a specific feature, d represents the dimension of the feature, e i The node representing the current entity, e j Indicates the adjacent nodes corresponding to the adjacent entities of the current entity, r i Represents the node e i Connected edges, h lis the vector representation of the lth layer, is the attention coefficient, is node e i The set of adjacent nodes.

[0104] In the knowledge graph, the relationship between nodes is relatively complex and the types of connected adjacent nodes are different. When extracting graph structure embedding features, the attention coefficient between nodes can be dynamically adjusted by adding edge features, which effectively helps eGAT extract graph structure information and thus improve the entity alignment effect.

[0105] Then, the node features of the adjacent entities of the current entity are aggregated according to the attention coefficient to obtain the graph structure embedding features of the current entity in the knowledge graph.

[0106] Specifically, eGAT obtains neighborhood structure embedding by capturing and aggregating adjacent nodes of entity nodes, and uses the output as graph structure embedding features. The final graph structure embedding feature formula is expressed as follows:

[0107]

[0108] Here, ⊕ represents the stack of attention heads and σ is the activation function.

[0109] Optionally, in order to obtain stable and effective feature aggregation results, the number of attention heads K in eGAT can be set to 2.

[0110] Optionally, in order to avoid the over-smoothing problem caused by the multi-layer attention graph network, the number of network layers in eGAT can be set to l = 1.

[0111] In the embodiments of the present application, the connection characteristics between nodes are strengthened by integrating the edge feature information in the knowledge graph, thereby improving the relationship perception ability of eGAT, thereby more accurately reflecting the complex structural relationships in the knowledge graph and improving the performance of the algorithm when processing heterogeneous information.

[0112] Since the graph structure has the characteristic of information diversity, it is impossible to fully understand the semantic information of the entity without distinguishing the categories of the data. In order to better understand the semantics of the entity, in one example, the multiple modal information includes at least one of a relation embedding feature and an attribute embedding feature.

[0113] Taking the current entity contained in any entity sample pair of multiple entity sample pairs in the same batch as an example, when constructing a training sample set based on the knowledge graph, the association relationship between the current entity and the adjacent entities connected to the current entity, as well as the attributes of the current entity (referring to the additional information of the current entity, including but not limited to category, occupation, shape, etc.) have been defined in the training sample set. Therefore, the bag-of-words model can be used to generate the relational embedding features of the current entity based on the association relationship between the current entity and the adjacent entities in the knowledge graph. Similarly, the bag-of-words model can be used to generate the attribute embedding features of the current entity based on the attributes of the current entity.

[0114] Considering that it is a huge task to pre-label the entity sample pairs of the source knowledge graph and the target knowledge graph, there is a phenomenon that most overlapping relationships are not labeled. Similarly, the same entities in the source knowledge graph and the target knowledge graph also have a large number of overlapping attributes. In addition, relying solely on graph structure embedding features, relationship embedding features, and attribute embedding features cannot achieve a good alignment effect.

[0115] To this end, in one example, the multiple modal features also include a richness embedding feature. Specifically, still taking the current entity included in any one of the multiple entity sample pairs in the same batch as an example, the extraction process of the richness embedding feature includes:

[0116] First, the association relationship between the current entity and the adjacent entity and the attributes of the current entity are obtained, and the association relationship and attributes are counted.

[0117] Then, the counting results are one-hot encoded to obtain the richness embedding features of the current entity.

[0118] The entity alignment model EAMF of the embodiment of the present application uses the bag-of-words model to extract features from the associations and attributes of entities, and also uses the bag-of-words model to introduce richness embedding features. By counting the modal information such as the associations and attributes of each entity, and counting the modal information such as the associations and attributes of each entity, and then inputting them into the forward feedback neural network in the entity alignment model EAMF in a unique hot encoding manner, the association embedding features, attribute embedding features and richness embedding features are generated, and the formula is expressed as follows:

[0119]

[0120] in, They represent the relationship embedding feature, attribute embedding feature and richness embedding feature respectively, and W m and b m are the learnable matrices weights and biases of a feedforward neural network, is the association feature in the bag-of-words model. is the attribute feature in the bag-of-words model, It is the richness feature in the bag-of-words model.

[0121] In the entity alignment model EAMF, the bag-of-words model is used to introduce entity association embedding features, attribute embedding features, and richness embedding features, thereby enhancing the semantic expression effect of the entity and improving the alignment accuracy of the model.

[0122] For the image modal information of the entity, the pre-trained visual model is used in the entity alignment model EAMF to generate the visual embedding features of the current entity according to the pixel information of the current entity in the image.

[0123] Optionally, the pre-trained visual model uses ResNet-152 as a feature extractor to describe the image-level information of the entity, uses the image features extracted by the last layer of ResNet-152 as output, and inputs the image features into the feedforward neural network to obtain the visual embedding features of the entity, which is expressed as:

[0124]

[0125] in, Node e representing the current entity i The image features, W v and b v are the learnable matrix weights and biases of the feedforward neural network, Represents node e i Visual embedding features.

[0126] S3022: Perform multi-view cross-attention calculations on the multiple modal features of each of the two entities in the multiple entity sample pairs to obtain a first mixed feature, and perform single-view merging on the multiple modal features of each of the two entities in the multiple entity sample pairs to obtain a second mixed feature.

[0127] The entity alignment model EAMF includes a Multi-view Cross-Attention Fusion (MACF) module, which can dynamically generate mutual attention weights for multiple modal features of an entity. By performing cross-attention operations between different modal features, the mutual scores between modal features are obtained, thereby constructing a more reasonable and comprehensive multimodal node feature at multiple perspective levels to perceive the modal preferences shown by entities with different information content.

[0128] Specifically, taking the current entity contained in any entity sample pair in the same batch as an example, the process of multi-view cross-attention fusion between different modal features is as follows: Figure 4 As shown, it mainly includes the following steps:

[0129] S3022_1: Concatenate multiple modal features of the current entity to obtain intermediate features.

[0130] Among them, the formula of the intermediate feature is expressed as follows:

[0131]

[0132] In the formula Represents the node e of the current entity respectively i There are five modal features: graph structure embedding features, relationship association embedding features, attribute embedding features, richness embedding features and visual embedding features.

[0133] S3022_2: According to the three weight matrices of multiple attention heads, the intermediate features are transformed respectively to obtain the query matrix, key matrix and value matrix across perspectives.

[0134] Assuming that MCAF adopts a network structure containing N attention heads, the three weight matrices corresponding to each attention head are defined as The transformation process formula of the intermediate features is expressed as follows:

[0135]

[0136] in, They represent the query matrix, key matrix and value matrix respectively, and n∈N represents the nth head.

[0137] S3022_3: Determine the multi-view cross-attention coefficient based on the query matrix and the key matrix.

[0138] In each attention head, the formula of the multi-view cross-attention coefficient is expressed as follows:

[0139]

[0140] Where s1 and s2 represent any two modes in the multi-mode set S = {g, r, a, n, v}, represents the query matrix corresponding to the s1 modality, represents the bond matrix corresponding to the s2 mode. Among them, The calculation process of can represent the process of independently calculating the relative importance weights of different modal features of each entity in the same batch of entity sample pairs.

[0141] S3022_4: Determine multi-view modal features based on multi-view cross-attention coefficients and value matrices.

[0142] In each attention head, the formula of multi-view modality features is expressed as follows:

[0143]

[0144] In the formula Represents the value matrix corresponding to the mode of s1.

[0145] S3022_5: Perform residual connection and standardization processing on the intermediate features and multi-view modal features to obtain the first mixed features of the current entity.

[0146] Among them, the formula of the first mixed feature is expressed as follows:

[0147]

[0148] The first hybrid feature obtained by multi-view cross-attention fusion It can be used in the multi-view contrastive learning scenario of the entity alignment model EAMF. It stabilizes the training process through residual connection and standardization, and enhances the robustness of the algorithm model.

[0149] In the entity alignment model EAMF, the relative importance weights of different modal features of each entity in the same batch of entity sample pairs are calculated independently, realizing an effective mechanism to parse and integrate multiple modal features of entity nodes from multiple perspectives, ensuring that the algorithm model can flexibly respond to the needs of entity expressions with different modal information.

[0150] Considering that the first cross-perspective hybrid features generated by MCAF in the late fusion stage will weaken the performance of distinguishing entities in the single-perspective contrastive learning scenario, the embodiment of the present application generates a second hybrid feature suitable for single-perspective contrastive learning based on multiple modal features of the current entity.

[0151] Specifically, still taking the current entity contained in any entity sample pair in the same batch of multiple entity sample pairs as an example, the process of single-view fusion between different modal features is as follows: Figure 4 As shown, it mainly includes the following steps:

[0152] S3022_6: Determine each weighted weight according to the information density corresponding to each of the multiple modal features of the current entity.

[0153] S3022_7: According to each weighted weight, multiple modal features of the current entity are merged to obtain a second mixed feature.

[0154] In the entity alignment model EAMF, an integrated weight distribution strategy is adopted. On the basis of fully considering the density of different modal information, multiple modal features are converged and merged to construct a second hybrid feature suitable for single-view contrastive learning. The formula of the second hybrid feature is as follows:

[0155]

[0156] Where ⊕ represents matrix stacking, S = {g, r, a, n, v}, Node e representing the current entity i In the characteristics of s1 mode, and Represents the weights of different modes.

[0157] In an embodiment of the present application, features from different modalities are uniformly projected into a common representation space through the fusion of the first hybrid feature and the second hybrid feature, which promotes the fusion of cross-modal features and helps to deepen the perception of single modal characteristics, thereby enabling the algorithm model to more comprehensively and accurately capture and grasp the multimodal information of entities in multiple dimensions, thereby improving the alignment accuracy of the entity alignment model.

[0158] S3023: Determine a target loss value based on the multiple modal features, the first mixed features, and the second mixed features of each of the two entities in the multiple entity sample pairs, combined with corresponding alignment labels, and adjust the model parameters based on the target loss value.

[0159] In order to fully explore the features of each modality and the interactive information between modal features, the training process of the entity alignment model EAMF introduces the loss of the joint calculation model of single-view contrastive learning and multi-view contrastive learning.

[0160] For details, see the loss calculation process of the entity alignment model EAMF. Figure 5 , mainly includes the following steps:

[0161] S3023_1: Determine the intra-modal loss value of single-view contrastive learning based on the multiple modal features and alignment labels of two entities in multiple entity sample pairs.

[0162] Assume that the positive sample set in multiple entity sample pairs in the same batch is The negative sample set is Among them, the cross-domain entity sample pair The alignment label is aligned, which is used as the positive sample for model training. and The alignment labels of are unaligned and serve as positive samples for model training.

[0163] In single-view contrastive learning, the alignment probability in the entity alignment model EAMF is defined as:

[0164]

[0165] In the formula, hs represents the modal characteristics of the s mode, and τ1 represents the temperature hyperparameter.

[0166] For multiple modal features of two entities in multiple entity sample pairs B of the same batch, the inner product calculation method of formula 12 is used to perform a normalization operation. Considering the bidirectionality between entities, for entity sample pairs By performing the same operation in formula 12, we obtain

[0167] Finally, the formula for the modality intra-sub-loss value of single-view contrastive learning is expressed as:

[0168]

[0169] The single-view contrastive learning of the entity alignment model EAMF is performed on all modal features, which makes each modal feature have more discriminative boundaries and maps similar entities to closer positions in the embedding space.

[0170] S3023_2: Determine an inter-modal sub-loss value of multi-view contrastive learning based on the multiple modal features, the first mixed features and the alignment labels of the two entities in the multiple entity sample pairs.

[0171] Although the interaction between various modal features is simulated in single-view contrastive learning, the feature representations of each modality are still propagated independently during the training process. To alleviate this problem, the entity alignment model EAMF adopts multi-view contrastive learning to shorten the mapping distance between multiple modalities in the embedding space and reduce the differences in the output distribution of different modalities.

[0172] Since the first mixed feature has integrated multiple modal features, the entity alignment model EAMF performs multi-perspective comparative learning on the first mixed feature and various modal features respectively. This method interactively introduces more comprehensive knowledge into independent modal features, aggregates embedded features of different modalities to complement other modal information missing from individual modal features, and improves the semantic representation ability of entities.

[0173] Among them, multi-view contrastive learning realizes the calculation of inter-modal sub-loss value through bidirectional relative entropy (also known as KL divergence), which is expressed as follows:

[0174]

[0175] In the formula, and Represent the bidirectional correlation probability of the first mixed features of the two entities, and Represent the bidirectional correlation probabilities of the single modal features of the two entities, and The calculation method is similar to formula 12, except that the first mixed feature The corresponding temperature hyperparameter is changed to τ2.

[0176] S3023_3: Determine a multimodal joint sub-loss value of single-view contrastive learning based on the second mixed features and alignment labels of each of the two entities in the multiple entity sample pairs.

[0177] When performing single-view contrast learning on the second mixed feature, the calculation process of the multimodal joint sub-loss value is similar to formulas 12 and 13, except that the second mixed feature The corresponding temperature hyperparameter is changed to τ3, so it is not described in detail.

[0178] S3023_4: Determine the target loss value based on the intra-modal sub-loss value, the inter-modal sub-loss value and the multi-modal joint sub-loss value.

[0179] Among them, the formula of the target loss value is expressed as:

[0180]

[0181] In the formula and They represent the multimodal joint sub-loss value, the intra-modal sub-loss value, and the inter-modal sub-loss value respectively.

[0182] In the entity alignment model EAMF, by locally minimizing the loss value of each part comparison, the distance between entity nodes with high similarity in the feature space within a single modality is made closer, and the similar entity representations across modalities are mapped to close positions in the feature space. At the same time, the distance between entity nodes with less similarity is made farther, so as to achieve the distinction between aligned entity sample pairs and unaligned entity sample pairs. In addition, by jointly adjusting the loss function of each part comparison, the global optimal balance point is found to guide the iterative optimization of the algorithm model, thereby improving the training speed and performance of the model.

[0183] Among them, the loss functions of single-view contrastive learning and multi-view contrastive learning of multiple modal features can be regarded as multi-task learning tasks, using homomorphic uncertainty to dynamically weight and balance each sub-loss value during the algorithm model training process. Therefore, the intra-modal sub-loss value in Formula 15 is and the inter-modal loss value

[0184] It can also be expressed as:

[0185]

[0186] Where λ s and θ sThe dynamic weights of the loss function during single-view contrastive learning and multi-view contrastive learning are respectively used to adjust the relative weight of each modal loss in the loss function by modeling the uncertainty of the corresponding local loss.

[0187] In one example, the weighted probability representation of the classification function based on the Boltzmann distribution is:

[0188]

[0189] Furthermore, the weighted maximum likelihood estimation can be obtained, the formula is:

[0190]

[0191] Among them, f c W (x) represents the cth type of output.

[0192] It is known that the loss function of the classification task is defined as:

[0193]

[0194] assumed Represents the probability of alignment of two input entities. From this, the loss function of single-view contrastive learning and multi-view contrastive learning alignment can be derived as:

[0195]

[0196] in, Therefore, Formula 21 can be transformed into:

[0197]

[0198] Therefore, the intra-modal loss value in Equation 16 is and the inter-modal loss value in Formula 17 The expression is:

[0199]

[0200] Considering that the loss value may be negative after logarithmic transformation, the SrM module is designed in the entity alignment model EAMF, which uses the loss value itself as a penalty term to avoid the loss function being negative and affecting the convergence of the algorithm. Finally, the alignment loss function of the entity alignment model EAMF is expressed as:

[0201]

[0202] In an embodiment of the present application, in order to fully explore each modal information and the interactive information between modalities, single-view and multi-view contrastive learning are introduced in the entity alignment model EAMF. Among them, single-view contrastive learning distinguishes the uniqueness of the entity to generate a more reasonable representation by modeling the modal features of the aligned entity sample pairs (positive samples) and the modal features of the unaligned entity sample pairs (negative samples) in the same modality, while multi-view contrastive learning can simulate the interaction between positive and negative samples of the same entity under different modalities, reduce the difference caused by different modalities, and then obtain the mixed features of the entity nodes and the more fine-grained representation of each modal feature. Through this algorithm, the entity alignment model EAMF can fully capture the dynamic changes within and between modalities, while maintaining the consistency of the semantics of the entity nodes, and minimize the differences between different modalities, thereby improving the accuracy and performance of model alignment.

[0203] In some embodiments, the performance of the EAMF model is experimentally analyzed using cross-knowledge graph datasets FreeBase, DBpedia, and Yago, where the cross-knowledge graph datasets include text, number, and image information of entities, and supplement the attributes of entities, and link the aligned entities in the cross-knowledge graph through the "sameAs" relationship. The experiment selects hit rate (Hits@k) and mean reciprocal rank (MRR) to evaluate the performance of the EAMF model.

[0204] Hits@k represents the probability that the correct entity is included in the first k entities to be aligned. During the experiment, a similarity score is calculated for each entity in the knowledge graph, and then the scores are arranged in descending order to obtain a sorted list. By calculating how many entities in the first k entities in the list are correctly aligned, the evaluation result of Hits@k can be obtained. Generally, the hit rate generally uses Hits@1 and Hits@10. Hits@1 measures the proportion of correctly aligned entities ranked first in the algorithm model, which can be used to evaluate the accuracy of the highest confidence of the algorithm model; Hits@10 evaluates whether the correctly aligned entity appears in the first 10 candidate entities given by the algorithm model, which gives the algorithm model more guessing opportunities. Specifically, the expression of the hit rate is:

[0205]

[0206] Among them, H represents the set of triples, |S| represents the number of triples, is an indicator function. If the condition rank i If ≤z is true, the function value is 1, otherwise it is 0.

[0207] MRR is an indicator for evaluating the quality of the predicted ranking results, and is used to measure the ranking performance of the algorithm model for the query entity. During the experiment, for each queried entity, it is sorted from high to low according to the similarity score. For the reciprocal ranking of a single query, if the ranking of the entity is b, then its reciprocal ranking is 1 / b. By calculating the average reciprocal ranking of all queries, the average reciprocal ranking can be obtained, and the formula is expressed as:

[0208]

[0209] Among them, the value range of Hits@k and MRR is [0,1]. The larger the value, the more accurate the predicted entity to be aligned.

[0210] The experiment first analyzes the sensitivity of the temperature hyperparameters τ1 and τ2 of the EAMF model in single-multi-view contrastive learning, and finds the optimal combination by evaluating the impact of different temperature hyperparameters on entity alignment performance. Among them, the value range of τ1 is (0.001, 0.005, 0.01, 0.05, 0.1, 0.2, 0.4, 0.8, 1), and the value range of τ2 is (0.2, 0.5, 1, 2, 3, 4, 5, 6, 8). The schematic diagram of the optimization process is shown in the figure. Figure 5 shown.

[0211] according to Figure 5 It can be seen that the EAMF model is highly sensitive to the temperature hyperparameter τ1, and even a small adjustment may lead to a cliff-like decline in model performance. This is because, in the single-view contrastive learning scenario, the feature representations between different modalities are relatively independent. At this time, τ1 plays a greater role in imposing penalties on negative samples. Therefore, the appropriate τ1 value is crucial for effectively identifying potential positive samples and optimizing model performance. In contrast, the value of another temperature hyperparameter τ2 has relatively little effect on model performance. This is mainly due to the multi-view contrastive learning architecture, which integrates feature representations of different modalities to improve the adaptability of the model in complex multimodal information environments, and its introduction of KL divergence helps alleviate the overly strong penalty effect on negative samples.

[0212] In summary, based on a comprehensive consideration of various factors, the temperature hyperparameters of the single-view and multi-view contrastive learning of the EAMF model are finally set to τ1=0.1 and τ2=5, respectively.

[0213] After setting the temperature hyperparameters of the single-view and multi-view comparative learning of the EAMF model, a comparative experiment was carried out with six multimodal entity alignment models commonly used in the field of natural language processing, namely PoE, HMEA, MMEA, EVA, MSNEA and MCLEA. The comparison results are shown in Table 1.

[0214] Table 1. Comparative experimental results of EAMF model

[0215]

[0216] Among them, bold fonts indicate the best results, and underlined fonts indicate suboptimal results. It can be seen that EAMF achieved the best Hits@1, Hits@10 and MRR values ​​in all three experiments on the two knowledge graph datasets. In the comparative experiment with a pre-alignment ratio of 20%, the performance of the EAMF model was significantly improved. The Hits@1 on the FB15K-DB15K dataset was improved by 52.5% compared with the best algorithm MCLEA model, the best Hits@10 improvement rate was 16.5%, and the relative improvement of MRR reached 36.5%. At the same time, Hits@1 on the FB15K-YAGO15K dataset increased by 73.9%, Hits@10 increased by 32.3%, and the MRR improvement rate reached 55.2%. When the pre-alignment ratio was set to 50% and 80% respectively, EAMF also performed better than other methods, with the highest comprehensive improvement rates of 34.4% and 19.6% respectively.

[0217] Although mainstream multimodal entity alignment methods have made improvements in integrating multiple modal features, as a heterogeneous network with complex relationships, the knowledge graph has an irreplaceable role in guiding the algorithm to mine potential aligned entities. This is because the graph structure information plays a key role in identifying and associating different entities. In the process of modeling the graph structure, the EAMF model not only fully considers the complex relationships between different entities, but also effectively captures implicit structural information by introducing a graph attention network enhanced by edge features. At the same time, the EAMF model also introduces a richness encoding mechanism, which is similar to the bag-of-words method. By counting the modal information richness of the entity and using it as a supplementary feature, the algorithm is further prompted to focus on entities with higher information richness when constructing joint representations. In addition, the EAMF model also adopts a contrastive learning strategy to overcome the data scarcity problem in the entity alignment task, compensate for the shortcomings of the entity in the overall structural context representation, and significantly improve the algorithm's ability to distinguish by optimizing the feature representation in the latent space. Finally, the self-constraint mechanism adopted by the EAMF model effectively coordinates multiple loss functions, solves the problem that the optimal point may be unreachable due to local loss optimization alone, and enhances the robustness of the algorithm.

[0218] In order to further explore how the EAMF model effectively applies various modal features, the embodiment of the present application visualizes the attention coefficient adjusted by the EAMF model when fusing each modal information during the reasoning process. The processing results are as follows: Figure 6As shown in the figure. Since the EAMF model gives nearly half of the weight to the graph structure information when modeling, this is not only consistent with the basic intuition of graph representation learning, but also emphasizes the significant effect of the edge information enhancement strategy in modeling the multivariate graph structure in the knowledge graph. At the same time, the attention coefficients of the association relationship and image modality information are similar, both of which provide key supplementary information for the model to learn the features of entity nodes. In addition, the attention coefficient of the richness information introduced by the EAMF model exceeds the attention coefficient of the attribute, indicating that the attributes of the entity may be missing, while the richness effectively assists the model to balance the attention to the entity nodes with complete information and missing information, ensuring the robustness and accuracy of the model. The MCLEA model has significantly different distribution of attention coefficients on the two datasets. Since the number of attributes in the DB15K knowledge graph dataset is significantly higher than that in the YAGO15K knowledge graph dataset, it is nearly twice as high. Therefore, when processing the FB15K-DB15K knowledge graph dataset, the MCLEA model tends to focus on the information of attribute triples and relatively ignores the content of graph structure and visual modality. In the FB15K-YAGO15K knowledge graph dataset, the MCLEA model pays more attention to the graph structure and significantly reduces the attention to attribute modal information. This adjustment method of modal attention is greatly affected by the dataset, which may lead to unstable model performance and fail to achieve the optimal multimodal fusion effect.

[0219] To further analyze the effectiveness of the EAMF model compared to the MCLEA model, a comparative experiment was conducted on the performance of the two models under the condition that the pre-alignment ratio was set to 0.2. The results are shown in the figure. Figure 7 As shown in the figure. Among them, the performance of the EAMF model is significantly improved after 500 rounds of training. This is because the iterative training strategy is adopted after 500 rounds of training, that is, the model can autonomously explore potential entity alignment seeds and incorporate these seeds into the alignment benchmark, so that the model can identify more possible similar entity pairs. Although the MCLEA model also uses iterative learning, its performance growth is relatively limited. This is because when the MCLEA model parses multimodal information, the visual modality will take away the weight of part of the graph structure information, which in turn limits the overall perception. The model's all-round information perception of entities mainly depends on the graph structure. In contrast, the EAMF model effectively compensates for this deficiency by introducing an edge feature enhancement mechanism and multi-view cross-attention fusion. At the same time, the self-constraint mechanism also avoids the instability of performance fluctuations during the training of the EAMF model, making the algorithm training more stable and performing better.

[0220] To ensure the stability of the EAMF model with a small number of entity alignment seeds and to evaluate the performance of the model in low-resource scenarios, a comparative experiment was conducted on two knowledge graph datasets by setting different seed pre-alignment ratios. The alignment ratio was set from 0.01 to 0.3. The experimental results are shown in Figure 2. Figure 8 As shown. For the case of fewer entity alignment seeds, the EAMF model shows significant advantages over the MCLEA model, especially when the entity alignment seed ratio is only 1%, the EAMF model outperforms the MCLEA model in the performance indicators of Hits@1, Hits@10 and MRR, which is particularly prominent on the FB15K-YAGO15K dataset. As the entity alignment seed ratio increases to 5%, the performance gap between the EAMF model and the MCLEA model further widens, and this advantage increases with the increase in the proportion of entity alignment seeds, indicating the stability of the EAMF model in a low alignment resource environment, and fully demonstrating its applicability in processing entity alignment tasks on cross-knowledge graph datasets with scarce alignment links. In the case of less manual annotation, it can still effectively learn the ability to identify potential semantically similar entities.

[0221] Aiming at the influence of edge information enhancement, multi-view cross attention fusion and self-constraint mechanism in the EAMF model on improving entity alignment performance, an ablation experiment was designed, and the experimental settings are shown in Table 2. Among them, EAMF-1 means removing the image modality information based on the EAMF model, and only using the graph structure, relationship, attribute and richness information; EAMF-2 means removing the edge information enhancement module based on the EAMF model, and only using the graph attention network to model the graph structure representation; EAMF-3 means removing the multi-view cross attention fusion module based on the EAMF model, and only using the joint features as multi-modal guidance information; EAMF-4 removes the self-constraint mechanism based on the EAMF model, and only uses a simple summation to calculate the three-part loss function value.

[0222] Table 2. EAMF model ablation experiment settings

[0223]

[0224] The ablation experiment results are shown in Table 3. Under the EAMF framework, even without integrating image modal information, the entity alignment performance is still better than the MCLEA model, especially in the environment with less alignment resources, the advantage is more significant, indicating the effectiveness of the EAMF model under the condition of limited alignment resources. If the edge information is not introduced to enhance the graph structure modeling, the algorithm performance will decline significantly, indicating that the lack of multi-relation information makes it difficult for the algorithm to effectively capture the information that guides the learning of entity representation similarity, thereby affecting the learning ability of potential knowledge elements. At the same time, when multi-view mixed features are not used, the overall performance of the algorithm will decline, indicating that in the process of multi-view contrastive learning, if multi-modal information cannot be effectively integrated, the ability of the algorithm to accurately distinguish entities will be weakened. In addition, if the self-constraint mechanism is not adopted, the EAMF model cannot reach its optimal performance level. It fails to show suboptimal performance in the two experimental settings of 20% and 80% of the FB15K-DB15K dataset, indicating the important role of the self-constraint mechanism in helping the algorithm balance the single-view and multi-view contrastive learning losses, and effectively promotes the algorithm to extract better feature information from both aspects. Combining the side information enhancement and self-constraint mechanism, and with the assistance of image information, the best alignment performance can be achieved, indicating that the combination of the two can not only promote each other, but also jointly improve the recognition ability of similarity between entities.

[0225] Table 3. EAMF model ablation experimental results

[0226]

[0227]

[0228] Based on the same technical concept, an embodiment of the present application also provides a training device for an entity alignment model, which can implement the steps of the above-mentioned training method for the entity alignment model and achieve the same technical effect.

[0229] See also Fig. 9 , the training device comprises a sample construction unit 901 and an iterative training unit 902, wherein:

[0230] A sample construction unit 901 is used to construct a training sample set, wherein the alignment label of each entity sample pair in the training sample set is used to indicate whether two entities in the corresponding entity sample pair originating from different knowledge graphs are aligned;

[0231] The iterative training unit 902 is used to perform multiple rounds of iterative training on the entity alignment model according to the training sample set until the model converges, wherein each round of iterative training includes:

[0232] Extract multiple modal features of two entities in multiple entity sample pairs respectively;

[0233] Performing cross-attention calculations on the multiple modal features of two entities in the multiple entity sample pairs to obtain a first mixed feature, and merging the multiple modal features of two entities in the multiple entity sample pairs to obtain a second mixed feature;

[0234] According to the multiple modal features, the first mixed features and the second mixed features of each of the two entities in the multiple entity sample pairs, combined with the corresponding alignment labels, a target loss value is determined, and the model parameters are adjusted according to the target loss value.

[0235] Optionally, the multiple modal features include at least a graph structure embedding feature, and the iterative training unit 902 is specifically used for:

[0236] For the current entity in any entity sample pair, perform the following operations:

[0237] A multi-head attention mechanism is used to extract node features of the current entity and its adjacent entities, and a multi-head attention mechanism is used to extract edge features between the current entity and its adjacent entities, and an attention coefficient is determined based on the node features and edge features; the edge features represent the association relationship between the current entity and its adjacent entities;

[0238] The node features of the adjacent entities of the current entity are aggregated according to the attention coefficient to obtain the graph structure embedding features of the current entity in the knowledge graph.

[0239] Optionally, the multiple modal features include at least a richness embedding feature, and the iterative training unit 902 is specifically used for:

[0240] For the current entity in any entity sample pair, perform the following operations:

[0241] Obtain the association relationship between the current entity and the adjacent entity and the attributes of the current entity, and count the association relationship and additional information. The attributes represent the additional information of the current entity.

[0242] Perform one-hot encoding on the counting results to obtain the richness embedding features of the current entity.

[0243] Optionally, the multiple modal features include at least a relation embedding feature, an attribute embedding feature, and a visual embedding feature, and the iterative training unit 902 is specifically used for:

[0244] For the current entity in any entity sample pair, perform the following operations:

[0245] The bag-of-words model is used to generate the relational embedding features of the current entity based on the association between the current entity and the adjacent entities in the knowledge graph. The bag-of-words model is also used to generate the attribute embedding features of the current entity based on the attributes of the current entity. The attributes represent additional information of the current entity.

[0246] A pre-trained visual model is used to generate the visual embedding features of the current entity based on the pixel information of the current entity in the image.

[0247] Optionally, the iterative training unit 902 is specifically used for:

[0248] For the current entity in any entity sample pair, perform the following operations:

[0249] Combine multiple modal features of the current entity to obtain intermediate features;

[0250] According to the three weight matrices of multiple attention heads, the intermediate features are transformed to obtain the query matrix, key matrix and value matrix across perspectives;

[0251] Determine the multi-view cross-attention coefficient according to the query matrix and the key matrix;

[0252] Determine the multi-view modal features according to the multi-view cross-attention coefficients and value matrices;

[0253] The intermediate features and multi-view modal features are residually connected and standardized to obtain the first mixed features of the current entity.

[0254] Optionally, the iterative training unit 902 is specifically used for:

[0255] For the current entity in any entity sample pair, perform the following operations:

[0256] Determine each weighted weight according to the information density corresponding to each of the multiple modal features of the current entity;

[0257] According to each weighted weight, multiple modal features of the current entity are merged to obtain a second mixed feature.

[0258] Optionally, the iterative training unit 902 is specifically used for:

[0259] Determine the intra-modal sub-loss value of single-view contrastive learning based on the multiple modal features and alignment labels of the two entities in the multiple entity sample pairs;

[0260] Determine an inter-modal sub-loss value of multi-view contrastive learning according to the respective multiple modal features, the first mixed features and the alignment labels of the two entities in the multiple entity sample pairs;

[0261] Determine a multimodal joint sub-loss value of single-view contrastive learning according to respective second mixed features and alignment labels of two entities in the plurality of entity sample pairs;

[0262] The target loss value is determined based on the intra-modal sub-loss value, inter-modal sub-loss value and multi-modal joint sub-loss value.

[0263] For the convenience of description, the above parts are divided into modules (or units) according to their functions and described separately. Of course, when implementing this application, the functions of each module (or unit) can be implemented in the same or multiple software or hardware.

[0264] After introducing the training method and device of the entity alignment model according to the exemplary embodiment of the present application, next, an electronic device according to another exemplary embodiment of the present application is introduced.

[0265] Those skilled in the art will appreciate that various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to as "circuit", "module" or "system" herein.

[0266] Based on the same inventive concept as the above method embodiment, the electronic device provided in the embodiment of the present application can be a server or a terminal device. In this embodiment, the structure of the electronic device can be as follows: Fig.10 As shown, it includes a memory 1001 , a communication interface 1003 and one or more processors 1002 .

[0267] The memory 1001 is used to store computer programs executed by the processor 1002. The memory 1001 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and programs required for running the instant messaging function, etc.; the data storage area may store various instant messaging information and operation instruction sets, etc.

[0268] The memory 1001 may be a volatile memory, such as a random-access memory (RAM); the memory 1001 may also be a non-volatile memory, such as a read-only memory, a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); or the memory 1001 may be any other medium that can be used to carry or store a desired computer program in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 1001 may be a combination of the above memories.

[0269] The processor 1002 may include one or more central processing units (CPU) or a digital processing unit, etc. The processor 1002 is used to implement the above-mentioned entity alignment model training method when calling the computer program stored in the memory 1001 .

[0270] The communication interface 1003 is used to communicate with other network devices.

[0271] The specific connection medium between the memory 1001, the communication interface 1003 and the processor 1002 is not limited in the embodiment of the present application. Fig.10 In the embodiment, the memory 1001 and the processor 1002 are connected via a bus 1004. The bus 1004 is connected to the processor 1002 via a bus 1004. Fig.10 The connections between the other components are only for illustration and are not intended to be limiting. The bus 1004 can be divided into an address bus, a data bus, a control bus, etc. For ease of description, Fig.10 The diagram shows that only one thick line is used, but this does not mean that there is only one bus or only one type of bus.

[0272] The memory 1001 stores a computer storage medium, and the computer storage medium stores computer executable instructions, and the computer executable instructions are used to implement the training method of the entity alignment model of the embodiment of the present application. The processor 1002 is used to execute the training method of the entity alignment model, such as Figure 3 shown.

[0273] In some possible implementations, various aspects of the entity alignment model training method provided in the present application may also be implemented in the form of a program product, which includes a computer program. When the program product is run on an electronic device, the computer program is used to enable the electronic device to execute the steps in the entity alignment model training method according to various exemplary embodiments of the present application described above in this specification. For example, the electronic device may execute the following steps: Figure 3 Follow the steps shown in .

[0274] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0275] The program product of the embodiment of the present application may adopt a portable compact disk read-only memory (CD-ROM) and include a computer program, and can be run on an electronic device. However, the program product of the present application is not limited thereto, and in this document, a readable storage medium may be any tangible medium containing or storing a program, which can be used by or in combination with a command execution system, apparatus, or device.

[0276] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, wherein a readable computer program is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with a command execution system, apparatus, or device.

[0277] The computer program embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0278] The computer program for performing the operation of the present application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and also conventional procedural programming languages ​​such as "C" language or similar programming languages. The computer program can be executed entirely on the user electronic device, partially on the user electronic device, as a separate software package, partially on the user electronic device and partially on a remote electronic device, or entirely on a remote electronic device or server. In the case of a remote electronic device, the remote electronic device can be connected to the user electronic device through any type of network including a local area network (LAN) or a wide area network (WAN), or can be connected to an external electronic device (for example, using an Internet service provider to connect through the Internet).

[0279] It should be noted that, although several units or subunits of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into multiple units to be embodied.

[0280] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0281] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0282] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0283] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0284] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0285] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A training method for an entity alignment model, characterized in that: The method comprises: Constructing a training sample set, wherein the alignment label of each entity sample pair in the training sample set is used to indicate whether two entities in the corresponding entity sample pair originating from different knowledge graphs are aligned; According to the training sample set, the entity alignment model is trained for multiple rounds of iterations until the model converges, wherein each round of iteration training includes: Extract multiple modal features of two entities in multiple entity sample pairs respectively; Performing multi-view cross-attention calculations on the multiple modal features of each of the two entities in the multiple entity sample pairs to obtain a first mixed feature, and performing single-view merging on the multiple modal features of each of the two entities in the multiple entity sample pairs to obtain a second mixed feature; According to the multiple modal features, the first mixed features and the second mixed features of each of the two entities in the multiple entity sample pairs, combined with the corresponding alignment labels, a target loss value is determined, and the model parameters are adjusted according to the target loss value.

2. The method according to claim 1, characterized in that The multiple modal features include at least a graph structure embedding feature, and the extracting of the multiple modal features of two entities in the multiple entity sample pairs includes: For the current entity in any entity sample pair, perform the following operations: A multi-head attention mechanism is used to extract node features of the current entity and adjacent entities of the current entity, and a multi-head attention mechanism is used to extract edge features between the current entity and the adjacent entities, and an attention coefficient is determined according to the node features and the edge features; the edge features represent the association relationship between the current entity and the adjacent entities; The node features of the adjacent entities of the current entity are aggregated according to the attention coefficient to obtain the graph structure embedding features of the current entity in the knowledge graph.

3. The method according to claim 1, characterized in that The multiple modal features include at least a richness embedding feature, and the extracting of the multiple modal features of two entities in the multiple entity sample pairs includes: For the current entity in any entity sample pair, perform the following operations: Acquire the association relationship between the current entity and the adjacent entity and the attribute of the current entity, and count the association relationship and the additional information, wherein the attribute represents the additional information of the current entity; The counting result is one-hot encoded to obtain the richness embedding feature of the current entity.

4. The method according to claim 1, characterized in that The multiple modal features include at least a relation embedding feature, an attribute embedding feature, and a visual embedding feature. The extracting multiple modal features of two entities in the multiple entity sample pairs includes: For the current entity in any entity sample pair, perform the following operations: Using a bag-of-words model, according to the association between the current entity and adjacent entities in the knowledge graph, a relational embedding feature of the current entity is generated; and, using a bag-of-words model, according to the attributes of the current entity, an attribute embedding feature of the current entity is generated, wherein the attribute represents additional information of the current entity; A pre-trained visual model is used to generate a visual embedding feature of the current entity according to pixel information of the current entity in the image.

5. The method according to claim 1, characterized in that The cross-attention calculation is performed on the multiple modal features of the two entities in the multiple entity sample pairs to obtain multi-view mixed features, including: For the current entity in any entity sample pair, perform the following operations: splicing multiple modal features of the current entity to obtain intermediate features; According to the three weight matrices of the multiple attention heads, the intermediate features are transformed respectively to obtain a query matrix, a key matrix and a value matrix across perspectives; Determining a multi-view cross-attention coefficient according to the query matrix and the key matrix; Determining a multi-view modal feature according to the multi-view cross-attention coefficient and the value matrix; The intermediate features and the multi-view modal features are subjected to residual connection and standardization processing to obtain a first mixed feature of the current entity.

6. The method according to claim 1, characterized in that The step of merging the multiple modal features of the two entities in the multiple entity sample pairs to obtain a second mixed feature includes: For the current entity in any entity sample pair, perform the following operations: Determining respective weighting weights according to the information density corresponding to each of the multiple modal features of the current entity; According to the respective weighted weights, the multiple modal features of the current entity are combined to obtain a second mixed feature.

7. The method according to any one of claims 1 to 6, characterized in that The determining of the target loss value according to the respective multiple modal features, the first mixed feature and the second mixed feature of the two entities in the multiple entity samples in combination with the corresponding alignment labels includes: Determining a modality intra-sub-loss value for single-view contrastive learning according to the respective multiple modal features and alignment labels of two entities in the multiple entity sample pairs; Determining an inter-modal sub-loss value of multi-view contrastive learning according to the respective multiple modal features, the first mixed feature, and the alignment label of the two entities in the multiple entity sample pairs; Determining a multimodal joint sub-loss value for single-view contrastive learning according to respective second mixed features and alignment labels of two entities in the plurality of entity sample pairs; The target loss value is determined according to the intra-modal sub-loss value, the inter-modal sub-loss value and the multi-modal joint sub-loss value.

8. A training device for an entity alignment model, characterized in that: include: A sample construction unit, used to construct a training sample set, wherein the alignment label of each entity sample pair in the training sample set is used to indicate whether two entities in the corresponding entity sample pair originating from different knowledge graphs are aligned; The iterative training unit is used to perform multiple rounds of iterative training on the entity alignment model according to the training sample set until the model converges, wherein each round of iterative training includes: Extract multiple modal features of two entities in multiple entity sample pairs respectively; Performing cross-attention calculations on the multiple modal features of each of the two entities in the multiple entity sample pairs to obtain a first mixed feature, and merging the multiple modal features of each of the two entities in the multiple entity sample pairs to obtain a second mixed feature; According to the multiple modal features, the first mixed features and the second mixed features of each of the two entities in the multiple entity sample pairs, combined with the corresponding alignment labels, a target loss value is determined, and the model parameters are adjusted according to the target loss value.

9. An electronic device, characterized in that: It includes a processor, a memory and a communication interface, wherein the communication interface, the memory and the processor are connected via a bus; The communication interface is used to send and receive data; The memory is used to store computer instructions; The processor is used to read the computer instructions in the memory and execute the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: When the computer program is executed on an electronic device, the computer program is used to enable the electronic device to execute the steps of any one of the methods of claims 1 to 7.

Images