Multimodal entity alignment method, device and medium based on structural prefix injection
Multimodal information is processed through structural prefix injection and multi-head self-attention mechanism, combined with Gaussian modal noise masking mechanism, the problem of graph structural heterogeneity in multimodal entity alignment is solved, and more accurate entity alignment is achieved.
Patent Information
- Application Number
- CN202411204157.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-30
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-08-30
AI Technical Summary
The existing multimodal entity alignment method fails to effectively solve the problem of graph structure heterogeneity, resulting in degradation of entity alignment performance and incorrect alignment.
Using multimodal entity alignment method based on structural prefix injection, the graph structural features and multimodal information are learned through graph attention network, combined with Gaussian modal noise masking mechanism, and feature embedding using multi-head self-attention mechanism and Vanilla Transformer model. and interactive processing, calculate in-modal comparison loss and inter-modal alignment loss.
More accurate multimodal entity alignment is achieved, the processing ability of graph structure heterogeneity is enhanced, and the alignment performance is improved.
Smart Images

Figure CN118734252B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of knowledge graph technology, and in particular to a multimodal entity alignment method, device and medium based on structure prefix injection. Background Art
[0002] Knowledge graph is a structured knowledge representation method that helps to gain a deeper understanding of the associations and intrinsic meanings between information by describing entities, concepts and their relationships in the form of a graph. Knowledge graphs cover a wide range of areas, from general common-sense knowledge to complex knowledge in professional fields. Nevertheless, different knowledge graphs may prioritize different aspects of knowledge description. For entities representing the same real-world objects, some knowledge graphs may highlight specific attributes of the entities themselves, while other knowledge graphs focus on describing the relationships between entities. Knowledge fusion integrates different knowledge graphs to produce a unified, precise and comprehensive entity representation. Entity alignment is a key technology in knowledge fusion, which is used to identify corresponding relationships between entities across different knowledge graphs. For multiple entities representing the same object, alignment relationships must be established between these entities to consolidate and merge the information they contain.
[0003] Multimodal knowledge graphs integrate various data modalities, each of which provides a unique perspective and representation of entities, thus enriching their representation. Compared with traditional knowledge graphs that mainly focus on the relationships between entities, multimodal knowledge graphs leverage multimodal information such as text and images to capture more comprehensive associations and semantic features between entities. This approach has been widely used in different fields, including video question answering, recommendation systems, and other scenarios. Multimodal entity alignment takes into account multimodal data such as images and text, enhancing the semantic depth of entities. It adopts multimodal knowledge graphs to align equivalent entities between different graphs.
[0004] However, in most multimodal entity alignment methods, such as Chinese patent CN117725230A, the impact of modal-specific noise and graph structure heterogeneity in different knowledge graphs on entity alignment performance is not taken into account. Instead, multimodal information is simply fused and aligned, resulting in a decrease in multimodal entity alignment performance and possible misalignment. Summary of the invention
[0005] The purpose of the present invention is to provide a multimodal entity alignment method, device and medium based on structural prefix injection in order to solve the problem that the existing multimodal entity alignment method cannot solve the graph structure heterogeneity.
[0006] The above object of the present invention is achieved through the following technical solutions:
[0007] S1: Obtain two multimodal knowledge graphs and perform preprocessing;
[0008] S2: Construct a graph attention network with a multi-head self-attention mechanism; use the graph attention network to learn the graph structure feature vectors of the two multimodal knowledge graphs respectively to generate graph structure features of the entity;
[0009] S3: Using the graph attention network to learn the multimodal information of the two multimodal knowledge graphs respectively, and generate multimodal features of the entity;
[0010] S4: Through the structural prefix injection interaction module of the graph attention network, the graph structure features and multimodal features are combined to obtain a joint embedding;
[0011] S5: Entity alignment via joint embedding, combining intra-modality contrastive loss and inter-modality contrastive loss of entities.
[0012] Optionally, the two multimodal knowledge graphs include: as well as ,in Represents a collection of entities; Represents a set of relations; Represents a collection of attributes; An associated image representing the entity; A set of triples representing entity relationships.
[0013] Optionally, step S2 includes:
[0014] S21: Obtaining an entity index and an adjacency matrix according to the entity set and the triple set; the adjacency matrix represents the connection relationship between the entity and its neighbor entities;
[0015] S22: Pre-embed the entity through the entity index to obtain the hidden state of each entity;
[0016] S23: Use the graph attention network to embed the graph structure features of the entity. For each entity pair , calculate the attention coefficient of the entity pair ,as follows
[0017]
[0018] in is the attention coefficient of the entity pair, Represents a splicing operation, is the weight matrix of the graph structure mode; Activation function; Is Entity The hidden state of Is Entity The hidden state of represents the learnable weights;
[0019] S24: By calculating all entities The attention coefficient of the connected entity pair, obtaining the attention score , the expression is as follows:
[0020]
[0021] in, is the attention score of the entity pair, Representing Entity Neighbors; Represents a neighbor entity; represents the natural exponential function; represents the normalized exponential function;
[0022] S25: Calculate entity based on graph attention network and attention score Neighbor aggregation , the expression is:
[0023]
[0024] in, is the activation function;
[0025] Aggregate the neighbors computed by the graph attention network , as an entity The graph structure characteristics of .
[0026] Optionally, step S3 includes:
[0027] The multimodal information includes: associated pictures, relationship modal information and attribute modal information;
[0028] S31: For entities The relationship modal information and attribute modal information of the entity are preprocessed using bag-of-words features and one-hot encoding to obtain the entity relational modal input and attribute modal input;
[0029] S32: Use pre-trained image encoder to Related pictures Preprocess and get the entity Corresponding related pictures Visual modal input ,as follows:
[0030]
[0031] in, Indicates that VGG-16 is selected as the pre-trained image encoder;
[0032] S33: For entities relational modal input, attribute modal input, and visual modal input , use the fully connected network to process and obtain the multimodal features of the entity, as follows:
[0033]
[0034] in, and Represent relational modality, attribute modality and visual modality respectively, The weight matrix representing the linear transformation; Representing modality Fully connected network layer; Indicates Multimodal features of entities; multimodal features include: relation embedding, attribute embedding and visual embedding;
[0035] S34: Use the Gaussian modal noise masking mechanism to process the multimodal features and generate the final multimodal features, as follows:
[0036] The probability transformation of each modal feature of the multimodal feature is as follows:
[0037]
[0038] in, represents a uniformly distributed random variable that determines whether to apply noise, is the threshold probability of adding noise to each multimodal feature, is the masking rate;
[0039] Multimodal features The noise vector The definition is as follows:
[0040]
[0041] in, and denote the standard deviation and mean of the modal noise data, respectively, and z denotes the distribution from Gaussian The sample drawn from represents the unit covariance matrix; Represents a dot product operation.
[0042] Optionally, step S4 includes:
[0043] S41: Use the Vanilla Transformer model to process the graph structure features of entities and generate queries for graph structure features ,key Sum :
[0044]
[0045] in, It is the graph structure feature; Represent the weight matrices for query, key, and value respectively;
[0046] S42: Calculate the score of each attention head of the graph structure feature through the multi-head self-attention mechanism , determine the output of multi-head self-attention ;
[0047] Using layer normalization and residual connections to process the output of multi-head attention ,as follows:
[0048]
[0049]
[0050]
[0051] in, Represents the graph structure features after layer normalization and residual connection processing; Representation layer normalization operation; and represents the weight matrix used for linear transformation; and represents the bias used for linear transformation; is the activation function; represents a fully connected feed-forward network layer; It is the structural feature of the processed graph;
[0052] S43: Keys based on graph structural features Sum , combined with the structural prefix injection interaction module, redefine the key of multimodal features Sum The calculation is as follows:
[0053]
[0054] in, and is the redefined key value; is the key-value concatenation weight; Represents a splicing operation; Multimodal features representing entities; Indicates that it is used for modal The weight matrix of the key; Indicates that it is used for modal The weight matrix of the values;
[0055] Use keys that redefine multimodal features Sum , process the multimodal features and obtain the processed multimodal features ;
[0056] S44: Keys based on graph structural features Sum , calculate the cross weight of graph structure features ; Based on the key of the redefined multimodal feature Sum , calculate the cross weight of multimodal features ;
[0057] S45: According to the cross weight and , compute the joint embedding for modality fusion;
[0058] Joint embedding includes: Early embedding and late embedding of hidden states ;
[0059]
[0060] in, Indicates The cross-weight of multimodal features of an entity; Indicates The cross weights of the graph structure features of each entity.
[0061] Optionally, step S5 includes:
[0062] S51: For each pre-aligned seed pair, define a negative set by destroying the seed pair ; A seed pair is an entity pair;
[0063] S52: Adopt the intra-batch negative sampling strategy to limit the sampling of negative sets to small batches. The expression for calculating the alignment distribution probability is:
[0064]
[0065] in, , represents the temperature parameter; represents the pre-aligned seed pair; Representing Entities ; Representing Entities Transpose of the multimodal features; Representing Entities Multimodal features;
[0066] S53: For each mode of the entity , define the bidirectional alignment objective, and use the intra-modality alignment loss to constrain the input embedding; for the intra-modality alignment loss, two methods are used to calculate it: early joint embedding and late joint embedding, as follows:
[0067]
[0068]
[0069] in, represents the intra-modality contrastive loss, computed using early joint embedding, represents the late intra-modality contrastive loss, which is calculated using the late joint embedding; Indicates the use of late joint embedding Calculated alignment distribution probability;
[0070] S54: Design inter-modal alignment loss for unimodal features of entities ; then the total loss function .
[0071] An electronic device comprises a processor, a memory, a user interface and a network interface, wherein the memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory so that the electronic device performs a multimodal entity alignment method based on structural prefix injection.
[0072] A computer-readable storage medium stores instructions. When the instructions are executed, a multimodal entity alignment method based on structural prefix injection is performed.
[0073] The beneficial effects brought by the technical solution provided by the present invention are:
[0074] The multimodal feature module is used to embed the modal features of entities in the multimodal knowledge graph, and the interactive module of graph structure prefix injection and cross-modal weights are used to perform multimodal fusion operations. The multimodal contrastive learning mechanism is used to calculate the intra-modal contrast loss and inter-modal alignment loss, ultimately completing a more accurate multimodal entity alignment task. BRIEF DESCRIPTION OF THE DRAWINGS
[0075] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:
[0076] Figure 1is a step diagram of a multimodal entity alignment method based on structural prefix injection in an embodiment of the present invention;
[0077] Figure 2 It is a schematic diagram of the structure of an electronic device according to a multimodal entity alignment method based on structural prefix injection in an embodiment of the present invention. DETAILED DESCRIPTION
[0078] In order to have a clearer understanding of the technical features, purposes and effects of the present invention, specific embodiments of the present invention are now described in detail with reference to the accompanying drawings.
[0079] An embodiment of the present invention provides a multimodal entity alignment method based on structure prefix injection.
[0080] Please refer to Figure 1 , Figure 1 : is a step diagram of a multimodal entity alignment method based on structural prefix injection in an embodiment of the present invention, comprising:
[0081] S1: Obtain two multimodal knowledge graphs and perform preprocessing;
[0082] The two multimodal knowledge graphs include: as well as ,in Represents a collection of entities; Represents a set of relations; Represents a collection of attributes; An associated image representing the entity; A set of triples representing entity relationships.
[0083] S2: Construct a graph attention network with a multi-head self-attention mechanism; use the graph attention network to learn the graph structure feature vectors of the two multimodal knowledge graphs respectively to generate graph structure features of the entity;
[0084] Step S2 includes:
[0085] S21: Obtaining an entity index and an adjacency matrix according to the entity set and the triple set; the adjacency matrix represents the connection relationship between the entity and its neighbor entities;
[0086] S22: Pre-embed the entity through the entity index to obtain the hidden state of each entity;
[0087] S23: Use the graph attention network to embed the graph structure features of the entity. For each entity pair , calculate the attention coefficient of the entity pair ,as follows
[0088]
[0089] in is the attention coefficient of the entity pair, Represents a splicing operation, is the weight matrix of the graph structure mode; Activation function; Is Entity The hidden state of Is Entity The hidden state of represents the learnable weights;
[0090] S24: By calculating all entities The attention coefficient of the connected entity pair, obtaining the attention score , the expression is as follows:
[0091]
[0092] in, is the attention score of the entity pair, Representing Entity Neighbors; Represents a neighbor entity; represents the natural exponential function; represents the normalized exponential function;
[0093] S25: Calculate entity based on graph attention network and attention score Neighbor aggregation , the expression is:
[0094]
[0095] in, is the activation function;
[0096] Aggregate the neighbors computed by the graph attention network , as an entity The graph structure characteristics of .
[0097] Specifically, neighbor aggregation means that the entity features in the graph structure depend not only on its own features, but also on the features of its neighbor entities. By aggregating neighbor entity features, more representative entity features are generated to represent the hidden state of the entity. Through multi-layer neighbor aggregation, the hidden state of the entity will contain more contextual information.
[0098] S3: Using the graph attention network to learn the multimodal information of the two multimodal knowledge graphs respectively, and generate multimodal features of the entity;
[0099] Step S3 includes:
[0100] The multimodal information includes: associated pictures, relationship modal information and attribute modal information;
[0101] S31: For entities The relationship modal information and attribute modal information of the entity are preprocessed using bag-of-words features and one-hot encoding to obtain the entity relational modal input and attribute modal input;
[0102] Specifically, the features of the relational modality and the features of the attribute modality are represented by vectors of preset length; bag-of-words feature processing is performed on the relational modality and the attribute modality, and an alignment editing method is used to reduce error accumulation.
[0103] S32: Use pre-trained image encoder to identify entities Related pictures Preprocess and get the entity Corresponding related pictures Visual modal input ,as follows:
[0104]
[0105] in, Indicates that VGG-16 is selected as the pre-trained image encoder;
[0106] S33: For entities relational modal input, attribute modal input, and visual modal input , use the fully connected network to process and obtain the multimodal features of the entity, as follows:
[0107]
[0108] in, and Represent relational modality, attribute modality and visual modality respectively, The weight matrix representing the linear transformation; Representing modality Fully connected network layer; Indicates Multimodal features of entities; multimodal features include: relation embedding, attribute embedding and visual embedding;
[0109] S34: Use the Gaussian modal noise masking mechanism to process the multimodal features and generate the final multimodal features, as follows:
[0110] The probability transformation of each modal feature of the multimodal feature is as follows:
[0111]
[0112] in, represents a uniformly distributed random variable that determines whether to apply noise, is the threshold probability of adding noise to each multimodal feature, is the masking rate;
[0113] Multimodal features The noise vector The definition is as follows:
[0114]
[0115] in, and denote the standard deviation and mean of the modal noise data, respectively, and z denotes the distribution from Gaussian The sample drawn from represents the unit covariance matrix; Represents a dot product operation.
[0116] Specifically, multimodal feature representation is enhanced by controlling noise injection. The noise of multimodal information refers to the errors or interferences generated in the process of processing multimodal data, which usually affect the quality of the data and the processing effect. In the entity alignment task, modality-specific noise often exists. For example, different entity images are stored in different knowledge graphs, and the misleading differences in the images are image noise. Similar noise also exists in attributes, relationships and other modalities. Therefore, in this method, modal noise is intentionally incorporated to adapt to noisy real-world scenarios and improve the performance of entity alignment. In addition, different knowledge graphs are constructed by different methods, and their graph structures and entity connectivity are different. For example, in the alignment task, the graph structures of the two entities to be aligned, including the neighbor nodes of the entities and the relationships between the entities, are heterogeneous, which can cause misleading during alignment and cause entity alignment errors.
[0117] S4: Through the structural prefix injection interaction module of the graph attention network, the graph structure features and multimodal features are combined to obtain a joint embedding;
[0118] Step S4 includes:
[0119] S41: Use the Vanilla Transformer model to process the graph structure features of entities and generate queries for graph structure features ,key Sum :
[0120]
[0121] in, It is the graph structure feature; Represent the weight matrices for query, key, and value respectively;
[0122] Specifically, the Vanilla Transformer model is a neural network model based on the self-attention mechanism. It consists of multiple encoders and decoders, each of which contains multiple self-attention layers and feedforward neural network layers.
[0123] S42: Calculate the score of each attention head of the graph structure feature through the multi-head self-attention mechanism , determine the output of multi-head self-attention ;
[0124] Using layer normalization and residual connections to process the output of multi-head attention ,as follows:
[0125]
[0126]
[0127]
[0128] in, Represents the graph structure features after layer normalization and residual connection processing; Representation layer normalization operation; and represents the weight matrix used for linear transformation; and represents the bias used for linear transformation; is the activation function; represents a fully connected feed-forward network layer; It is the structural feature of the processed graph;
[0129] S43: Keys based on graph structural features Sum , combined with the structural prefix injection interaction module, redefine the key of multimodal features Sum The calculation is as follows:
[0130]
[0131] in, and is the redefined key value; is the key-value concatenation weight; Represents a splicing operation; Representing multimodal features of entities; Indicates that it is used for modal The weight matrix of the key; Indicates that it is used for modal The weight matrix of the values;
[0132] Use keys that redefine multimodal features Sum , process the multimodal features and obtain the processed multimodal features ;
[0133] Specifically, the attention distribution of the model is adjusted by concatenating key values for prefix injection, thereby realizing the fusion of graph structure features and multimodal features.
[0134] S44: Keys based on graph structural features Sum , calculate the cross weight of graph structure features ; Based on the key of the redefined multimodal feature Sum , calculate the cross weight of multimodal features ;
[0135] S45: According to the cross weight and , compute the joint embedding for modality fusion;
[0136] Joint embedding includes: Early embedding and late embedding of hidden states ;
[0137]
[0138] in, Indicates The cross-weight of multimodal features of an entity; Indicates The cross weights of the graph structure features of each entity.
[0139] Specifically, the interactive module of structural prefix injection that solves the problem of graph structure heterogeneity is applied to the multi-head self-attention mechanism. The graph structure features are distinguished from the multimodal features and are processed separately to generate queries, keys, and values of the graph structure modality. In the process of generating queries and key values from multimodal features, the key values of the graph structure features are weighted and spliced in turn by prefix injection to generate new multimodal key values and calculate the attention score. Subsequently, the cross-weights of the graph structure and multimodal features are calculated, and then multiplied with the embeddings of each modality and spliced to form the joint embedding of the entity.
[0140] S5: Entity alignment via joint embedding, combining intra-modality contrastive loss and inter-modality contrastive loss of entities.
[0141] Step S5 includes:
[0142] S51: For each pre-aligned seed pair, define a negative set by destroying the seed pair ; A seed pair is an entity pair;
[0143] S52: Adopt the intra-batch negative sampling strategy to limit the sampling of negative sets to small batches. The expression for calculating the alignment distribution probability is:
[0144]
[0145] in, , represents the temperature parameter; represents the pre-aligned seed pair; Representing Entities ; Representing Entities The transposition of the multimodal features; Representing Entities Multimodal features;
[0146] S53: For each mode of the entity , define the bidirectional alignment objective, and use the intra-modality alignment loss to constrain the input embedding; for the intra-modality alignment loss, two methods are used to calculate it: early joint embedding and late joint embedding, as follows:
[0147]
[0148]
[0149] in, represents the intra-modality contrastive loss, computed using early joint embedding, represents the late intra-modality contrastive loss, which is calculated using the late joint embedding; Indicates the use of late joint embedding Calculated alignment distribution probability;
[0150] S54: Design inter-modal alignment loss for unimodal features of entities ; then the total loss function .
[0151] Specifically, pre-aligned seed pairs represent a set of aligned entity pairs used for supervised learning and training. The multimodal entity alignment task aims to find new entity pairs using known supervised information and pre-aligned seed pairs, and predict potential alignment results.
[0152] Specifically, for the intra-modality alignment loss, early joint embedding and late joint embedding are used to calculate; for each modality, the inter-modality contrast loss is calculated. As the core of multimodal contrastive learning, two new losses are proposed on single modality and joint embedding to fully capture the dynamic characteristics of intra-modality and inter-modality and minimize the modality gap.
[0153] Specifically, the present invention obtains the embedding of entity multimodal features through a multimodal feature module, fuses structural information and multimodal information by means of graph structure prefix injection in a cross-modal fusion module, calculates cross weights, generates multimodal joint embedding of entities, calculates intra-modal contrast loss and inter-modal alignment loss through a multimodal contrast learning module, and deeply utilizes entity multimodal information to complete the multimodal entity alignment task.
[0154] The present invention also discloses an electronic device. Figure 2 , Figure 2 The electronic device 500 may include: at least one processor 501 , at least one network interface 504 , a user interface 503 , a memory 505 , and at least one communication bus 502 .
[0155] The communication bus 502 is used to realize the connection and communication between these components.
[0156] The user interface 503 may include a display screen, and the optional user interface 503 may also include a standard wired interface or a wireless interface.
[0157] The network interface 504 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).
[0158] The above are only exemplary embodiments of the present disclosure and cannot be used to limit the scope of the present disclosure. That is, any equivalent changes and modifications made according to the teachings of the present disclosure are still within the scope of the present disclosure. After considering the disclosure of the specification and the truth of practice, those skilled in the art will easily think of other embodiments of the present disclosure.
[0159] The present invention is intended to cover any variation, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary technical means in the art not described in the present disclosure. The description and examples are to be regarded as exemplary only, and the scope and spirit of the present disclosure are defined by the claims.
Claims
1. A multimodal entity alignment method based on structural prefix injection, It is characterized in that The method comprises the following steps: S1: Obtain two multimodal knowledge graphs and perform preprocessing; The two multimodal knowledge graphs include: as well as ,in Represents a collection of entities; Represents a set of relations; Represents a collection of attributes; An associated image representing the entity; A set of triples representing entity relationships; S2: Construct a graph attention network with a multi-head self-attention mechanism; use the graph attention network to learn the graph structure feature vectors of the two multimodal knowledge graphs respectively to generate graph structure features of the entity; Step S2 includes: S21: Obtaining an entity index and an adjacency matrix according to the entity set and the triple set; the adjacency matrix represents the connection relationship between the entity and its neighbor entities; S22: Pre-embed the entity through the entity index to obtain the hidden state of each entity; S23: Use the graph attention network to embed the graph structure features of the entity. For each entity pair , calculate the attention coefficient of the entity pair ,as follows in is the attention coefficient of the entity pair, Represents a splicing operation, is the weight matrix of the graph structure mode; Activation function; Is Entity The hidden state of Is Entity The hidden state of represents the learnable weight; S24: By calculating all entities The attention coefficient of the connected entity pair, obtaining the attention score , the expression is as follows: in, is the attention score of the entity pair, Representing Entity Neighbors; Represents a neighbor entity; represents the natural exponential function; represents the normalized exponential function; S25: Calculate entity based on graph attention network and attention score Neighbor aggregation , the expression is: in, is the activation function; Aggregate the neighbors computed by the graph attention network , as an entity The graph structure characteristics of ; S3: Using the graph attention network to learn the multimodal information of the two multimodal knowledge graphs respectively, and generate multimodal features of the entity; Step S3 includes: The multimodal information includes: associated pictures, relationship modal information and attribute modal information; S31: For entities The relationship modal information and attribute modal information of the entity are preprocessed using bag-of-words features and one-hot encoding to obtain the entity relational modal input and attribute modal input; S32: Use pre-trained image encoder to Related pictures Preprocess and get the entity Corresponding related pictures Visual modal input ,as follows: in, Indicates that VGG-16 is selected as the pre-trained image encoder; S33: For entities relational modal input, attribute modal input, and visual modal input , use the fully connected network to process and obtain the multimodal features of the entity, as follows: in, and Represent relational modality, attribute modality and visual modality respectively, The weight matrix representing the linear transformation; Representing modality Fully connected network layer; Indicates Multimodal features of entities; multimodal features include: relation embedding, attribute embedding and visual embedding; S34: Use the Gaussian modal noise masking mechanism to process the multimodal features and generate the final multimodal features, as follows: The probability transformation of each modal feature of the multimodal feature is as follows: in, represents a uniformly distributed random variable that determines whether to apply noise, is the threshold probability of adding noise to each multimodal feature, is the masking rate; Multimodal features The noise vector The definition is as follows: in, and denote the standard deviation and mean of the modal noise data, respectively, and z denotes the distribution from Gaussian The sample drawn from represents the unit covariance matrix; Represents the dot product operation; S4: Through the structural prefix injection interaction module of the graph attention network, the graph structure features and multimodal features are combined to obtain a joint embedding; Step S4 includes: S41: Use the Vanilla Transformer model to process the graph structure features of entities and generate queries for graph structure features ,key Sum : in, is the graph structure feature; Represent the weight matrices for query, key, and value respectively; S42: Calculate the score of each attention head of the graph structure feature through the multi-head self-attention mechanism , determine the output of multi-head self-attention ; Using layer normalization and residual connections to process the output of multi-head attention ,as follows: in, Represents the graph structure features after layer normalization and residual connection processing; Representation layer normalization operation; and represents the weight matrix used for linear transformation; and represents the bias used for linear transformation; is the activation function; represents a fully connected feed-forward network layer; It is the structural feature of the processed graph; S43: Keys based on graph structural features Sum , combined with the structural prefix injection interaction module, redefine the key of multimodal features Sum The calculation is as follows: in, and is the redefined key value; is the key-value concatenation weight; Represents a splicing operation; Representing multimodal features of entities; Indicates that it is used for modal The weight matrix of the key; Indicates that it is used for modal The weight matrix of the values; Use keys that redefine multimodal features Sum , process the multimodal features and obtain the processed multimodal features ; S44: Keys based on graph structural features Sum , calculate the cross weight of graph structure features ; Based on the key of the redefined multimodal feature Sum , calculate the cross weight of multimodal features ; S45: According to the cross weight and , compute the joint embedding for modality fusion; Joint embedding includes: Early embedding and late embedding of hidden states ; in, Indicates The cross-weight of multimodal features of an entity; Indicates The cross weight of the graph structure features of each entity; S5: Entity alignment via joint embedding, combining intra-modality contrastive loss and inter-modality contrastive loss of entities.
2. A multimodal entity alignment method based on structural prefix injection as claimed in claim 1, It is characterized in that Step S5 includes: S51: For each pre-aligned seed pair, define a negative set by destroying the seed pair ; A seed pair is an entity pair; S52: Adopt the intra-batch negative sampling strategy to limit the sampling of negative sets to small batches. The expression for calculating the alignment distribution probability is: in, , represents the temperature parameter; represents the pre-aligned seed pair; Representing Entities ; Representing Entities Transpose of the multimodal features; Representing Entities Multimodal features; S53: For each mode of the entity , define the bidirectional alignment objective, and use the intra-modality alignment loss to constrain the input embedding; for the intra-modality alignment loss, two methods are used to calculate it: early joint embedding and late joint embedding, as follows: in, represents the intra-modality contrastive loss, computed using early joint embedding, represents the late intra-modality contrastive loss, which is calculated using the late joint embedding; Indicates the use of late joint embedding Calculated alignment distribution probability; S54: Design inter-modal alignment loss for unimodal features of entities ; then the total loss function .
3. An electronic device, It is characterized in that The electronic device comprises a processor (501), a memory (505), a user interface (503) and a network interface (504), wherein the memory (505) is used to store instructions, the user interface (503) and the network interface (504) are used to communicate with other devices, and the processor (501) is used to execute the instructions stored in the memory (505) so that the electronic device executes the method according to any one of claims 1 to 2.
4. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores instructions, and when the instructions are executed by a computer, the method steps as described in any one of claims 1-2 are executed.
