An entity linking method based on multimodal feature fusion

Abstract

The present invention discloses an entity linking method based on multimodal feature fusion. First, the relevant multimodal information of the mentioned target and candidate entities is input into the semantic enhancement module, and the features of the mentioned text and image are enhanced respectively. Then, the method passes through a deep attention network to fully capture the semantic relevance between heterogeneous modal information. Then, a graph convolutional neural network is used to realize the fusion of multimodal features, and the corresponding multimodal representations of the mentioned target and candidate entities are obtained respectively. Finally, the candidate entities are sorted according to the similarity between the multimodal representations of the mentioned target and the candidate entity, and the candidate entity with the largest similarity is selected as the predicted linking entity.

Description

Technical Field

[0001] The present invention belongs to the technical field of natural language processing, and more specifically, relates to an entity linking method based on multimodal feature fusion. Background Art

[0002] Multimodal entity linking is a natural language processing task that aims to link mentions in text to entities in a knowledge base, thereby disambiguating entities. Multimodal entity linking has a wide range of applications in areas such as semantic understanding, question-answering, information mining, and knowledge graph construction.

[0003] In text-based, unimodal entity linking methods, entity linking primarily relies on features such as prior knowledge of the entity, entity type information, and textual information based on the entity's context. However, both prior knowledge and entity type information rely on manual statistical analysis. Therefore, mainstream entity linking methods rely on textual information for entity disambiguation. Text-based entity linking models have achieved good experimental performance in linking long texts, as long texts can provide rich semantic information related to the entity. However, linking performance for short texts is less satisfactory.

[0004] Short texts contain limited contextual information, and thus limited semantic information that can be used for disambiguation. This leads to a ceiling on the effectiveness of linking schemes that rely primarily on textual features for short texts, making it impossible to achieve accurate and reliable links based solely on textual information. The rise of multimodal data presents new opportunities for short text entity linking. Providing multimodal information, such as entity-related images, in entity linking tasks can help models gain a more comprehensive understanding of the data, enhancing their robustness and effectiveness, and thus potentially improving the performance of short text entity linking. Summary of the Invention

[0005] The purpose of the present invention is to overcome the shortcomings of the existing technology and provide an entity linking method based on multimodal feature fusion, which introduces image features and attribute features related to entities to complete the short text entity linking task, with the advantages of high linking accuracy and no need for artificial features.

[0006] To achieve the above-mentioned object of the invention, the present invention provides an entity linking method based on multimodal feature fusion, characterized by comprising the following steps:

[0007] (1) Download the entity linking dataset;

[0008] Download the entity link dataset, where each link sample contains the contextual information and image information of the mentioned target m;

[0009] Download the candidate entity set corresponding to the target m mentioned in each link sample, and each candidate entity e r Contains description text, image information and attribute information;

[0010] (2) Embedding representations of text and images;

[0011] The context information and image information of the mentioned target m are input into the BERT encoder and ResNet encoder respectively to obtain the context representation T of the mentioned target m and image representation V m ;

[0012] The candidate entity e r The description text and image information are input into the BERT encoder and ResNet encoder respectively to obtain the candidate entity e r Text representation T er and image representation V er ;

[0013] (3) semantic enhancement of different modal features of the mentioned target m;

[0014] The context representation T of the mentioned target m and candidate entity e r Text representation T er The two are input together into the semantic enhancement module, and the enhanced context representation T' is output. m ;

[0015] The image representation V of the mentioned target m m and the image representation V of the candidate entity er The image representation V' after the target enhancement is output. m ;

[0016] (4) Complementary enhancement of text and image representations;

[0017] The enhanced context representation T' of the mentioned target m m and image representation V' m Input into the deep attention network and output the context feature vector t of the mentioned target m m and image feature vector v m ;

[0018] The candidate entity e r Text representation T er and image representation V er Input into the deep attention network and output the text feature vector t of the candidate entity er and image feature vector v er ;

[0019] (5) Attribute feature extraction;

[0020] The candidate entity e r The attribute information of the candidate entity is input into the attribute feature extraction module, and the attribute feature vector p of the candidate entity is output. er ;

[0021] (6) Extracting multimodal representations;

[0022] (6.1) Construct the initialization feature matrix H 0 ;

[0023] H 0 =[t m ,v m ,t e1 ,v e1 ,p e1 ,…,t er ,v er ,p er ,…,t eR ,v eR ,p eR ]

[0024] H 0 Each element in H is considered as a node, so 0 Simplified to:

[0025]

[0026] in, And so on, R represents the number of candidate entities, t er ,v er ,p er Represents the rth candidate entity e r Text feature vector, image feature vector and attribute feature vector;

[0027] (6.2), construct the initial adjacency matrix A 0 ;

[0028]

[0029] Among them, a ij Represents the weight value of the relationship edge connecting the i-th node and the j-th node;

[0030] (6.3) Update the initialized feature matrix and the initialized adjacency matrix through loop iteration;

[0031]

[0032] Among them, l represents the current number of iterations, represents the weight matrix, σ(·) represents the activation function, and M represents the size of n×n mask matrix. The elements in the matrix satisfy:

[0033]

[0034] (6.4), multimodal representation extraction;

[0035] After L cycles of iteration, the feature matrix is ​​recorded as H L , the adjacency matrix is ​​recorded as A L ;

[0036] Extract feature matrix H L Chinese elements Multimodal representation f as a mention target m , extract the feature matrix H L middle As multimodal representations f of 1 to R candidate entities respectively er ;

[0037] (7) Extract the entities that need to be linked to the mentioned target;

[0038] Calculate the mention target m and each candidate entity e r The similarity of multimodal representation is:

[0039] S(m,e r )=cos(f m ,f er )

[0040] Among them, cos() represents the cosine similarity calculation, S(m,e r ) indicates the mention target m and the rth candidate entity e r The similarity between them;

[0041] Select the candidate entity with the greatest similarity for prediction output:

[0042]

[0043] Among them, e * Indicates the entity to which the mention target m is to be linked.

[0044] The object of the invention of the present invention is achieved like this:

[0045] The present invention is based on an entity linking method based on multimodal feature fusion. First, the relevant multimodal information of the mentioned target and candidate entity is input into the semantic enhancement module, and the feature enhancement of the mentioned text and image is performed respectively. Then, the method passes through a deep attention network to fully capture the semantic relevance between heterogeneous modal information. Then, a graph convolutional neural network is used to realize the fusion of multimodal features, and the corresponding multimodal representations of the mentioned target and candidate entity are obtained respectively. Finally, the candidate entities are sorted according to the similarity between the multimodal representations of the mentioned target and the candidate entity, and the candidate entity with the largest similarity is selected as the predicted link entity.

[0046] At the same time, the entity linking method based on multimodal feature fusion of the present invention also has the following beneficial effects:

[0047] (1) The semantic enhancement module can enhance the information features of the mentions and reduce the noise based on the attention weight value, thereby enriching the feature representation of the mentioned target. At the same time, the design of the gated fusion strategy helps to alleviate the problem of feature loss during network transmission.

[0048] (2) The design of the deep attention network can fully capture the semantic correlation between heterogeneous features and at the same time play the role of feature complementarity enhancement, thereby enhancing the representation ability of the model.

[0049] (3) Utilize graph structures to achieve the transmission and dynamic fusion of different modal features, thereby obtaining a more accurate and robust multimodal representation of the entity. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is a flow chart of the entity linking method based on multimodal feature fusion of the present invention;

[0051] Figure 2 is the semantic enhancement model graph;

[0052] Figure 3 This is a diagram of the deep attention network structure;

[0053] Figure 4 It is a schematic diagram of the graph structure. DETAILED DESCRIPTION

[0054] The following describes the specific embodiments of the present invention in conjunction with the accompanying drawings so that those skilled in the art can better understand the present invention. It should be noted that in the following description, when detailed descriptions of known functions and designs may dilute the main content of the present invention, such descriptions will be omitted here.

[0055] Example

[0056] Figure 1 This is a flow chart of the entity linking method based on multimodal feature fusion of the present invention.

[0057] In this embodiment, if Figure 1 As shown, the present invention provides an entity linking method based on multimodal feature fusion, comprising the following steps:

[0058] S1, download the entity linking dataset;

[0059] Download the entity link dataset, where each link sample contains the contextual information and image information of the mentioned target m;

[0060] Download the candidate entity set corresponding to the target m mentioned in each link sample, and each candidate entity e r Contains description text, image information and attribute information;

[0061] S2, embedding representation of text and images;

[0062] In this embodiment, the BERT encoder is used for text information to extract the corresponding feature vector for each word in the text sequence and generate the corresponding text representation. The ResNet encoder is used for image information to divide the image into several regions and extract the corresponding feature vector for each region to generate the corresponding image representation.

[0063] Therefore, the context information and image information of the mentioned target m are input into the BERT encoder and ResNet encoder respectively to obtain the context representation T of the mentioned target m and image representation V m ;

[0064] Similarly, the candidate entity e r The description text and image information are input into the BERT encoder and ResNet encoder respectively to obtain the candidate entity e r Text representation T er and image representation V er ;

[0065] S3, semantic enhancement of different modal features of the mentioned target m;

[0066] Represent the context of the mentioned target T m and candidate entity e r Text representation T er The two are input together into the semantic enhancement module, and the enhanced context representation T' is output. m ;

[0067] In this embodiment, if Figure 2 As shown, the semantic enhancement module includes a modality enhancement part and a gated fusion part;

[0068] Among them, the modality enhancement part mainly consists of the attention-based embedding matrix and the softmax layer;

[0069] The context representation T of the mentioned target m m and candidate entity e r Text representation T er Input to the modality enhancement part, first mapped to the same embedding matrix to obtain the attention score matrix S:

[0070]

[0071] Where A is the embedding matrix;

[0072] Then extract the maximum value of each column in the attention score matrix S and calculate it through the softmax function to get the corresponding attention weight value sequence, and then represent the context T of the mentioned target m. m Multiplying the corresponding element of the weight value sequence to obtain the context representation T after modality enhancement of the mentioned target m att ;

[0073] The gated fusion part mainly consists of an average pooling layer, a splicing layer, and a fully connected layer;

[0074] The context representation T of the mentioned target m m and the context representation T after modality enhancement att The input is sent to the gated fusion part, and the vectors are concatenated after passing through the average pooling layer. The vectors are then sent to the fully connected layer to output the fusion ratio coefficients g1 and g2:

[0075] [g1,g2]=f(concat(Avg(T m ),Avg(T att )))

[0076] Among them, Avg() represents the average pooling operation, concat() represents vector concatenation, and f() represents the fully connected layer;

[0077] Then the context representation T of the mentioned target m is m and the context representation T after modality enhancement att According to the fusion ratio coefficient, the addition operation is performed to output the context representation T' after the target is mentioned. m :

[0078] T' m =g1*T m +g2*T att

[0079] Similarly, the image representation V of the mentioned target m m and the image representation V of the candidate entity er The image representation V' after the target enhancement is output.m .

[0080] S4, complementary enhancement of text and image representations;

[0081] The enhanced context representation T' of the mentioned target m m and image representation V' m Input into the deep attention network and output the context feature vector t of the mentioned target m m and image feature vector v m ;

[0082] In this embodiment, if Figure 3 As shown in Figure 2, the deep attention network mainly consists of a text attention layer, a visual attention layer, and a maximum pooling layer;

[0083] Among them, the text attention layer and the visual attention layer are both composed of a multi-head attention mechanism, a normalization layer, a feedforward neural network and a normalization layer in series. The difference lies in the different inputs;

[0084] The enhanced context representation T' of the mentioned target m m and image representation V' m Input to the deep attention network, and then pass through the visual attention layer, text attention layer and maximum pooling layer to output the context feature vector t of the target m m ;

[0085] The enhanced context representation T' of the mentioned target m m and image representation V' m Input to the visual attention layer and output the image representation after complementary enhancement of the mentioned target m

[0086]

[0087] Where VisualLayler() represents the visual attention layer;

[0088] The enhanced context representation T' of the mentioned target m m and image representation V' m Input to the text attention layer and output the context representation after complementary enhancement of the mentioned target m

[0089]

[0090] Where TextLayler() represents the text attention layer;

[0091] Then the enhanced context representation T' of the mentioned target m is m and complementary enhanced image representation Input into the text attention layer, and then pass through the maximum pooling layer to output the context feature vector t of the target m m :

[0092]

[0093] Among them, Maxpool() represents the maximum pooling layer;

[0094] Finally, we will mention the context representation of the target m after complementary enhancement and image representation V' m Input into the visual attention layer, and then pass through the maximum pooling layer to output the image feature vector v of the target m m :

[0095]

[0096] Similarly, the candidate entity e r Text representation T er and image representation V er Input into the deep attention network and output the text feature vector t of the candidate entity er and image feature vector v er .

[0097] S5, attribute feature extraction;

[0098] The candidate entity e r The attribute information of the candidate entity is input into the attribute feature extraction module, and the attribute feature vector p of the candidate entity is output. er ;

[0099] In this embodiment, the attribute feature extraction module includes a coding layer and a one-dimensional convolution layer;

[0100] The candidate entity e r The attribute information is input into the encoding layer, and the BERT model is used to encode the attribute representation sequence P e ={p1,p2,…,p l}, where l represents the length of the attribute representation sequence;

[0101] The attribute representation sequence is input into the convolution layer. The convolution layer sets three convolution kernels of different scales, with the sizes of the convolution kernels being 2, 3, and 4 respectively. A one-dimensional convolution operation is performed on the kth convolution kernel. The feature value extracted at the i-th position of the attribute representation sequence is recorded as f i k :

[0102] f i k =ReLU(conv(p i ;...;p i+k-1 ))

[0103] Where conv() represents a one-dimensional convolution operation and ReLU() represents an activation function;

[0104] Then select the maximum value as the eigenvalue f at the i-th position i :

[0105] f i =max(f i k ),k=2,3,4

[0106] Finally, the feature values ​​at each position are combined to output the entity attribute feature vector p of the candidate entity er :

[0107] p er ={f1,f2,…,f l}

[0108] S6, extracting multimodal representations;

[0109] S6.1. Construct the initialization feature matrix H 0 ;

[0110] In this embodiment, if Figure 4 As shown, the graph structure includes nodes and edges;

[0111] For the mentioned target, its context information and image information are abstracted into two nodes: context node and image node. The initialization values ​​of the two nodes are the context feature vector t of the mentioned target m. m and image feature vector v m ;

[0112] For each candidate entity e r , its text information, image information and attribute information are abstracted into three nodes: text node, image node and attribute node, and the initialization values ​​of the three nodes are candidate entity e r The text feature vector t er , image feature vector v er and attribute feature vector p er ;

[0113] Finally, the initial values ​​of all nodes form the characteristic matrix H 0

[0114] H 0 =[t m ,v m ,t e1 ,v e1 ,p e1 ,…,t er ,v er ,p er,…,t eR ,v eR ,p eR ]

[0115] H 0 Each element in H is considered as a node, so 0 Simplified to:

[0116]

[0117] in, And so on, R represents the number of candidate entities, t er ,v er ,p er Represents the rth candidate entity e r Text feature vector, image feature vector and attribute feature vector;

[0118] S6.2. Construct the initial adjacency matrix A 0 ;

[0119] Five types of relationship edges are established to show the relationship between nodes. A relationship edge is established between the context node of the target m and the text node of the candidate entity, a relationship edge is established between the context node of the target m and the image node of the candidate entity, a relationship edge is established between the image node of the target m and the text node of the candidate entity, a relationship edge is established between the image node of the target m and the image node of the candidate entity, and a relationship edge is established between the text node of the candidate entity and the attribute node of the candidate entity. The initial value of the edge is the cosine similarity between the node values; therefore, the relationship between all nodes constructs the initial adjacency matrix A 0 ;

[0120]

[0121] Among them, a ij Represents the weight value of the relationship edge connecting the i-th node and the j-th node;

[0122] S6.3. Update the initialized feature matrix and the initialized adjacency matrix through a loop iteration method;

[0123]

[0124] Among them, l represents the current number of iterations, represents the weight matrix, σ(·) represents the activation function, and M represents the size of n×n mask matrix. The elements in the matrix satisfy:

[0125]

[0126] S6.4, Multimodal Representation Extraction;

[0127] After L cycles of iteration, the feature matrix is ​​recorded as H L , the adjacency matrix is ​​recorded as A L ;

[0128] Extract feature matrix H L Chinese elements Multimodal representation f as a mention target m , extract the feature matrix H L middle As multimodal representations f of 1 to R candidate entities respectively er ;

[0129] S7, extract the entities that need to be linked to the mentioned target;

[0130] Calculate the mention target m and each candidate entity e r The similarity of multimodal representation is:

[0131] S(m,e r )=cos(f m ,f er )

[0132] Among them, cos() represents the cosine similarity calculation, S(m,e r ) indicates the mention target m and the rth candidate entity e r The similarity between them;

[0133] Select the candidate entity with the greatest similarity for prediction output:

[0134]

[0135] Among them, e * Indicates the entity to which the mention target m is to be linked.

[0136] Experiment and result analysis

[0137] This example verifies the effectiveness of the Entity Linking Technology Based on Multimodal Feature Fusion (ELT-MFF) through a series of experiments.

[0138] Download two multimodal entity linking datasets, WikiMEL and WikiDiverse. Each sample in the dataset contains the text and image information corresponding to the mention, the candidate entity set corresponding to the mention, and the text, image, and attribute information corresponding to each candidate entity;

[0139] The WikiMEL dataset contains more than 20,000 multimodal samples, including common entity types such as names of people, places, and organizations. The number of candidate entities for each mention in the dataset is 100.

[0140] The WikiDiverse dataset is constructed from news media such as Wikinews and BBC, as well as the Wikipedia database. The constructed dataset covers a wide range of topics and entities; the number of candidate entities for each mention is 10;

[0141] This example is based on the open-source machine learning library Pytorch, with Pytorch version 1.13.1, Python version 3.9, and CUDA version 11.6. The experimental hardware environment GPU model is NVIDIA GeForce RTX3090Ti, with a video memory size of 24.0GB.

[0142] The entity linking method based on multimodal feature fusion is experimented and performance analyzed on the WikiMEL and WikiDiverse datasets. The datasets are divided into training set, test set and validation set.

[0143] In order to verify the effectiveness of the entity linking method based on multimodal feature fusion proposed in the present invention, this embodiment selects some representative models since the development of multimodal entity linking technology for comparison. The evaluation index adopted in the experiment is Top-k, which is an experimental index used to predict the proportion of the correct labels contained in the top k results with the highest probability. The number of candidate entities corresponding to each mention in WikiMEL is 100, and Top-1, Top-5, Top-10 and Top-20 are used as evaluation indicators of the experiment. The number of candidate entities corresponding to each mention in WikiDiverse is 10, and the three experimental evaluation indicators of Top-1, Top-3 and Top-5 are used. The results of the comparative experiment are shown in Table 1.

[0144]

[0145] Table 1 Comparison of experimental results of different models on multimodal datasets

[0146] The experimental results in Table 1 show that ELT-MMF outperforms other models in all Top-k metrics on the WikiMEL dataset. In the Top-1 metric, ELT-MMF achieved an accuracy of 66.2%, 0.7 percentage points higher than the DRIN model. In the Top-5, Top-10, and Top-20 metrics, the ELT-MMF model showed significant performance improvements over the DRIN model. Notably, among the compared models, the ELT-MMF model outperformed the GHMFC model in the Top-1 metric by 22.3%, validating the effectiveness of the ELT-MMF model.

[0147] In experimental results on the WikiDiverse dataset, the ELT-MMF model achieved a Top-1 score of 60.2%, achieving the best performance among the compared models and significantly improving upon the DRIN model by 9.1%. Furthermore, the ELT-MMF model achieved improvements of 1.4% and 2.2% in the Top-3 and Top-5 metrics, respectively, compared to the DRIN model.

[0148] From the overall analysis, the G-MFDA model proposed in this paper achieved good experimental results on the multimodal datasets WikiMEL and WikiDiverse datasets, and the Top-k indicators were better than the mainstream models compared, verifying the effectiveness and advancement of the ELT-MMF model proposed in this paper.

[0149] Although the above describes the illustrative specific embodiments of the present invention to facilitate understanding of the present invention by those skilled in the art, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the appended claims, these changes are obvious, and all inventions and creations using the concepts of the present invention are protected.

Claims

1. An entity linking method based on multimodal feature fusion, characterized in that: The following steps are involved: (1) Download the entity linking dataset; Download the entity link dataset, where each link sample contains the contextual information and image information of the mentioned target m; Download the candidate entity set corresponding to the target m mentioned in each link sample, and each candidate entity e r Contains description text, image information and attribute information; (2) Embedding representations of text and images; The context information and image information of the mentioned target m are input into the BERT encoder and ResNet encoder respectively to obtain the context representation T of the mentioned target m and image representation V m ; The candidate entity e r The description text and image information are input into the BERT encoder and ResNet encoder respectively to obtain the candidate entity e r Text representation T er and image representation V er ; (3) semantic enhancement of different modal features of the mentioned target m; Represent the context of the mentioned target T m and candidate entity e r Text representation T er The two are input together into the semantic enhancement module, and the enhanced context representation T of the mentioned target is output. m '; The image representation V of the mentioned target m m and the image representation V of the candidate entity er The two images are input into the semantic enhancement module together, and the enhanced image representation V is output. m '; (4) Complementary enhancement of text and image representations; The enhanced context representation T of the mentioned target m m 'and image representation V' m Input into the deep attention network and output the context feature vector t of the mentioned target m m and image feature vector v m ; The candidate entity e r Text representation T er and image representation V er Input into the deep attention network and output the text feature vector t of the candidate entity er and image feature vector v er ; (5) Attribute feature extraction; The candidate entity e r The attribute information of the candidate entity is input into the attribute feature extraction module, and the attribute feature vector p of the candidate entity is output. er ; (6) Extracting multimodal representations; (6.1) Construct the initialization feature matrix H 0 ; H 0 =[t m ,v m ,t e1 ,v e1 ,p e1 ,…,t er ,v er ,p er ,…,t eR ,v eR ,p eR ] H 0 Each element in H is considered as a node, so 0 Simplified to: in, And so on, R represents the number of candidate entities, t er ,v er ,p er Represents the rth candidate entity e r Text feature vector, image feature vector and attribute feature vector; (6.2), construct the initial adjacency matrix A 0 ; Among them, a ij Represents the weight value of the relationship edge connecting the i-th node and the j-th node; (6.3) Update the initialized feature matrix and the initialized adjacency matrix through loop iteration; Among them, l represents the current iteration number, represents the weight matrix, σ(·) represents the activation function, and M represents the size of n×n mask matrix. The elements in the matrix satisfy: (6.4), multimodal representation extraction; After L cycles of iteration, the feature matrix is ​​recorded as H L , the adjacency matrix is ​​recorded as A L ; Extract feature matrix H L Chinese elements Multimodal representation f as a mention target m , extract the feature matrix H L middle As multimodal representations f of 1 to R candidate entities respectively er ; (7) Extract the entities that need to be linked to the mentioned target; Calculate the mention target m and each candidate entity e r The similarity of multimodal representation is: S(m,e r )=cos(f m ,f er ) Among them, cos() represents the cosine similarity calculation, S(m,e r ) indicates the mention target m and the rth candidate entity e r The similarity between them; Select the candidate entity with the greatest similarity for prediction output: Among them, e * Indicates the entity to which the mention target m is to be linked.

2. The entity linking method based on multimodal feature fusion according to claim 1, characterized in that: The semantic enhancement module includes a modality enhancement part and a gated fusion part; Among them, the modality enhancement part mainly consists of the attention-based embedding matrix and the softmax layer; The context representation T of the mentioned target m m and candidate entity e r Text representation T er Input to the modality enhancement part, first mapped to the same embedding matrix to obtain the attention score matrix S: Where A is the embedding matrix; Then extract the maximum value of each column in the attention score matrix S and calculate it through the softmax function to get the corresponding attention weight value sequence, and then represent the context T of the mentioned target m. m Multiplying the corresponding element of the weight value sequence to obtain the context representation T after modality enhancement of the mentioned target m att ; The gated fusion part mainly consists of an average pooling layer, a splicing layer, and a fully connected layer; The context representation T of the mentioned target m m and the context representation T after modality enhancement att The input is sent to the gated fusion part, and the vectors are concatenated after passing through the average pooling layer. The vectors are then sent to the fully connected layer to output the fusion ratio coefficients g1 and g2: [g1,g2]=f(concat(Avg(T m ),Avg(T att ))) Among them, Avg() represents the average pooling operation, concat() represents vector concatenation, and f() represents the fully connected layer; Then the context representation T of the mentioned target m is m and the context representation T after modality enhancement att According to the fusion ratio coefficient, the addition operation is performed to output the enhanced context representation T of the mentioned target. m ': T m '=g1*T m +g2*T att Similarly, the image representation V of the mentioned target m m and the image representation V of the candidate entity er The two images are input into the semantic enhancement module together, and the enhanced image representation V is output. m '.

3. The entity linking method based on multimodal feature fusion according to claim 1, characterized in that: The deep attention network mainly consists of a text attention layer, a visual attention layer and a maximum pooling layer; Among them, the text attention layer and the visual attention layer are both composed of a multi-head attention mechanism, a normalization layer, a feedforward neural network and a normalization layer in series. The difference lies in the different inputs; The enhanced context representation T' of the mentioned target m m and image representation V' m Input to the deep attention network, pass through the visual attention layer, text attention layer and pooling layer in sequence to output the context feature vector t of the target m m ; The enhanced context representation T' of the mentioned target m m and image representation V' m Input to the visual attention layer and output the image representation after complementary enhancement of the mentioned target m Where VisualLayler() represents the visual attention layer; The enhanced context representation T' of the mentioned target m m and image representation V' m Input to the text attention layer and output the context representation after complementary enhancement of the mentioned target m Where TextLayler() represents the text attention layer; Then the enhanced context representation T' of the mentioned target m is m and complementary enhanced image representation Input into the text attention layer, and then pass through the maximum pooling layer to output the context feature vector t of the target m m : Among them, Maxpool() represents the maximum pooling layer; Finally, we will mention the context representation of the target m after complementary enhancement and image representation V' m Input into the visual attention layer, and then pass through the maximum pooling layer to output the image feature vector v of the target m m : Similarly, the candidate entity e r Text representation T er and image representation V er Input into the deep attention network and output the text feature vector t of the candidate entity er and image feature vector v er .

4. The entity linking method based on multimodal feature fusion according to claim 1, characterized in that: The attribute feature extraction module includes a coding layer and a one-dimensional convolution layer; The candidate entity e r The attribute information is input into the encoding layer, and the BERT model is used to encode the attribute representation sequence P e ={p1,p2,…,p l }, where l represents the length of the attribute representation sequence; The attribute representation sequence is input into the convolution layer. The convolution layer sets three convolution kernels of different scales, with the sizes of the convolution kernels being 2, 3, and 4 respectively. A one-dimensional convolution operation is performed on the kth convolution kernel. The feature value extracted at the i-th position of the attribute representation sequence is recorded as f i k : f i k =ReLU(conv(p i ;...;p i+k-1 )) Where conv() represents a one-dimensional convolution operation, and ReLU() represents an activation function; Then select the maximum value as the eigenvalue f at the i-th position i : f i =max(f i k ),k=2,3,4 Finally, the feature values ​​at each position are combined to output the entity attribute feature vector p of the candidate entity er : p er ={f1,f2,…,f l }。

Images