Method and system for multi-modal named entity recognition based on knowledge graph enhancement

Abstract

This invention discloses a multimodal named entity recognition method and system based on knowledge graph enhancement. The method first inputs text and images into a feature extraction module to obtain initial text and visual features, respectively. The text is then input into a candidate entity recognition module to identify and obtain a set of candidate entities. Next, using the candidate entities as query objects, a triplet knowledge set and an entity-associated image set are constructed and encoded to obtain sets of triplet knowledge features and entity-associated image feature representations, respectively. Then, text and visual features are enhanced, and cross-modal feature fusion is performed to obtain a text representation that incorporates visual information. Finally, the text representation with fused visual information is fused with the initial text features to obtain the final text representation, which is then input into a conditional random field decoder to output the named entity recognition result. This invention effectively alleviates entity ambiguity caused by insufficient text and image information and significantly improves the accuracy of multimodal named entity recognition results.

Description

Technical Field

[0001] This invention belongs to the field of multimodal named entity recognition, specifically involving a multimodal named entity recognition method and system based on knowledge graph enhancement, which is suitable for accurately identifying named entities in text from image-text pairs. Background Technology

[0002] Multimodal Named Entity Recognition (MNER) identifies and classifies entities within text by fusing textual and image information. In scenarios such as social media, text is often concise and has limited context, making it difficult to determine entity types solely based on textual information. Existing methods typically incorporate image information as an auxiliary modality to supplement the semantic information of the text. However, in text-image pairs lacking sufficient contextual information, relying solely on textual and image information may be insufficient for accurate entity identification. For example, when the text lacks explicit contextual information and the image does not provide relevant clues, the model may introduce noise, thus affecting the entity recognition results.

[0003] To address this, existing research has attempted to introduce external knowledge to provide additional information for multimodal named entity recognition. However, current research primarily enhances textual representation by introducing external textual knowledge, with relatively limited utilization of visual knowledge related to entity associations. Image resources associated with entities can provide information on their appearance features, brand identifiers, or specific scenes, helping to alleviate ambiguity caused by insufficient textual and image information. Furthermore, in the knowledge acquisition process, document-based retrieval methods are prone to introducing redundant information, making it difficult to guarantee the precise relevance between knowledge and target entities; while methods based on large language models to generate knowledge typically face high computational costs.

[0004] Therefore, how to choose a suitable knowledge representation form and perform effective knowledge enhancement to improve the performance of multimodal named entity recognition remains an urgent problem to be solved. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the present invention aims to provide a multimodal named entity recognition method and system based on knowledge graph enhancement. This method retrieves corresponding knowledge subgraphs and associated visual resources from the Wikidata knowledge graph, using candidate entities as the core, and enhances both textual and visual features. Subsequently, a fused text representation is obtained through a cross-modal interaction module, and a gating mechanism is further used to adaptively fuse the fused text representation and the original text representation.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows: In one aspect, this invention provides a multimodal named entity recognition method based on knowledge graphs, comprising the following steps: Step 1: Input the text and image into the feature extraction module to obtain the initial text features. Compared with initial visual features .

[0007] Step 2: Input the text into the candidate entity recognition module, identify the candidate entities in the text, and obtain the candidate entity set. .

[0008] Step 3: Using candidate entities as query objects, extract local knowledge subgraphs related to each candidate entity from the knowledge graph, and convert the subgraphs into triples to construct a triple knowledge set. Simultaneously, external image data associated with candidate entities are acquired to construct an entity-associated image set. .

[0009] Step 4: Set up triplet knowledge Image set associated with entities The input knowledge feature extraction module performs encoding processing to obtain triplet knowledge feature representation sets. Set of image feature representations associated with entities .

[0010] Step 5: Enhance the initial text representation using triple knowledge sets Simultaneously, it integrates the feature representation set of entity-related images with the initial visual features. The enhanced text features are obtained. and visual features .

[0011] Step Six: Enhance the text features With enhanced visual features The input cross-modal interaction module performs cross-modal feature fusion to obtain a text representation that incorporates visual information. .

[0012] Step 7: Integrate visual information into the text representation Compared with initial text features The two parts of information are merged and weighted using a gating mechanism to obtain the final text representation. ; Represent the final text Input a conditional random field decoder and output named entity recognition results.

[0013] The technical solution of this invention includes a feature extraction module, a candidate entity recognition module, a knowledge feature extraction module, a knowledge enhancement module, a cross-modal interaction module, and a gating fusion module, and outputs entity prediction results through a conditional random field decoder.

[0014] The input text and image first pass through a feature extraction module to obtain initial textual and visual features. Then, a candidate entity recognition module identifies candidate entities in the text and retrieves relevant knowledge information from a knowledge graph centered on these candidate entities. The retrieved structured knowledge and visual resources are encoded in the knowledge feature extraction module. In the knowledge enhancement module, the aforementioned knowledge information is used to enhance both textual and visual features. Subsequently, a cross-modal interaction module fuses the enhanced multimodal features to obtain a text representation incorporating visual information. This fused visual representation is then fused with the original textual features using a gating mechanism to preserve original semantic information and suppress potential noise interference, resulting in the final text representation. Finally, this text representation is input into a conditional random field decoder for sequence labeling prediction, outputting the named entity recognition result.

[0015] By introducing external knowledge information, this invention can effectively supplement entity-related information when the original textual semantic information is insufficient, thereby improving the accuracy of multimodal named entity recognition.

[0016] In another aspect, the present invention also provides a knowledge graph-enhanced multimodal named entity recognition system for the aforementioned multimodal named entity recognition method, comprising the following units: The feature extraction and candidate entity recognition unit is used to input text and images into the feature extraction module to obtain initial text features. The initial visual features are used, and the text is input into the candidate entity recognition module to identify candidate entities in the text and obtain a candidate entity set.

[0017] The feature representation unit is used to construct a triple knowledge set and an entity association image set using candidate entities in the candidate entity set as query objects. The triple knowledge set and the entity association image set are encoded to obtain the triple knowledge feature representation set and the entity association image feature representation set, respectively.

[0018] The feature enhancement unit is used to enhance the initial text representation by utilizing the triplet knowledge set, and at the same time, it integrates the feature representation set of entity association images with the initial visual features to obtain enhanced text features and visual features. The visual fusion unit is used to input the enhanced text features and enhanced visual features into the cross-modal interaction module for cross-modal feature fusion, so as to obtain a text representation with fused visual information.

[0019] Named entity recognition output unit, used to combine text representations fused with visual information with initial text features. The data is fused to obtain the final text representation, which is then input into the conditional random field decoder to output the named entity recognition result.

[0020] Compared with the prior art, the beneficial effects of the present invention are reflected in: 1. Unlike existing multimodal named entity recognition technologies that primarily rely on external textual knowledge to enhance textual representations while making limited use of visual knowledge related to entities, thus failing to alleviate ambiguity caused by insufficient textual and graphical information through visual information, this invention focuses on candidate entities. It retrieves corresponding knowledge subgraphs and associated visual resources from external knowledge graphs and simultaneously enhances textual and visual features using both types of knowledge. This not only supplements the structured semantic information of entities but also leverages visual information such as the appearance and identification of entity-associated images, effectively mitigating entity ambiguity caused by insufficient textual and graphical information and significantly improving the accuracy of multimodal named entity recognition results.

[0021] 2. Unlike existing technologies where document retrieval easily introduces redundant information and makes it difficult to guarantee the precise relevance of knowledge to the target entity, and knowledge generation based on large language models faces high computational overhead, this invention uses candidate entities as the core of accurate retrieval. It directly obtains knowledge subgraphs and associated visual resources that are strongly related to the entity from the knowledge graph, without the need for document retrieval or large language model generation. This avoids the introduction of redundant information, ensures the precise relevance of knowledge to the target entity, and reduces the computational overhead of knowledge acquisition. At the same time, it combines a gating mechanism to adaptively control the injection intensity of external knowledge, effectively suppressing potential noise and further improving the robustness and efficiency of entity recognition. Attached Figure Description

[0022] Figure 1 This is a diagram illustrating the overall architecture of the knowledge graph-enhanced multimodal named entity recognition method of this invention. Figure 2 This is a schematic diagram of a cross-modal interaction module. Detailed Implementation

[0023] This embodiment provides a multimodal named entity recognition method based on knowledge graph enhancement, such as... Figure 1 As shown, it includes the following steps: Step 1: Input the text and image into the feature extraction module to obtain the initial text features. Compared with initial visual features .

[0024] Step one includes: Given the input text The initial text features are obtained by encoding using the BERT encoder: in, Indicates the first The characteristic representation of a token This represents the output dimension of the BERT encoder.

[0025] The input image is processed using a pre-trained ResNet-152 model. By performing feature extraction and obtaining the features from the last convolutional layer as a visual representation, we can obtain... Features of each spatial region: in, Indicates the first Characteristics of each region This represents the output dimension of the ResNet-152 convolutional layer.

[0026] Step 2: Input the text into the candidate entity recognition module, identify the candidate entities in the text, and obtain the candidate entity set. .

[0027] Step two includes: Input text The RoBERTa-large encoder is used to obtain the representation for candidate entity detection: in, Indicates the first The characteristic representation of a token This represents the feature dimension of the candidate entity detection module. A linear classification layer is introduced to predict the BIO label for each token, following the formula below: in, and These represent the weight matrix and bias term of the linear classification layer, respectively. Indicates the first The predicted probability distribution of each token across all tag categories This represents the number of label categories.

[0028] During training, the cross-entropy loss function is used to optimize the model, following the formula below: in, Indicates the first Each token in the category The predicted probability, This corresponds to the real label (one-hot) format.

[0029] The maximum value of the predicted probability distribution for each token is taken to obtain the predicted BIO tag for the corresponding token, and then the BIO tag sequence corresponding to the input text is obtained. ,in For the first The predicted label corresponds to each token. This invention uses the BIO annotation method to annotate text entities, with a label structure of "location marker - entity type". The location markers include B (Begin, entity start word), I (Inside, entity middle or end word), and O (OutSide, non-entity word). Entity types include PER (person), LOC (location), ORG (organizational structure), and MISC (other). Combining the location markers and entity types, the tag set is {B-PER, I-PER, I-LOC, B-LOC, B-ORG, I-ORG, B-MISC, I-MISC, O}, totaling... =9 predicted labels .

[0030] Based on the BIO annotation rules, entity merging is performed on the predicted tag sequences. Token sequences corresponding to consecutive "B-entity type + I-entity type" tags of the same type are merged into a complete entity fragment, with each fragment corresponding to a candidate entity. Tokens corresponding to non-entity tags O are not included in the candidate entity set. The final candidate entity set is as follows: in, Indicates the first One candidate entity.

[0031] Step 3: Using candidate entities as query objects, extract local knowledge subgraphs related to each candidate entity from the knowledge graph, and convert the subgraphs into triples to construct a triple knowledge set. Simultaneously, external image data associated with candidate entities are acquired to construct an entity-associated image set. .

[0032] Step three includes: In obtaining the candidate entity set Then, for each candidate entity Using these as search keywords, the system retrieves entities from the Wikidata knowledge graph. Mapped to corresponding knowledge graph entity nodes For each node ,by Using the central node as an example, obtain its first-order neighbor nodes and their corresponding relationships, thereby obtaining a local knowledge subgraph related to each candidate entity.

[0033] To facilitate subsequent model processing, the knowledge subgraphs are uniformly converted into triples and integrated into a triple knowledge set for the current sample, represented as: in, Indicates the head entity. Indicates the relation type, Indicates the tail entity. The total number of triplets.

[0034] Furthermore, the Wikidata knowledge graph contains rich multimodal information, such as image resources related to entities. Attributes with the relation "image" or "logo image" are used to represent representative images of the entity. For each candidate entity... The corresponding knowledge graph entity is If the entity contains an `image` or `logo image` attribute, the URL (Uniform Resource Locator) of the corresponding image is extracted, and a network request is initiated based on the URL to download the image data from the corresponding network resource server. Furthermore, the images corresponding to all entities in the sample are merged to obtain a sample-level set of entity-associated images. in, This represents the total number of images. This set contains external visual information related to candidate entities in the current sample, used to supplement the entity's background semantic information.

[0035] Step 4: Set up triplet knowledge Image set associated with entities The input knowledge feature extraction module performs encoding processing to obtain triplet knowledge feature representation sets. Set of image feature representations associated with entities .

[0036] Step four includes: First, structured triple information is encoded using a triple textual encoding method. For sets of triples... The first in triplet Convert to a unified text template format: will sequence The input is encoded using the BERT encoder to obtain the context representation: Because BERT introduces a special marker "[CLS]" at the beginning of the sequence, the corresponding vector is typically used to represent the global semantic information of the entire sequence. Therefore, the vector corresponding to the "[CLS]" position is used as the overall semantic representation of the triple: in, Indicates the first Semantic representation of triples. For a set of triples... All triples in the set can be used to obtain the corresponding knowledge representation set: in, This indicates the number of triples.

[0037] Secondly, for entity association image information, the same ResNet-152 visual encoder as the original input image is used to extract the global feature representation of the image. For any image... Its characteristics are represented as follows: in, The global feature vector is obtained by global average pooling from the last layer feature map. For all entity association images in the current sample, the corresponding set of entity association image feature representations can be obtained. : in, This represents the total number of images.

[0038] Step 5: Enhance text representation using triple knowledge Simultaneously, it integrates the features of entity-related images with the original visual features. The enhanced text features are obtained. and visual features .

[0039] Step five includes: First, to incorporate structured semantic information from the knowledge graph, a cross-attention mechanism is employed to enhance the text representation. Specifically, based on the initial text features... As a query, it represents a set of knowledge. We model the semantic relationships between text and knowledge, using them as keys and values.

[0040] The cross-attention calculation process is as follows: in, , , Learnable parameter matrices are used to generate queries, key and value representations, respectively; This indicates that contextual features are enhanced by triple knowledge.

[0041] To control the intensity of external knowledge injection, a gating mechanism is introduced to adaptively adjust the augmentation information: in, This represents a concatenation operation along the feature dimension; Represents the sigmoid function; Represents the ReLU activation function; This indicates the gating weight.

[0042] Finally, the enhanced text representation is obtained as follows: in, This indicates element-wise multiplication.

[0043] Secondly, to integrate semantic information from entity-related image features, a cross-attention mechanism is also used to enhance the visual features. (Based on the original visual features...) As a query, the set of image features associated with entities. As keys and values, this establishes a connection between visual and external image information: in, , , A learnable parameter matrix, used to generate queries, key and value representations respectively; This represents visual context features that incorporate external image information.

[0044] Similarly, a gating mechanism is introduced to control the fusion strength of external image information: in, This represents a concatenation operation along the feature dimension; Represents the sigmoid function; Represents the ReLU activation function; This indicates the gating weight.

[0045] Ultimately, the enhanced visual representation is obtained as follows: in, This indicates element-wise multiplication.

[0046] Step Six: Enhance the text features With enhanced visual features The input cross-modal interaction module performs cross-modal feature fusion to obtain a text representation that incorporates visual information. .

[0047] Step six includes: Since the enhanced visual representation and the text representation are inconsistent in feature dimensions, the visual features are first projected onto a feature space that is unified with the text through a linear mapping. Specifically, the initial input of the cross-modal interaction module is defined as: in, A learnable linear mapping matrix used to transform visual features from their original dimensions. Projected onto feature dimensions consistent with the text .

[0048] Based on this, such as Figure 2 As shown, a cross-modal interaction module based on Transformer is constructed. By combining intra-modal modeling and inter-modal interaction, a layer-by-layer fusion of text and visual information is achieved. This module consists of... It consists of a stacked structure, with each layer containing an Intra-Modal Transformer Block (IMT) and a Cross-Modal Transformer Block (CMT).

[0049] In the Within the layer, the context information within each mode is first modeled using intramodal Transformer blocks. The calculation process is as follows: in, This represents an intramodal Transformer block, composed of a multi-head self-attention mechanism, a feedforward neural network, residual connections, and a layer normalization cascade, used to model the contextual semantic information within a single modality. Through this mechanism, text tokens can model semantic dependencies between other tokens, while visual region features can capture semantic associations between different regions.

[0050] After obtaining the intramodal representation, semantic interaction between different modalities is further achieved through cross-modal Transformer blocks. Specifically, this is represented by text. Visual representation as a query Using these as keys and values, we obtain a text representation that incorporates visual information: Meanwhile, to enhance the visual representation's ability to perceive textual semantics, a symmetrical structure is adopted for visual representation. As a query, the text represents Used as keys and values ​​for cross-modal interaction: in, This represents a cross-modal Transformer block, whose structure is consistent with IMT, but the attention computation adopts a cross-modal form, that is, the query comes from one modality, while the key and value come from another modality. Through this mechanism, text representation can focus on visual regions that are semantically related to it, and visual representation can also fuse corresponding text semantic information, thereby achieving bidirectional cross-modal semantic alignment and information interaction.

[0051] By repeating In this multi-interaction process, text and visual features can engage in multiple rounds of semantic interaction, gradually enhancing cross-modal semantic association capabilities, and ultimately obtaining a fused multimodal representation. and .

[0052] Step 7: Integrate visual information into the text representation Features of the original text The two parts of information are merged and weighted using a gating mechanism to obtain the final text representation. ; Represent the final text Input a conditional random field decoder and output named entity recognition results.

[0053] Step seven includes: In order to fully utilize cross-modal enhancement information while preserving the original semantic information of the text, a gating fusion mechanism is used to adaptively fuse the original text representation and the cross-modal enhanced text representation.

[0054] Specifically, the initial text is represented as The text output by the cross-modal interaction module is represented as First, the two are concatenated along the feature dimension, and then a gated vector is generated through a linear transformation. : in, Indicates the gating weight, This represents the sigmoid function. and These are the learnable parameter matrix and the bias term, respectively. Subsequently, the two feature sets are weighted and fused using a gating vector to obtain the final text representation. : in, This represents element-wise multiplication. (Gated vector) Used to adaptively adjust the contribution ratio of cross-modal augmentation information to original text information: when When the size is large, the model tends to utilize the semantic information enhanced across modalities; when When the text is smaller, more of the original text representation is preserved, thus achieving a balance between information enhancement and semantic preservation.

[0055] Ultimately, the merged text representation The input is fed into the CRF (Conditional Random Field) decoder to predict the corresponding label sequence. For the label sequence... Its conditional probability is defined as: in, Represents the potential function. This represents the set of all possible label sequences.

[0056] During the training phase, negative log-likelihood is used as the optimization objective. This is achieved within a training array of size [value missing]. Within a batch, the loss of the multimodal named entity recognition task is defined as: During the inference phase, the label sequence with the highest conditional probability is selected as the output label sequence, i.e.: In another aspect, the present invention also provides a knowledge graph-enhanced multimodal named entity recognition system for use in the aforementioned multimodal named entity recognition method, comprising a feature extraction and candidate entity recognition unit, a feature representation unit, a feature enhancement unit, a visual fusion unit, and a named entity recognition output unit: The feature extraction and candidate entity recognition unit is used to input text and image into the feature extraction module to obtain initial text features respectively. The initial visual features are used, and the text is input into the candidate entity recognition module to identify candidate entities in the text and obtain a candidate entity set.

[0057] The feature representation unit is used to construct a triple knowledge set and an entity association image set using candidate entities in the candidate entity set as query objects, and to encode the triple knowledge set and the entity association image set to obtain the triple knowledge feature representation set and the entity association image feature representation set, respectively.

[0058] The feature enhancement unit is used to enhance the initial text representation using the triplet knowledge set, and at the same time fuse the feature representation set of entity association image with the initial visual features to obtain enhanced text features and visual features. The visual fusion unit is used to input the enhanced text features and enhanced visual features into the cross-modal interaction module for cross-modal feature fusion to obtain a text representation with fused visual information.

[0059] The named entity recognition output unit is used to combine the text representation fused with visual information with the initial text features. The data is fused to obtain the final text representation, which is then input into the conditional random field decoder to output the named entity recognition result.

[0060] This experiment was conducted on an Ubuntu 18.04.5 operating system environment, with an NVIDIA GeForce RTX4080 SUPER GPU, CUDA version 11.8, and PyTorch 2.2.2 as the deep learning framework.

[0061] In the text encoding stage, BERT-base-uncased is used as the text encoder to encode the input text, obtaining an initial token-level representation, and the output dimension. The value is 768. In the visual feature extraction stage, ResNet-152 is used as the visual extraction network. For the original image, ResNet-152 is used to extract the visual feature representation of each region, and the feature dimension of each region is... The value is 2048. Simultaneously, for the auxiliary images obtained during the knowledge retrieval stage, ResNet-152 is used to extract their global features, with the feature dimensions being [missing information]. The value is 2048. The cross-modal interaction module consists of... The experiment is constructed using a stacked Transformer structure, with each layer containing intra-modal modeling and cross-modal interaction. =3, with 8 multi-head attention heads. During training, the AdamW optimizer was used, with 11 training epochs, a batch size of 8, and a learning rate of 5e-5.

[0062] Furthermore, the candidate entity generation module uses RoBERTa-large as the encoder, with a feature dimension of [missing information]. 1024. Maximum text length is set to 128. The AdamW optimizer is used during training, with 3 training epochs and a batch size of 16. The learning rate is set to 2e-5, and the dump rate is set to 0.1. After the candidate entity generation model is trained, candidate entity prediction is performed on the text in the training, validation, and test sets, respectively, and candidate entity sets for each dataset are constructed accordingly for subsequent knowledge retrieval.

[0063] Compared with existing methods, the proposed knowledge graph-enhanced multimodal named entity recognition method achieves significant performance improvements. On the Twitter-2015 and Twitter-2017 datasets, the F1 score of the proposed KGA model is improved by 0.16% and 1.09%, respectively. Specific comparison results are shown in Tables 1 and 2.

[0064] Table 1. Comparative experiment results (%) on the Twitter-2015 dataset

[0065] Table 2. Comparative experiment results (%) on the Twitter-2017 dataset

Claims

1. A multimodal named entity recognition method based on knowledge graph enhancement, characterized in that, Includes the following steps: Step 1: Input the text and image into the feature extraction module to obtain the initial text features. Compared with initial visual features The text is then input into the candidate entity recognition module, which identifies the candidate entities in the text and obtains a candidate entity set. ; Step 2: Using the candidate entity set Using candidate entities as query objects, construct a triple knowledge set. Image sets associated with entities ; Step 3: Set up triplet knowledge Image set associated with entities Encoding processing is performed to obtain sets of triplet knowledge feature representations. Set of image feature representations associated with entities ; Step 4: Enhance the initial text representation using triple knowledge sets Simultaneously, it integrates the feature representation set of entity-related images with the initial visual features. The enhanced text features are obtained. and visual features ; Step 5: Enhance the text features With enhanced visual features The input cross-modal interaction module performs cross-modal feature fusion to obtain a text representation that incorporates visual information. ; Step Six: Integrate visual information into the text representation Compared with initial text features The data is then fused to obtain the final text representation. Input a conditional random field decoder and output named entity recognition results.

2. The multimodal named entity recognition method based on knowledge graph enhancement according to claim 1, characterized in that, In step one, the initial text features are obtained respectively. Compared with initial visual features The specific implementation process is as follows: Given the input text The initial text features are obtained by encoding using the BERT encoder. ,in, For the first The characteristic representation of a token This refers to the output dimension of the BERT encoder. Using a pre-trained convolutional model on the input image Feature extraction is performed to obtain the features from the last convolutional layer as a visual representation, resulting in... Features of each spatial region ,in, Indicates the first Characteristics of each region is the output dimension of the convolutional layer.

3. The multimodal named entity recognition method based on knowledge graph enhancement according to claim 2, characterized in that, In step one, the candidate entity set is obtained. The specific implementation process is as follows: Input text The RoBERTa-large encoder is used to obtain the representation for candidate entity detection: in, Indicates the first The characteristic representation of a token To define the feature dimensions of the candidate entity detection module, a linear classification layer is introduced for each... Perform BIO tag prediction to obtain the first Predicted probability distribution of each token across all tag categories , The number of label categories; during training, the cross-entropy loss function is used for optimization; For each token, the maximum value of the predicted probability distribution is taken to obtain the predicted BIO tag for the corresponding token, and the BIO tag sequence corresponding to the input text is obtained. ,in For the first The predicted tags for each token are annotated using the BIO annotation method. The tag structure is location marker-entity type. The location marker includes the entity start word (B), the entity middle or end word (I), and non-entity words (O). The entity type includes Person (PER), Location (LOC), Organizational Structure (ORG), and other MISCs. Combining the location marker and entity type, the tag set is {B-PER, I-PER, I-LOC, B-LOC, B-ORG, I-ORG, B-MISC, I-MISC, O}, totaling [number missing]. =9 predicted labels ; Based on the BIO annotation rules, entity merging is performed on the predicted tag sequences. Token sequences corresponding to consecutive, same-type B-entity-type + I-entity-type tags are merged into a single complete entity fragment, with each fragment corresponding to a candidate entity. Tokens corresponding to non-entity tags O are not included in the candidate entities. The final result is a set of candidate entities. ,in, Indicates the first One candidate entity.

4. The multimodal named entity recognition method based on knowledge graph enhancement according to claim 3, characterized in that, The specific implementation process of step two is as follows: In obtaining the candidate entity set Then, for each candidate entity Using these as search keywords, a search is conducted within the Wikidata knowledge graph, and... Mapped to corresponding knowledge graph entity nodes For each node ,by Using the central node, obtain its first-order neighbor nodes and corresponding relationships to obtain a local knowledge subgraph related to each candidate entity; The knowledge subgraphs are uniformly converted into triples, and integrated into a triple knowledge set for the current sample, represented as follows: ,in, Indicates the head entity. Indicates the relation type, Indicates the tail entity. The total number of triples; The Wikidata knowledge graph also contains multimodal information, where the relation is an image or a logo, and the attribute of the image is used to represent a representative image of the entity; for each candidate entity The corresponding knowledge graph entity is If the entity contains an `image` or `logo image` attribute, the corresponding Uniform Resource Locator (URL) of the image is extracted. A network request is initiated based on the URL to download the image data from the corresponding network resource server. The images corresponding to all entities in the sample are then merged to obtain a sample-level set of entity-associated images. ,in, This represents the total number of images.

5. The multimodal named entity recognition method based on knowledge graph enhancement according to claim 4, characterized in that, The specific implementation process of step three is as follows: First, for structured triple information, it is encoded using a triple textual approach. For a set of triples... The first in triplet Convert to a unified text template format ; will sequence The input is encoded using a BERT encoder to obtain the context representation. BERT introduces a special marker [CLS] at the beginning of the sequence. The corresponding vector is used to represent the global semantic information of the entire sequence. The vector at the [CLS] position is used as the overall semantic representation of the triple. For sets of triples All triples in the set yield the corresponding knowledge representation set. ,in, Indicates the number of triples; Secondly, for entity-related image information, the same convolutional model as the original input image is used to extract the global feature representation of the image. For any image... Its characteristics are represented as The feature representation set of the entity association images is obtained by global average pooling from the last layer feature map. For all entity association images in the current sample, the corresponding entity association image feature representation set is obtained. ,in, This represents the total number of images.

6. The multimodal named entity recognition method based on knowledge graph enhancement according to claim 5, characterized in that, The specific implementation process of step four is as follows: First, a cross-attention mechanism is used to enhance the text representation, based on the initial text features. As a query, it represents a set of knowledge. Using these as keys and values, we model the semantic relationships between text and knowledge, compute cross-attention, and obtain contextual features that incorporate triple-enhanced knowledge. ; Introducing gating mechanism to Adaptive adjustment: in, This represents a concatenation operation along the feature dimension; Represents the sigmoid function; Represents the ReLU activation function; Indicates the gating weight; Ultimately, the enhanced text representation is obtained as follows: ,in This represents element-wise multiplication; Secondly, the same cross-attention mechanism is used to enhance visual features, using visual features As a query, the set of image features associated with entities. As keys and values, a connection is established between visual and external image information, cross-attention is calculated, and visual context features that fuse external image information are obtained. ; Introducing a gating mechanism to control the fusion strength of external image information: in, This represents a concatenation operation along the feature dimension. Indicates the gating weight; Ultimately, the enhanced visual representation is obtained as .

7. The multimodal named entity recognition method based on knowledge graph enhancement according to claim 6, characterized in that, The specific implementation process of step five is as follows: First, visual features are projected onto a feature space unified with the text using linear mapping. The initial input of the cross-modal interaction module is defined as follows: ， in, It is a learnable linear mapping matrix; A Transformer-based cross-modal interaction module is constructed, which achieves layer-by-layer fusion of text and visual information by combining intra-modal modeling and inter-modal interaction. This cross-modal interaction module consists of... It consists of a layered stacked structure, with each layer containing an intra-modal Transformer block (IMT) and a cross-modal Transformer block (CMT). In the Within the layer, the context information within each mode is first modeled using intramodal Transformer blocks. The calculation process is as follows: ， in, This represents an intramodal Transformer block, which consists of a multi-head self-attention mechanism, a feedforward neural network, residual connections, and a layer normalization cascade. After obtaining the intramodal representation, semantic interaction between different modalities is achieved through cross-modal Transformer blocks. As a query Using the key and value as the key and value, we obtain a text representation that incorporates visual information: At the same time, a symmetrical structure is adopted to As a query Used as keys and values ​​for cross-modal interaction: in, This represents a cross-modal Transformer block, with a structure consistent with IMT; By repeating In this multi-interaction process, text and visual features can engage in multiple rounds of semantic interaction, gradually enhancing cross-modal semantic association capabilities, and ultimately obtaining a fused multimodal representation. and .

8. The multimodal named entity recognition method based on knowledge graph enhancement according to claim 7, characterized in that, The specific implementation process of step six is ​​as follows: The initial text features are represented as The text output by the cross-modal interaction module is represented as First, the two are concatenated along the feature dimension, and then a gated vector is generated through a linear transformation. : in, Indicates the gating weight, This represents the sigmoid function. and These are the learnable parameter matrix and the bias term, respectively. Then, gated weights are used... The two feature sets are weighted and fused to obtain the final text representation. : The merged text representation The input is fed into the CRF decoder to predict the corresponding label sequence. For the label sequence... The conditional probability is ; During the inference phase, the label sequence with the highest conditional probability is selected as the output label sequence. Complete named entity recognition.

9. A knowledge graph-enhanced multimodal named entity recognition system, used to implement the multimodal named entity recognition method according to any one of claims 1 to 8, characterized in that, Includes the following units: The feature extraction and candidate entity recognition unit is used to input text and images into the feature extraction module to obtain initial text features. The initial visual features are used, and the text is input into the candidate entity recognition module to identify candidate entities in the text and obtain a candidate entity set. The feature representation unit is used to construct a triple knowledge set and an entity association image set with candidate entities in the candidate entity set as the query object, and encodes the triple knowledge set and the entity association image set to obtain the triple knowledge feature representation set and the entity association image feature representation set, respectively. The feature enhancement unit is used to enhance the initial text representation by utilizing the triplet knowledge set, and at the same time, it integrates the feature representation set of entity association images with the initial visual features to obtain enhanced text features and visual features. The visual fusion unit is used to input the enhanced text features and enhanced visual features into the cross-modal interaction module for cross-modal feature fusion to obtain a text representation with fused visual information; Named entity recognition output unit, used to combine text representations fused with visual information with initial text features. The data is fused to obtain the final text representation, which is then input into the conditional random field decoder to output the named entity recognition result.

Images