Multi-modal named entity recognition method based on multi-dimensional similarity dynamic fusion

Abstract

The invention discloses a multi-modal named entity recognition method based on multi-dimensional similarity dynamic fusion, and relates to the technical field of natural language processing and computer vision. According to the method, multi-modal named entity recognition is achieved through a dynamic semantic alignment network (DSANet), and a feature extraction module, a cross-modal semantic interactive attention module, a dynamic similar weighted fusion module and a local global sequence modeling module are mainly included. According to the method, more complete modal semantic alignment is realized through a double-path cross-modal attention mechanism, the image-text correlation judgment accuracy is improved through a three-dimensional similarity evaluation system, and the context dependency relationship modeling capability is enhanced through a local-global feature fusion architecture; the problems that in the prior art, image-text semantic alignment is insufficient, similarity evaluation is single, and a fusion weight adjustment mechanism is limited are effectively solved.

Description

Technical Field

[0001] This invention relates to the field of [the field of research], and more particularly to a multimodal named entity recognition method based on dynamic fusion of multidimensional similarity. Background Technology

[0002] Multimodal Named Entity Recognition (MNER) is an important research direction in the field of artificial intelligence. It aims to overcome the limitations of traditional unimodal MNER tasks in short texts and ambiguous contexts by fusing multimodal data such as text and images. MNER achieves cross-modal entity semantic disambiguation and joint annotation, and is a core technical component in constructing multimodal knowledge graphs. With the rapid development of social media and news platforms, mixed text and image content is becoming increasingly abundant. However, semantic mismatches, information redundancy, or conflicts may exist between text and images, posing significant challenges to MNER. Especially in social media scenarios, images may be completely unrelated to the text content or contain only partial relevant information. How to accurately identify entities in such situations remains a pressing technical challenge.

[0003] Existing technologies mainly include the following solutions:

[0004] Option 1: MAF Framework (Bo X et al., 2022)

[0005] It adopts a general matching and alignment framework to fully reflect the complex semantic relationships between modalities.

[0006] Option 2: FRCL-MNER framework (Tianwei Y et al., 2025)

[0007] The image-text relevance assessment scheme based on contrastive learning adopts a fine-grained ranking contrastive learning framework.

[0008] Option 3: CRISP framework (Haitao L et al., 2025)

[0009] The fusion scheme is based on adaptive modal weighting and uses a surprisingly popular algorithm to dynamically adjust the modal weights.

[0010] Option 4: Dual Similarity Enhancement Fusion Scheme (Chunmao Jiang et al., 2026)

[0011] This is based on a hybrid orthogonal fusion network with dual similarity enhancement, combining word-level and modality-level similarity.

[0012] Analysis of the shortcomings of existing technologies:

[0013] Insufficient semantic alignment and one-sided modal association capture

[0014] Most existing technical solutions employ a one-way attention mechanism, which can only establish modality mapping from a single direction and cannot fully capture the bidirectional semantic relationship between text and images. For example, MAF uses a one-way attention mechanism of "text → image" and cannot use key regions in the image to correct the semantic understanding of ambiguous entities in the text.

[0015] Reason: The bidirectional semantic interaction requirements of multimodal data were not considered, and the unidirectional information flow limited the integrity of modal association.

[0016] The limitations and incompleteness of similarity assessment

[0017] Existing technical solutions suffer from a significant limitation in assessing the relevance between text and images. MAF uses only cosine similarity to calculate the similarity score between text and image modalities; FRCL relies solely on a contrastive learning mechanism to learn a single-dimensional similarity representation; CRISP uses a surprisingly popular algorithm to calculate a single importance weight; and while dual-similarity enhancement methods consider two dimensions, they still lack a comprehensive approach.

[0018] Reasons: Failure to comprehensively assess the relevance of images and text from multiple dimensions such as semantics, structure, and feature differences; lack of understanding of the semantic depth of images and text; the similarity calculation method is relatively simple and cannot adapt to the complexity of image-text relationships in different scenarios.

[0019] Limitations of the fusion weight adjustment mechanism

[0020] Existing fusion weight adjustment mechanisms have significant limitations. While CRISP employs an adaptive modality weighting strategy, this strategy is primarily based on surprisingly popular algorithms and lacks consideration of the semantic relevance between text and images. Other solutions rely solely on a single similarity score when adjusting weights, lacking multi-dimensional considerations.

[0021] Therefore, this invention proposes a multimodal named entity recognition method based on dynamic fusion of multidimensional similarity. Summary of the Invention

[0022] The purpose of this invention is to address the shortcomings of existing technologies by proposing a multimodal named entity recognition method based on dynamic fusion of multidimensional similarity.

[0023] To achieve the above objectives, the present invention adopts the following technical solution:

[0024] A multimodal named entity recognition method based on dynamic fusion of multidimensional similarity includes the following steps:

[0025] S1: Feature extraction. The feature extraction module extracts features from the input text and image, using a pre-trained language model and a convolutional neural network to extract initial features, respectively.

[0026] S2: Cross-modal semantic interaction. The Cross-modal Semantic Interaction Attention Module (CMSIA) adopts a dual-path cross-modal attention mechanism to achieve bidirectional semantic interaction and enhancement between text and image features.

[0027] S3: Dynamic similarity weighted fusion. The dynamic similarity weighted fusion module DSWF dynamically adjusts the fusion weights of text and images based on multidimensional similarity, and dynamically adjusts the multimodal fusion weights to ensure that the influence of image features can be automatically reduced when the text and images are unrelated.

[0028] S4: Local-Global Sequence Modeling. The Local-Global Sequence Modeling module (LGSM) applies local attention and global token aggregation mechanisms to the fused feature sequences to enhance sequence modeling capabilities and capture the contextual dependencies of entities.

[0029] S5: Entity labeling. The entity labeling layer finally uses a Conditional Random Field (CRF) to perform sequence labeling on the fused features and outputs the entity prediction results.

[0030] Preferably, in step S1, the method for text feature extraction is as follows:

[0031] S11a: The text input undergoes feature extraction using a BERT (Bidirectional Encoder Representations from Transformers) pre-trained model, whose model is: H text =BERT(x text A mask ,T type ), where x text ∈R B×L A represents an input token sequence of batch size B and sequence length L. mask ∈R B×L T represents the attention mask. type ∈R B×L Indicates token type ID, H text ∈R B×L×768 Represents the text features output by BERT;

[0032] S12a: Text features are mapped to a unified dimensional space through a projection layer. The model is as follows: in, and These are the weights and biases of the projection layer, where D = 768 is the hidden dimension, and T ∈ R. B×L×DThese are the text features after projection.

[0033] Preferably, in step S1, the image feature extraction method is as follows:

[0034] S11b: The image input undergoes feature extraction via ResNet50 (Residual Network), and its model is: F img =ResNet50(X img ), where x img ∈R B×3×H×W This represents an input image (3 channels, height H, width W) with a batch size of B, F img ∈R B×2048×h×w This represents the feature map output by ResNet50;

[0035] S12b: The feature map is reshaped into a sequence and projected. The processing model is as follows: in, It is the reshaped feature sequence. and These are the weights and biases of the projection layer, V∈R B ×(h×w)×D These are the features of the projected image.

[0036] Preferably, step S1 further includes a hash encoding step, which maps high-dimensional features to a low-dimensional binary space through hash encoding, enhances the semantic distinguishability of features, and provides a basis for hash similarity calculation of the DSWF module, thereby improving the accuracy and efficiency of similarity calculation.

[0037] Preferably, the implementation steps of hash encoding in step S1 are as follows:

[0038] The hash encoding process is as follows: H hash =tanh(φ(X,W1,b1,w2,b2)), where φ represents two layers of MLP operations;

[0039] The specific calculation process is as follows: φ(X,W1,b1,W2,b2)=R e LU(XW1+b1)W2+b2, where X∈R B×D Represents the input features (which can be text or global image features), W1∈ and These are the weights and biases of the first layer MLP. and It refers to the weights and biases of the second-layer MLP, H. d It is the hash code dimension (128). It is a continuous hash code representation;

[0040] During the inference phase, the consecutive hash codes are binarized, the process being: H binary =sign(H hash ).

[0041] Preferably, in step S1, the hash encoding supports both global and local granularities, which are as follows:

[0042] Global hash encoding: Encodes features after average pooling; its model is as follows:

[0043]

[0044] Local hashing encoding: Selects key tokens / regional features for encoding, and its model is: W attn =softmax(TT) T ), T key =T[batchindices,TopK]、 Among them, W attn For the attention weight matrix, TopK i T represents the indexes of the top K key tokens selected for the i-th sample. key ∈R B×K×D Indicates key token characteristics, It is a local hash code;

[0045] Ultimately, the global and local hash codes are combined to enhance the feature representation, as follows:

[0046] Preferably, step S2 includes the following steps:

[0047] S21: Multi-head attention preprocessing, performing Q, K, V linear transformations on text and image features respectively; the transformed feature dimensions are [batch size, sequence length, hidden dimension]; the features are reshaped into a multi-head format [batch size, number of heads, sequence length, head dimension], and the computation model is as follows: in, It is a linear transformation matrix. It is the corresponding bias term;

[0048] S22: Feature reshaping into a multi-head format, reshaping Q, K, and V into a multi-head format, the calculation model is as follows: Where h is the number of attention heads, It is the dimension of each head. It is the reshaped multi-head format, K text V text Q img ,Kimg V img The processing method is the same;

[0049] S23: Self-attention calculation, calculating self-attention within text and images, the calculation model is as follows: in, and These are the self-attention matrices for text and image, respectively, and S is the length of the image feature sequence;

[0050] S24: Cross-modal attention calculation, which calculates cross-modal attention between text and image and between image and text. The calculation model is as follows: Among them, A text→img ∈R B×h×L×S A represents the attention of a text query to the image key. img→text ∈R B×h×S×L This indicates the attention given to the text key by the image query;

[0051] S25: Apply masks and normalization to text self-attention and image-to-text cross-modal attention, then perform softmax normalization on all attention scores. The calculation models are as follows: Where M∈R B×1×1×L It is the expanded text mask;

[0052] S26: Calculate the attention output, using normalized attention weights to calculate the outputs of self-attention and cross-modal attention. The calculation models are as follows:

[0053] S27: Reshape back to the original dimension and fuse. Reshape the attention output back to the original dimension and fuse the results of self-attention and cross-modal attention proportionally. The fusion model is as follows: Among them, T enhanced ∈R B×L×D and V enhanced ∈R B×S×D These are the enhanced text and image features, respectively.

[0054] S28: Final Projection and Regularization. The fused features are then subjected to final projection and dropout regularization. The models are as follows: in,

[0055] R D×D and These are the weights and biases of the text and image output projections, respectively, T. final ∈R B×L×D and V final ∈R B×S×D These are the enhanced features of the final output.

[0056] Preferably, step S3 includes the following steps:

[0057] S31: Feature projection and global representation calculation. The enhanced text and image features from the CMSIA module are projected, and the global representation vector is calculated. The calculation process model is as follows: in, and These are the weights and biases of the DSWF projections of the text and image, respectively, T proj V proj ∈R B×L / S×D It is the projected feature, T global V global ∈R B×D It is a global representation;

[0058] S32: Three-dimensional similarity calculation:

[0059] The dot product of normalized text and image global representations is calculated using cosine similarity:

[0060]

[0061] The differences between the global representations of text and images are non-linearly transformed using MLO similarity:

[0062] sim mlp =MLP(|T global -V global |);

[0063] The Hamming distance complement of text and image hash codes is calculated using hash similarity.

[0064]

[0065] in, and It is the normalized global representation. MLP is a two-layer perceptron (specifically, the structure is Linear(D,D / 4)→ReLU→Linear(D / 4,1)→Sigmoid). H d It is the hash code dimension;

[0066] S33: Linear similarity fusion, which fuses the three similarities through a linear layer to obtain a dynamic fusion weight α. The calculation model is: α = σ(W linear [sim cos;sim mlp ;sim hash ]+b linear ), where W linear ∈R 3×1 and b linear ∈R represents the weights and biases of the linear fusion, σ is the sigmoid activation function, and α∈R B×1 It is a dynamic fusion weight, with a value range of [0,1], used to adaptively adjust the fusion ratio of text and image features;

[0067] S34: Cross-modal enhancement and fusion:

[0068] Text features are enhanced through self-transformation, and the model is as follows: in, and It refers to the weights and biases of the text enhancement layer, T. enhanced ∈R B×L×D These are enhanced text features;

[0069] Image features are transformed through a weight matrix, and the model is as follows:

[0070] The interaction between computational image features and text features is modeled as follows: C cross =T final ·(V weighted ) T ;

[0071] in, It is the expanded global image feature, W fusion ∈R D×D It is the fusion weight matrix, V weighted ∈R B×1×D It is the image feature after weighted transformation, C cross ∈R B×L×1 It is a cross-modal interaction result;

[0072] S35: Final fusion output, using dynamic weights α to fuse textual features and cross-modal interaction results, with the model being: O = T enhanced +α·C cross , where α∈R B×1×1 These are the extended dynamic fusion weights, O∈R B×L×D It is the final fusion output.

[0073] Preferably, step S4 includes the following steps:

[0074] S41: Feature preprocessing, saving residual connections of input features, and performing local and global normalization on the input features. The processing model is: R = O.clone(), X local =LayerNorm local (O), X global =LayerNorm global (O), where R∈R B×L×D It is a residual connection, X local ,X global ∈R B×L×D These are the features after local and global normalization, respectively;

[0075] S42: Local attention mechanism, which uses multi-head self-attention to process normalized features and capture local contextual relationships in the sequence. Its model is as follows: in, These are features resulting from local attention processing;

[0076] S43: Global token aggregation, which performs query transformation on globally normalized features and calculates the similarity with global tokens. Its model is as follows: G = G init .unsqueeze(0).expand(B,-1,-1)、S=Q global ·G T A global =softmax(S),C global =A global ·G, where, and It is the weight and bias of the global query transformation, Q global ∈R B×L×D This is the result of the query transformation. It is the initial global token (N) t This refers to the number of tokens, usually 8. It is an expanded global token. It is a similarity score. It is attention weight, C globa l∈R B×L×D It is the global context;

[0077] S44: Local-Global Feature Fusion, which adds the local attention results to the global context, and processes them through a projection layer and dropout. The processing model is as follows: X proj =Dropout(X fused W proj +b proj ), X res =R+X proj, where X fused ∈R B×L×D It is a fusion feature, W proj ∈R D×D and b proj ∈R D These are the weights and biases of the projection layer, X proj ∈R B×L×D It is the projected feature, X res ∈R B×L×D These are characteristics resulting from residual joins;

[0078] S45: Feedforward Neural Network, which further processes features through two layers of feedforward network. Its model is: X norm =LayerNorm(X res ), X ffn =GELU(X norm W ffn1 +b ffn1 W ffn2 +b ffn2 O final =X res +Dropout(X ffn ), where X norm ∈R B×L×D It is the normalized feature, W ffn1 ∈R D×4D W ffn2 ∈R 4D×D and b ffn1 ∈R 4D ,b ffn2 ∈R D These are the weights and biases of the feedforward network, X ffn ∈R B×L×D It is the output of the feedforward network, O final ∈R B×L×D This is the final output of the LGSM module.

[0079] Preferably, step S5 includes the following steps:

[0080] S51: Feature transformation, applying dropout regularization to the features output by LGSM; mapping to the label space through a linear classifier;

[0081] S52: CRF sequence labeling; CRF loss is calculated during the training phase; the best label sequence is decoded using the Viterbi algorithm.

[0082] S53: Multi-task learning. During the training phase, in addition to the CRF loss, a consistency constraint loss is calculated to ensure consistency between text, images, and fused features; and a hash contrast loss is calculated to make the hash representations of text and images more similar. The total loss is the weighted sum of these three losses.

[0083] The beneficial effects of this invention are as follows:

[0084] More complete modal semantic alignment: By establishing bidirectional semantic correspondence through a dual-path cross-modal attention mechanism, the problem of incomplete semantic alignment in existing technologies is solved.

[0085] More accurate image-text relevance assessment: By using a three-dimensional similarity evaluation system to comprehensively assess the relevance of images and text from multiple dimensions, the accuracy of relevance assessment is improved.

[0086] Enhanced context dependency modeling capabilities: The local-global feature fusion architecture effectively enhances sequence modeling capabilities and improves the model's understanding of complex semantic scenes. Attached Figure Description

[0087] Figure 1 This is a schematic diagram of the overall architecture of a multimodal named entity recognition method based on dynamic fusion of multidimensional similarity proposed in this invention;

[0088] Figure 2 This is a detailed structural diagram of the feature extraction module of a multimodal named entity recognition method based on dynamic fusion of multidimensional similarity proposed in this invention.

[0089] Figure 3 This is a schematic diagram of the dual-path cross-modal attention mechanism of the CMSIA module in a multimodal named entity recognition method based on dynamic fusion of multidimensional similarity proposed in this invention.

[0090] Figure 4 This is a schematic diagram of the three-dimensional similarity evaluation and dynamic weight fusion mechanism of the DSWF module in a multimodal named entity recognition method based on dynamic fusion of multidimensional similarity proposed in this invention.

[0091] Figure 5 This is a schematic diagram of the local-global feature fusion architecture of the LGSM module of a multimodal named entity recognition method based on dynamic fusion of multidimensional similarity proposed in this invention. Detailed Implementation

[0092] The technical solution of the present invention will be further described in detail below with reference to specific embodiments.

[0093] Example 1:

[0094] 1 Feature Extraction Module

[0095] This invention first extracts features from the input text and image, using a pre-trained language model and a convolutional neural network to extract initial features, respectively. For example... Figure 2 As shown.

[0096] (1) Text feature extraction:

[0097] The text input is processed by a BERT (Bidirectional Encoder Representations from Transformers) pre-trained model for feature extraction, as shown in Equation (1):

[0098] H text =BERT(x text A mask ,T type (1)

[0099] Where, x text ∈R B×L A represents an input token sequence of batch size B and sequence length L. mask ∈R B×L T represents the attention mask. type ∈R B×L Indicates token type ID, H text ∈R B×L×768 This represents the text features output by BERT.

[0100] Text features are mapped to a unified dimensional space through a projection layer, and the mapping process is shown in Equation (2):

[0101]

[0102] in, and These are the weights and biases of the projection layer, respectively, where D is the hidden dimension (768), and T∈R. B×L×D These are the text features after projection.

[0103] (2) Image feature extraction:

[0104] The image input undergoes feature extraction via ResNet50 (Residual Network), and the processing procedure is shown in Equation (3):

[0105] F img =ResNet50(X img (3)

[0106] Where, x img ∈R B×3×H×W This represents an input image (3 channels, height H, width W) with a batch size of B, F img ∈R B ×2048×h×w This represents the feature map output by ResNet50.

[0107] The feature map is reshaped into a sequence and projected, as shown in formulas (4) and (5):

[0108]

[0109] in, It is the reshaped feature sequence. and These are the weights and biases of the projection layer, V∈R B×(h×w)×D These are the features of the projected image.

[0110] (3) Hash encoding:

[0111] This part is one of the key innovations of this invention. By mapping high-dimensional features to a low-dimensional binary space through hash encoding, the semantic distinguishability of the features is enhanced. At the same time, it provides a basis for hash similarity calculation of the DSWF module, improving the accuracy and efficiency of similarity calculation. The hash encoding process is shown in formula (6):

[0112] H h ash =tanh(φ(X,W1,b1,w2,b2)) (6)

[0113] Where φ represents two layers of MLP operations, and the specific calculation process is shown in formula (7):

[0114] φ(X,W1,b1,W2,b2)=R e LU(XW1+b1)W2+b2 (7)

[0115] Where, X∈R B×D This represents the input features (which can be text or global image features). and These are the weights and biases of the first layer MLP. and It refers to the weights and biases of the second-layer MLP, H. d It is the hash code dimension (128). It is a continuous hash code representation.

[0116] During the inference phase, the consecutive hash codes are binarized, as shown in formula (8):

[0117] H binary =sign(H h ash (8)

[0118] Hash encoding supports both global and local granularities:

[0119] Global hash encoding: Encodes the features after average pooling, as shown in formulas (9) and (10):

[0120]

[0121] Local hashing encoding: Select key tokens / regional features for encoding, as shown in formula (11) for text:

[0122] W attn =softmax(TT) T (11)

[0123]

[0124] T key =T[batch indices,TopK] (13)

[0125]

[0126] Among them, W attn For the attention weight matrix, TopK i T represents the indexes of the top K key tokens selected for the i-th sample. key ∈R B×K×D Indicates key token characteristics, It is a local hash code. A similar method is used for local hash encoding of images.

[0127] Finally, the global and local hash codes are combined to enhance the feature representation, as shown in Equations (15) and (16):

[0128]

[0129] 2. Cross-modal semantic interaction attention module (CMSIA)

[0130] The CMSIA module employs a dual-path cross-modal attention mechanism to achieve bidirectional semantic interaction and enhancement between text and image features. The specific module structure framework is as follows: Figure 3 As shown.

[0131] Specific implementation:

[0132] (1) Multi-head attention preprocessing:

[0133] Perform Q, K, and V linear transformations on the text and image features respectively; the transformed feature dimensions are [batch size, sequence length, hidden dimension]; reshape the features into a multi-head format [batch size, number of heads, sequence length, head dimension], and the calculation process is shown in formulas (17) to (22).

[0134]

[0135] in, It is a linear transformation matrix.

[0136] and, This is the corresponding bias term.

[0137] (2) Feature reshaping into a multi-head format:

[0138] Q, K, and V are reshaped into a multi-head format, and the calculation process is shown in formula (23).

[0139]

[0140] Where h is the number of attention heads, It is the dimension of each head. It is the reshaped multi-head format. K text V text Q ing ,K ing V ing The processing method is the same.

[0141] (3) Self-attention calculation:

[0142] Calculate the self-attention within the text and image, as shown in Equations (24) and (25).

[0143]

[0144] in, and These are the self-attention matrices for text and image, respectively, and S is the length of the image feature sequence.

[0145] (4) Cross-modal attention calculation:

[0146] Cross-modal attention between text and image is calculated as shown in Equations (26) and (27).

[0147]

[0148] Among them, A text→img ∈R B×h×L×S A represents the attention of a text query to the image key. img→text ∈R B×h×S×L This indicates that the image query focuses on the text key.

[0149] (5) Apply masking and normalization:

[0150] Masks are applied to text self-attention and image-to-text cross-modal attention, and then all attention scores are normalized by softmax, as shown in Equations (28) to (33).

[0151]

[0152]

[0153] Where M∈R B×1×1×L It is the expanded text mask.

[0154] (6) Calculate attention output:

[0155] The outputs of self-attention and cross-modal attention are calculated using normalized attention weights, as shown in equations (34) to (37).

[0156]

[0157] (7) Reshape back to the original dimension and merge:

[0158] The attention output is reshaped back to the original dimension and the results of self-attention and cross-modal attention are proportionally fused, as shown in Equations (38) to (41).

[0159]

[0160] Among them, T enh anced ∈R B×L×D and V enh anced ∈R B×S×D These are the enhanced text and image features, respectively.

[0161] (8) Final projection and regularization:

[0162] The fused features are then subjected to final projection and dropout regularization, as shown in equations (42) and (43).

[0163]

[0164] in, and These are the weights and biases of the text and image output projections, respectively, T. final ∈R B×L×D and V final ∈R B×S×D These are the enhanced features of the final output.

[0165] 3. Dynamic Similarity Weighted Fusion Module (DSWF)

[0166] The DSWF module dynamically adjusts the fusion weights of text and images based on multidimensional similarity, overcoming the shortcomings of existing fixed weights and single similarity methods. It dynamically adjusts multimodal fusion weights to ensure that the influence of image features is automatically reduced when the text and images are unrelated. The module structure is as follows: Figure 4 As shown, the technical implementation is as follows:

[0167] (1) Feature projection and global representation calculation:

[0168] The enhanced text and image features of the CMSIA module are projected and the global representation vector (average pooling) is calculated as shown in formulas (44) to (47).

[0169]

[0170] in, and These are the weights and biases of the DSWF projections of the text and image, respectively, T proj V proj ∈R B×L / S×D It is the projected feature, T global V global ∈R B×D It is a global representation.

[0171] (2) Three-dimensional similarity calculation:

[0172] Cosine similarity (semantic similarity): Calculates the dot product of normalized text and the global representation of an image;

[0173] MLP similarity (feature difference similarity): performs a non-linear transformation on the differences between the global representations of text and images;

[0174] Hash similarity (structural similarity): Calculates the complement of the Hamming distance between the hash codes of text and images.

[0175] The above three similarities are calculated as shown in formulas (48) to (50).

[0176]

[0177] sim mlp =MLP(|T global -V global |) (49)

[0178]

[0179] in, and It is the normalized global representation. MLP is a two-layer perceptron (specifically, the structure is Linear(D,D / 4)→ReLU→Linear(D / 4,1)→Sigmoid). H d It is the hash code dimension.

[0180] sim cos ,sim mlp ,sim h ash ∈R B×1These represent cosine similarity, MLP similarity, and hash similarity, respectively.

[0181] (3) Linear similarity fusion:

[0182] The three similarities are fused through a linear layer to obtain the dynamic fusion weight α, as shown in formula (51).

[0183] α=σ(W linear [sim cos ;sim mlp ;sim hash ]+b linear (51)

[0184] Among them, W linear ∈R 3×1 and b linear ∈R represents the weights and biases of the linear fusion, σ is the sigmoid activation function, and α∈R B×1 It is a dynamic fusion weight, with a value range of [0,1], used to adaptively adjust the fusion ratio of text and image features.

[0185] (4) Cross-modal enhancement and fusion:

[0186] Text features are enhanced through their own transformations, as shown in Equation (52).

[0187]

[0188] in, and It refers to the weights and biases of the text enhancement layer, T. enh anced ∈R B ×L×D It is an enhanced text feature.

[0189] Image features are transformed using a weight matrix. The interaction with text features is calculated, as shown in equations (53) to (55).

[0190]

[0191] C cross =T final ·(V weigh ted ) T (55)

[0192] in, It is the expanded global image feature, W fusion ∈R D×D It is the fusion weight matrix, V weigh ted ∈R B×1×DIt is the image feature after weighted transformation, C cross ∈R B×L×1 It is a cross-modal interaction result.

[0193] (5) Final fusion output:

[0194] The text features and cross-modal interaction results are fused using dynamic weight α, as shown in Equation (56).

[0195] O = T enh anced +α·C cross (56)

[0196] Where, α∈R B×1×1 These are the extended dynamic fusion weights, O∈R B×L×D It is the final fusion output.

[0197] 4. Local-Global Sequence Modeling Module (LGSM)

[0198] The LGSM module applies local attention and global token aggregation mechanisms to the fused feature sequences, enhancing sequence modeling capabilities and effectively capturing the contextual dependencies of entities. The specific model framework is as follows: Figure 5 As shown.

[0199] (1) Feature preprocessing:

[0200] The residual connections that preserve the input features are then used to perform local and global normalization on the input features, as shown in equations (57) to (59).

[0201] R = O.clone() (57)

[0202] X local =LayerNorm local (O) (58)

[0203] X global =LayerNorm global (O) (59)

[0204] Where, R∈R B×L×D It is a residual connection, X local ,X global ∈R B×L×D These are the features after local and global normalization, respectively.

[0205] (2) Local attention mechanism:

[0206] Multi-head self-attention is used to process the normalized features and capture the local contextual relationships in the sequence, as shown in Equation (60).

[0207]

[0208] in, These are features resulting from local attention processing.

[0209] (3) Global token aggregation:

[0210] The global normalized features are transformed by query and the similarity with the global token is calculated as shown in formulas (61) to (65).

[0211]

[0212] G = G init .unsqueeze(0).expand(B,-1,-1) (62)

[0213] S = Q global ·G T (63)

[0214] A global =softmax(S) (64)

[0215] C global =A global ·G (65)

[0216] in, and It is the weight and bias of the global query transformation, Q global ∈R B ×L×D This is the result of the query transformation. It is the initial global token (N) t This refers to the number of tokens, usually 8. It is an expanded global token. It is a similarity score. It is attention weight, C global ∈R B×L×D It is the global context.

[0217] (4) Local-global feature fusion:

[0218] The local attention result is added to the global context, processed through a projection layer and dropout, and residual connections are applied as shown in equations (66) to (68).

[0219]

[0220] X proj =Dropout(X fused W proj +b proj (67)

[0221] X res =R+X proj (68)

[0222] Among them, X fused ∈R B×L×D It is a fusion feature, W proj ∈R D×D and b proj ∈R D These are the weights and biases of the projection layer, X proj ∈R B×L×D It is the projected feature, X res ∈R B×L×D These are the characteristics after residual connection.

[0223] (5) Feedforward Neural Network:

[0224] Features are further processed through a two-layer feedforward network, as shown in equations (69) to (71).

[0225] X norm =LayerNorm(X res (69)

[0226] X ffn =GELU(X norm W ffn1 +b ffn1 W ffn2 +b ffn2 (70)

[0227] O final =X res +Dropout(X ffn (71)

[0228] Among them, X norm ∈R B×L×D It is the normalized feature, W ffn1 ∈R D×4D W ffn2 ∈R 4D×D and b ffn1 ∈R 4D ,b ffn2 ∈R D These are the weights and biases of the feedforward network, X ffn ∈R B×L×D It is the output of the feedforward network, O final ∈R B×L×D This is the final output of the LGSM module.

[0229] 5 Entity Annotation Layer

[0230] Finally, the system uses a Conditional Random Field (CRF) to perform sequence labeling on the fused features and outputs the entity prediction results. The features output by LGSM are subjected to Dropout regularization and a linear classifier to obtain the emission probability matrix. Then, the label transition probability is considered through a Conditional Random Field, and finally, the entity label sequence is decoded.

[0231] (1) Feature transformation:

[0232] Dropout regularization is applied to the features output by LGSM; the features are then mapped to the label space using a linear classifier.

[0233] (2) CRF sequence annotation:

[0234] During the training phase, the CRF loss is calculated; the Viterbi algorithm is used to decode the optimal label sequence.

[0235] (3) Multi-task learning:

[0236] During the training phase, in addition to the CRF loss, a consistency constraint loss is calculated to ensure consistency among text, images, and fused features; and a hash contrast loss is calculated to make the hash representations of text and images more similar. The total loss is a weighted sum of these three losses.

[0237] 3. Evaluation Methods

[0238] 3.1 Evaluation Indicators:

[0239] Precision: Number of correctly predicted entities / Total number of predicted entities;

[0240] Recall: The number of correctly predicted entities / the number of entities in the standard answer;

[0241] F1 score: the harmonic mean of precision and recall.

[0242] 3.2 Entity-level assessment:

[0243] Precision, recall, and F1 score were evaluated separately for each entity type (PER, LOC, ORG, MISC).

[0244] Table 1 compares this model with other multimodal named entity recognition models in recent years (Twitter 2015 dataset).

[0245] Table 2 compares the model with other multimodal named entity recognition models in recent years (Twitter 2017 dataset).

[0246]

[0247]

[0248] The values ​​in bold red indicate the optimal performance.

[0249] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A multimodal named entity recognition method based on dynamic fusion of multidimensional similarity, characterized in that, Includes the following steps: S1: Feature extraction. The feature extraction module extracts features from the input text and image, using a pre-trained language model and a convolutional neural network to extract initial features, respectively. S2: Cross-modal semantic interaction. The Cross-modal Semantic Interaction Attention Module (CMSIA) adopts a dual-path cross-modal attention mechanism to achieve bidirectional semantic interaction and enhancement between text and image features. S3: Dynamic similarity weighted fusion. The dynamic similarity weighted fusion module DSWF dynamically adjusts the fusion weights of text and images based on multidimensional similarity, and dynamically adjusts the multimodal fusion weights to ensure that the influence of image features can be automatically reduced when the text and images are unrelated. S4: Local-Global Sequence Modeling. The Local-Global Sequence Modeling module (LGSM) applies local attention and global token aggregation mechanisms to the fused feature sequences to enhance sequence modeling capabilities and capture the contextual dependencies of entities. S5: Entity labeling. The entity labeling layer finally uses a Conditional Random Field (CRF) to perform sequence labeling on the fused features and outputs the entity prediction results.

2. The multimodal named entity recognition method based on dynamic fusion of multidimensional similarity according to claim 1, characterized in that, In step S1, the method for text feature extraction is as follows: S11a: The text input is processed by a BERT pre-trained model for feature extraction. The model is: H text =BERT(x text A mask ,T type ), where x text ∈R B×L A represents an input token sequence of batch size B and sequence length L. mask ∈R B×L T represents the attention mask. type ∈R B×L Indicates the token type ID, H text ∈R B×L×768 Represents the text features output by BERT; S12a: Text features are mapped to a unified dimensional space through a projection layer. The model is as follows: in, and These are the weights and biases of the projection layer, where D = 768 is the hidden dimension, and T ∈ R. B×L×D These are the text features after projection.

3. The multimodal named entity recognition method based on dynamic fusion of multidimensional similarity according to claim 1, characterized in that, In step S1, the image feature extraction method is as follows: S11b: The image input undergoes feature extraction via ResNet50 (Residual Network), and its model is: F img =ResNet50(X img ), where x img ∈R B×3×H×W This represents an input image (3 channels, height H, width W) with a batch size of B, F img ∈R B×2048×h×w This represents the feature map output by ResNet50; S12b: The feature map is reshaped into a sequence and projected. The processing model is as follows: in, It is the reshaped feature sequence. and These are the weights and biases of the projection layer, V∈R B ×(h×w)×D These are the features of the projected image.

4. The multimodal named entity recognition method based on dynamic fusion of multidimensional similarity according to claim 1, characterized in that, The S1 step also includes a hash encoding step, which maps high-dimensional features to a low-dimensional binary space through hash encoding, enhances the semantic distinguishability of features, and provides a basis for hash similarity calculation of the DSWF module, thereby improving the accuracy and efficiency of similarity calculation.

5. The multimodal named entity recognition method based on dynamic fusion of multidimensional similarity according to claim 4, characterized in that, The implementation steps of hash encoding in step S1 are as follows: The hash encoding process is as follows: H hash =tanh(φ(X,W1,b1,w2,b2)), where φ represents two layers of MLP operations; The specific calculation process is as follows: φ(X,W1,b1,W2,b2)=R e LU(XW1+b1)W2+b2, where X∈R B×D This represents the input features (which can be text or global image features). and These are the weights and biases of the first layer MLP. and It refers to the weights and biases of the second-layer MLP, H. d It is the hash code dimension (128). It is a continuous hash code representation; During the inference phase, the consecutive hash codes are binarized, the process being: H binary =sign(H hash ).

6. The multimodal named entity recognition method based on dynamic fusion of multidimensional similarity according to claim 5, characterized in that, In step S1, hash encoding supports two granularities: global and local, which are respectively: Global hash encoding: Encodes features after average pooling; its model is as follows: Local hashing encoding: Selects key tokens / regional features for encoding, and its model is: W attn =softmax(TT) T ), T key =T[batch indices,TopK]、 Among them, W attn For the attention weight matrix, TopK i T represents the indexes of the top K key tokens selected for the i-th sample. key ∈R B×K×D Indicates key token characteristics, It is a local hash code; Ultimately, the global and local hash codes are combined to enhance the feature representation, as follows:

7. The multimodal named entity recognition method based on dynamic fusion of multidimensional similarity according to claim 1, characterized in that, Step S2 includes the following steps: S21: Multi-head attention preprocessing, performing Q, K, V linear transformations on text and image features respectively; the transformed feature dimensions are [batch size, sequence length, hidden dimension]; the features are reshaped into a multi-head format [batch size, number of heads, sequence length, head dimension], and the computation model is as follows: in, It is a linear transformation matrix. It is the corresponding bias term; S22: Feature reshaping into a multi-head format, reshaping Q, K, and V into a multi-head format, the calculation model is as follows: Where h is the number of attention heads, It is the dimension of each head. It is the reshaped multi-head format, K text V text Q img ,K img V img The processing method is the same; S23: Self-attention calculation, calculating self-attention within text and images, the calculation model is as follows: in, and These are the self-attention matrices for text and image, respectively, and S is the length of the image feature sequence; S24: Cross-modal attention calculation, which calculates cross-modal attention between text and image and between image and text. The calculation model is as follows: Among them, A text→img ∈R B×h×L×S A represents the attention of a text query to the image key. img→text ∈R B×h×S×L This indicates the attention given to the text key by the image query; S25: Apply masks and normalization to text self-attention and image-to-text cross-modal attention, then perform softmax normalization on all attention scores. The calculation models are as follows: Where M∈R B×1×1×L It is the expanded text mask; S26: Calculate the attention output, using normalized attention weights to calculate the outputs of self-attention and cross-modal attention. The calculation models are as follows: S27: Reshape back to the original dimension and fuse. Reshape the attention output back to the original dimension and fuse the results of self-attention and cross-modal attention proportionally. The fusion model is as follows: Among them, T enhanced ∈R B×L×D and V enhanced ∈R B×S×D These are the enhanced text and image features, respectively. S28: Final Projection and Regularization. The fused features are then subjected to final projection and dropout regularization. The models are as follows: in, and These are the weights and biases of the text and image output projections, respectively, T. final ∈R B×L×D and V final ∈R B×S×D These are the enhanced features of the final output.

8. The multimodal named entity recognition method based on dynamic fusion of multidimensional similarity according to claim 1, characterized in that, Step S3 includes the following steps: S31: Feature projection and global representation calculation. The enhanced text and image features from the CMSIA module are projected, and the global representation vector is calculated. The calculation process model is as follows: in, and These are the weights and biases of the DSWF projections of the text and image, respectively, T proj V proj ∈R B×L / S×D It is the projected feature, T global V global ∈R B×D It is a global representation; S32: Three-dimensional similarity calculation: The dot product of normalized text and image global representations is calculated using cosine similarity: The differences between the global representations of text and images are nonlinearly transformed using MLO similarity: sim mlp =MLP(|T global -V global |); The Hamming distance complement of text and image hash codes is calculated using hash similarity. in, and It is the normalized global representation. MLP is a two-layer perceptron (specifically, the structure is Linear(D,D / 4)→ReLU→Linear(D / 4,1)→Sigmoid). H d It is the hash code dimension; S33: Linear similarity fusion, which fuses the three similarities through a linear layer to obtain a dynamic fusion weight α. The calculation model is: α = σ(W linear [sim cos ;sim mlp ;sim hash ]+b linear ), where W linear ∈R 3×1 and b linear ∈R represents the weights and biases of the linear fusion, σ is the sigmoid activation function, and α∈R B×1 It is a dynamic fusion weight, with a value range of [0,1], used to adaptively adjust the fusion ratio of text and image features; S34: Cross-modal enhancement and fusion: Text features are enhanced through self-transformation, and the model is as follows: in, and It refers to the weights and biases of the text enhancement layer, T enhanced ∈R B×L×D These are enhanced text features; Image features are transformed through a weight matrix, and the model is as follows: The interaction between computational image features and text features is modeled as follows: C cross =T final ·(V weighted ) T ; in, It is the expanded global image feature, W fusion ∈R D×D It is the fusion weight matrix, V weighted ∈R B×1×D It is the image feature after weighted transformation, C cross ∈R B×L×1 It is a cross-modal interaction result; S35: Final fusion output, using dynamic weights α to fuse textual features and cross-modal interaction results, with the model being: O = T enhanced +α·C cross , where α∈R B×1×1 These are the extended dynamic fusion weights, O∈R B×L×D It is the final fusion output.

9. The multimodal named entity recognition method based on dynamic fusion of multidimensional similarity according to claim 1, characterized in that, Step S4 includes the following steps: S41: Feature preprocessing, saving residual connections of input features, and performing local and global normalization on the input features. The processing model is: R = O.clone(), X local =LayerNorm local (O), X global =LayerNorm global (O), where R∈R B×L×D It is a residual connection, X local ,X global ∈R B×L×D These are the features after local and global normalization, respectively; S42: Local attention mechanism, which uses multi-head self-attention to process normalized features and capture local contextual relationships in the sequence. Its model is as follows: in, These are features resulting from local attention processing; S43: Global token aggregation, which performs query transformation on globally normalized features and calculates the similarity with global tokens. Its model is as follows: G = G init .unsqueeze(0).expand(B,-1,-1)、S=Q global ·G T A global =softmax(S),C global =A global ·G, where, and It is the weight and bias of the global query transformation, Q global ∈R B×L×D This is the result of the query transformation. It is the initial global token (N) t This refers to the number of tokens, usually 8. It is an expanded global token. It is a similarity score. It is attention weight, C global ∈R B×L×D It is the global context; S44: Local-Global Feature Fusion, which adds the local attention results to the global context, and processes them through a projection layer and dropout. The processing model is as follows: X proj =Dropout(X fused W proj +b proj ), X res =R+X proj , where X fused ∈R B×L×D It is a fusion feature, W proj ∈R D×D and b proj ∈R D These are the weights and biases of the projection layer, X proj ∈R B×L×D It is the projected feature, X res ∈R B×L×D These are characteristics resulting from residual joins; S45: Feedforward Neural Network, which further processes features through two layers of feedforward network. Its model is: X norm =LayerNorm(X res ), X ffn =GELU(X norm W ffn1 +b ffn1 W ffn2 +b ffn2 O final =X res +Dropout(X ffn ), where X norm ∈R B×L×D It is the normalized feature, W ffn1 ∈R D×4D W ffn2 ∈R 4D×D and b ffn1 ∈R 4D ,b ffn2 ∈R D These are the weights and biases of the feedforward network, X ffn ∈R B×L×D It is the output of the feedforward network, O final ∈R B×L×D This is the final output of the LGSM module.

10. A multimodal named entity recognition method based on dynamic fusion of multidimensional similarity according to claim 1, characterized in that, Step S5 includes the following steps: S51: Feature transformation, applying dropout regularization to the features output by LGSM; mapping to the label space through a linear classifier; S52: CRF sequence labeling; CRF loss is calculated during the training phase; the best label sequence is decoded using the Viterbi algorithm. S53: Multi-task learning. During the training phase, in addition to the CRF loss, a consistency constraint loss is calculated to ensure consistency between text, images, and fused features; and a hash contrast loss is calculated to make the hash representations of text and images more similar. The total loss is the weighted sum of these three losses.

Images