Multi-modal named entity recognition method based on image-text semantic alignment and interaction

Abstract

The invention provides a multi-modal named entity recognition method based on image-text semantic alignment and interaction, and belongs to the technical field of natural language processing. The method comprises the steps that S1, an input text sequence and an input image are acquired; s2, encoding the text sequence to obtain text feature representation; s3, encoding the image to obtain visual feature representation; s4, constructing a multi-modal graph based on the text feature representations and the visual feature representations; step S5, performing semantic alignment on the text feature representation and the visual feature representation by minimizing a Wasserstein distance between the text feature representation and the visual feature representation; s6, performing intra-modal and inter-modal interaction on the text feature representation and the visual feature representation after semantic alignment to generate fused multi-modal feature representation; and S7, outputting a named entity recognition result based on the fused multi-modal feature representation. The method has the advantage that the accuracy and robustness of named entity recognition are greatly improved.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to a multimodal named entity recognition method based on image-text semantic alignment and interaction. Background Technology

[0002] Named Entity Recognition (NER) is one of the core tasks of Natural Language Processing (NLP), aiming to identify entities with specific semantic categories from unstructured text, such as person names (PER), place names (LOC), organizations (ORG), and others (MISC). With the rapid development of social media platforms, users often post multimedia content with paired text and images, leading to the emergence of Multi-modal Named Entity Recognition (MNER). MNER improves the accuracy of entity recognition by fusing textual and visual modal data. For example, on platforms like Twitter, relying solely on text may result in misclassification of entity types, while image information can provide crucial clues.

[0003] The essence of MNER lies in the effective fusion and alignment of text and image modal information. Early methods have made some progress; for example, Moon et al. proposed the LSTM-CNN architecture, combining text and images through a universal modal attention module; Fu et al. introduced a common attention mechanism to dynamically control the combination of text and image representations; Yu et al. implemented a unified multimodal architecture based on Transformer; Liu et al. utilized uncertainty awareness techniques to improve performance; and Huang et al. mitigated modal heterogeneity differences through gating de-entanglement networks. These methods mainly rely on cross-modal attention mechanisms for global information interaction, but they have the following limitations: 1. Insufficient semantic alignment: Text and image features come from different encoders, resulting in semantic gaps and making it difficult to achieve fine-grained alignment, which weakens the guiding role of visual information for text entities.

[0004] 2. Coarse feature interaction: Existing methods such as multi-head attention mechanisms focus on global interaction and ignore the local semantic interaction between objects in images and entities in text, and cannot establish accurate cross-modal mapping.

[0005] Therefore, how to provide a multimodal named entity recognition method based on image-text semantic alignment and interaction to improve the accuracy and robustness of named entity recognition has become an urgent technical problem to be solved. Summary of the Invention

[0006] The technical problem to be solved by this invention is to provide a multimodal named entity recognition method based on image-text semantic alignment and interaction, so as to improve the accuracy and robustness of named entity recognition.

[0007] This invention is implemented as follows: a multimodal named entity recognition method based on image-text semantic alignment and interaction, comprising the following steps: Step S1: Obtain the input multimodal data, including text sequences and images; Step S2: Encode the text sequence using a pre-trained language model to obtain text feature representations; Step S3: Use a pre-trained visual model to detect and encode visual objects in the image to obtain visual feature representations; Step S4: Construct a multimodal graph based on the text feature representations and visual feature representations; Step S5: Semantically align the text feature representation and visual feature representation in the multimodal graph by minimizing the Wasserstein distance between the text feature representation and the visual feature representation; Step S6: Adopt an adaptive joint attention mechanism to perform intra-modal and inter-modal interactions on the semantically aligned text feature representation and visual feature representation to generate a fused multimodal feature representation; Step S7: Use a conditional random field decoder to perform sequence labeling on the fused multimodal feature representation and output the named entity recognition result.

[0008] Furthermore, in step S2, the language model adopts the BERT model, and the text features are represented as context-dependent vector sequences.

[0009] Furthermore, in step S3, the visual model adopts the ResNet model, the visual object is detected by Mask R-CNN, the visual features are represented as region vectors, and projected into the same dimensional space as the text feature representation.

[0010] Furthermore, in step S4, the multimodal graph is an undirected graph; The nodes of the multimodal graph include text feature representations and visual feature representations; the edges include intramodal edges and cross-modal edges, which are used to represent the semantic correspondence between text feature representations and visual feature representations.

[0011] Furthermore, in step S4, during the construction of the multimodal graph, a graph convolutional neural network is used for feature propagation to enhance the semantic representation of nodes; the construction of cross-modal edges is based on the association between noun phrases and visual objects, and is implemented through the Stanford parser and vision toolkit.

[0012] Furthermore, in step S5, the Wasserstein distance is calculated using normalized mean square error as the distance metric, and the minimum transmission cost is solved through optimal transmission theory. The loss function is used to supervise mode alignment.

[0013] Furthermore, in step S6, the adaptive joint attention mechanism includes: Adaptive attention unit: performs intramodal interaction on text feature representations or visual feature representations, and dynamically adjusts attention weights using a multi-head attention mechanism and adaptive factors; Adaptive Guided Attention Unit: Uses text feature representation as query and visual feature representation as key and value to perform cross-modal interaction and output attention-based visual feature representation; Multimodal feature generation unit: The text feature representation output by the adaptive attention unit is concatenated with the visual feature representation output by the adaptive guided attention unit to form the final fused multimodal feature representation.

[0014] Furthermore, both the adaptive attention unit and the adaptive guided attention unit include layer normalization, feedforward network, and residual connection to enhance feature stability.

[0015] Furthermore, in step S7, the conditional random field decoder is trained using maximum conditional likelihood estimation and infers the optimal label sequence using the Viterbi algorithm.

[0016] The advantages of this invention are: 1. By acquiring multimodal data including text sequences and images as input, a pre-trained language model is used to encode the text sequences to obtain text feature representations, and a pre-trained visual model is used to detect and encode visual objects in the images to obtain visual feature representations. Then, a multimodal graph is constructed based on the text and visual feature representations. Semantic alignment is performed on the text and visual feature representations in the multimodal graph by minimizing the Wasserstein distance between them. Next, an adaptive common attention mechanism is used to perform intra-modal and inter-modal interactions on the semantically aligned text and visual feature representations to generate a fused multimodal feature representation. Finally, a conditional random field decoder is used to perform sequence labeling on the fused multimodal feature representation, outputting the named entity recognition result. That is, firstly, Wasserstein... Erstein distance and cross-modal graph structure enable fine-grained semantic alignment between text sequences and visual objects in images, effectively narrowing the semantic gap between modalities and laying the foundation for precise interaction. Subsequently, an adaptive common attention mechanism is adopted to first strengthen the intra-modal feature representations within both text and visual feature representations, and then use text feature representations as a guide for directional cross-modal interaction. This allows for precise utilization of local visual cues in images to resolve ambiguities in text sequences (such as determining whether "apple" refers to a fruit or a company). This structured alignment and refined interaction mechanism enables more reliable fusion of multimodal information when faced with ambiguous expressions or missing information common in social media texts. Finally, conditional random field decoding outputs more accurate and stable entity recognition results, thereby greatly improving the accuracy and robustness of named entity recognition.

[0017] 2. Through a progressive design of “encoding → constructing a multimodal graph → semantic alignment → interactive fusion → decoding output”, deep information integration from raw data to the final recognition result is achieved. This structured process ensures that the image and text information can be fully aligned and interacted at different semantic levels, rather than superficial feature merging, thereby significantly improving the model’s ability to understand complex multimodal semantics.

[0018] 3. Employing Wasserstein distance (also known as "bulldozer distance") to minimize the difference between text and visual feature representations is a key innovation. Compared to traditional Euclidean distance or cosine similarity, Wasserstein distance can measure the overall difference between two probability distributions and is better at handling the structured distribution differences between text and image data in the representation space. By achieving semantic alignment through optimal transport theory, it can more accurately map text words and image regions describing the same entity to similar semantic spaces, fundamentally alleviating the "heterogeneous gap" problem in multimodal learning and laying a solid foundation for subsequent high-quality interactive fusion.

[0019] 4. The constructed multimodal graph (including intramodal and cross-modal edges) explicitly models the complex relationships between text units, visual objects, and text-image units, providing a structured pathway for information dissemination. Building upon this, the adaptive joint attention mechanism further enables dynamic and guided interaction. The "adaptive guided attention unit" uses text as the query and visual information as the key, allowing image information to be dynamically filtered and invoked by text semantics. The "adaptive attention unit" strengthens intramodal context modeling. This dual interaction mechanism, combining graph convolution and adaptive attention, can adaptively determine which complementary information to focus on and absorb from another modality based on the specific context, making the fusion strategy more flexible and intelligent.

[0020] 5. The use of strong foundational models such as BERT, ResNet, and Mask R-CNN ensures powerful single-modal feature extraction capabilities; the establishment of associations between noun phrases and visual objects through tools such as the Stanford parser enhances the rationality of cross-modal edge construction; layer normalization and residual connections are added to the attention mechanism (claim 8) to improve training stability; these detailed designs enable the scheme to robustly handle real-world challenges such as numerous objects in images and the incomplete direct correspondence between text descriptions and visual content, and finally outputs the globally optimal label sequence through the Conditional Random Field (CRF) decoder, improving the accuracy and boundary consistency of entity recognition.

[0021] 6. By constructing a multimodal graph structure to model the complex relationships between text and image units, and innovatively introducing Wasserstein distance for deep semantic alignment, the heterogeneity gap between text and visual modalities is effectively overcome. On this basis, an adaptive common attention mechanism is adopted to achieve dynamic and refined intramodal and cross-modal interactions, enabling the model to intelligently filter and fuse complementary information based on context. This series of collaboratively designed core modules ultimately enable the system to significantly improve the accuracy and robustness of named entity recognition when facing real-world scenarios such as ambiguous text references or ambiguous image content. Attached Figure Description

[0022] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0023] Figure 1 This is a flowchart of a multimodal named entity recognition method based on image-text semantic alignment and interaction according to the present invention.

[0024] Figure 2 This is a schematic diagram of the architecture of the present invention. Detailed Implementation

[0025] The technical solution in this application embodiment follows the following general approach: First, by utilizing Wasserstein distance and cross-modal graph structure, fine-grained semantic alignment is achieved between text sequences and visual objects in images, narrowing the semantic gap between modalities. Subsequently, an adaptive common attention mechanism is employed to first strengthen the intra-modal feature representations within both text and visual feature representations, and then guide directional cross-modal interaction using text feature representations. This allows for the precise utilization of local visual cues in images to resolve ambiguities in text sequences. This structured alignment and refined interaction mechanism enables more reliable fusion of multimodal information when faced with ambiguous expressions or missing information commonly found in social media texts. Finally, conditional random field decoding outputs more accurate and stable entity recognition results, thereby improving the accuracy and robustness of named entity recognition.

[0026] Please refer to Figures 1 to 2 As shown, a preferred embodiment of the multimodal named entity recognition method based on image-text semantic alignment and interaction of the present invention includes the following steps: Step S1: Obtain the input multimodal data, including text sequences and images; Step S2: Encode the text sequence using a pre-trained language model to obtain text feature representations; Step S3: Use a pre-trained visual model to detect and encode visual objects in the image to obtain visual feature representations; Step S4: Construct a multimodal graph based on the text feature representations and visual feature representations; Step S5: Semantically align the text feature representation and visual feature representation in the multimodal graph by minimizing the Wasserstein distance between the text feature representation and the visual feature representation; Step S6: Adopt an adaptive joint attention mechanism to perform intra-modal and inter-modal interactions on the semantically aligned text feature representation and visual feature representation to generate a fused multimodal feature representation; Step S7: Use a conditional random field decoder to perform sequence annotation on the fused multimodal feature representation and output the named entity recognition result. The types of named entities include at least Person (PER), Location (LOC), Organization (ORG), and Miscellaneous (MISC).

[0027] In step S2, the language model uses the BERT model, and the text features are represented as context-dependent vector sequences. The BERT model can extract different feature representations of the same word based on different contexts.

[0028] For the input text sequence Add a special [CLS] tag at the beginning and a [SEP] tag at the end to get Then, input the data into the BERT model to obtain text feature representations that include contextual information: ; in, This represents the text feature representation. n represents the length of the input text sequence; d represents the hidden dimension.

[0029] In step S3, the visual model adopts the ResNet model, the visual object is detected by Mask R-CNN, the visual features are represented as region vectors and projected into the same dimensional space as the text feature representation.

[0030] First, a pre-trained Mask R-CNN model is used to detect a set of visual objects in an image, represented as... For each visual object, a ResNet model is used as the image encoder to extract visual feature representations. Specifically, the size of each visual object is first adjusted to 224×224 pixels, and then the ResNet152 image recognition model is used to obtain the output of the last convolutional layer, i.e., the visual feature representation. ; It divides the original image into 7×7 = 49 regions of equal size, each region consisting of a 2048-dimensional vector. Representation. Projecting the visual feature representation onto the same dimension as the text feature representation: ; in, Represents learnable parameters; Represents visual features.

[0031] In step S4, the multimodal graph is an undirected graph; The nodes of the multimodal graph include text feature representations and visual feature representations; the edges include intramodal edges and cross-modal edges, used to represent the semantic correspondence between text feature representations and visual feature representations. In specific implementations, solid lines represent relationships within the same modality, and dashed lines represent relationships between text modalities and image modalities.

[0032] In step S4, during the construction of the multimodal graph, a graph convolutional neural network is used for feature propagation to enhance the semantic representation of nodes; the construction of cross-modal edges is based on the association between noun phrases and visual objects, and is implemented through the Stanford parser and vision toolkit.

[0033] To represent the fine-grained correspondence between text sequences and images, two different modalities, a multimodal graph was constructed, denoted as G=(V,E), where V represents nodes and E represents edges. Specifically, text feature representations are used as text nodes, while visual feature representations are used as visual object nodes.

[0034] Graph Convolutional Networks (GCNs) can effectively learn information from text nodes and visual object nodes in a graph. Therefore, by using GCNs to learn multimodal graphs and employing a multi-layered GCN propagation mechanism, graph convolution operations are applied to identify and model the complex relationships between text sequences and images, promoting semantic interaction between text feature representations and visual feature representations. The calculation formula is as follows: ; Where GCN stands for Graph Convolutional Neural Network; layer represents the number of layers; and σ represents the activation function.

[0035] To better integrate text and visual feature representations, two multi-layer perceptrons (MLPs) with ReLU activation functions are introduced. These MLPs project the text and visual feature representations onto a spatial vector of the same dimension. The computation process is as follows: ; ; in, All of these represent learnable parameters.

[0036] In step S5, the Wasserstein distance is calculated using normalized mean square error as the distance metric, and the minimum transmission cost is solved through optimal transmission theory. The loss function is used to supervise mode alignment.

[0037] In multimodal information fusion, discrepancies exist between text and image representations due to the lack of alignment between semantic knowledge between the two different modalities. Therefore, to achieve more consistent representations in the final feature extraction, the Wasserstein distance minimization method, also known as Earth Mover Distance (EMD), is introduced. As a metric for measuring the distance difference between two probability distributions, it aims to reduce the spatial distribution distance between text and image modalities. Using this metric, explicit alignment relationships across modalities can be established more reasonably to obtain semantic relationships between text entities and images, thereby promoting consistency and correlation between text and image feature representations. The "transmission cost" between text and image features is calculated using Wasserstein distance. The key to this method is calculating the optimal transmission between text and image features to achieve unsupervised alignment. Specifically, the core idea of ​​EMD originates from transportation theory, treating one distribution as a "pile of dirt" and another as a "target pile," and finding the minimum cost required to move the source distribution to the target distribution under the optimal transmission strategy. Here, the cost is assumed to be the amount of soil moved multiplied by the distance moved; based on this, a movement weight can be assigned to the feature representations of text and images respectively, and the calculation process is as follows: ; ; in, and The movement weights represent the text feature representation and the visual feature representation, respectively. These weights are initialized to 1 / N and 1 / W, where N is the length of the text sequence and W is the number of visual objects in the target region. To evaluate the movement weights... Move to The transmission cost, using the normalized mean squared error (MSE) as the distance metric, is calculated as follows: ; The core objective of local alignment is to find a transport flow matrix F that minimizes the transfer of features from... Mapped to The cumulative cost is calculated as follows: ; Where MSE represents the normalized mean square error; WORK represents the amount of handling work. The text features represent the transmission traffic that is moved to the visual features.

[0038] This optimization problem is solved using Optimal Transport (OT) theory. The cost of local alignment is expressed as the total amount of data required to complete the transmission, and then normalized using the total transmission amount. The total transmission amount between text and image is expressed as: ; The goal of this method, which measures the minimum "transportation cost" between two probability distributions, is to minimize this loss, i.e., minimize the overall difference between the two distributions. By minimizing the Wasserstein distance between text and image, semantic alignment between the text and image modalities can be facilitated, thereby better supervising the fusion of these two modalities. The calculation process is as follows: ; in, This represents the Wasserstein distance loss, used to measure the distributional difference between textual and visual feature representations.

[0039] In step S6, the adaptive joint attention mechanism includes: Adaptive attention unit: performs intramodal interaction on text feature representations or visual feature representations, and dynamically adjusts attention weights using a multi-head attention mechanism and adaptive factors; Adaptive Guided Attention Unit: Uses text feature representation as query and visual feature representation as key and value to perform cross-modal interaction and output attention-based visual feature representation; Multimodal feature generation unit: The text feature representation output by the adaptive attention unit is concatenated with the visual feature representation output by the adaptive guided attention unit to form the final fused multimodal feature representation.

[0040] Both the adaptive attention unit and the adaptive guided attention unit include layer normalization, feedforward network and residual connection to enhance feature stability.

[0041] After constructing the multimodal graph, it is necessary to model the relationships between different modalities based on the multimodal graph. The multimodal graph is then input into cross-modal semantic feature interaction for information fusion within and between modalities. To better facilitate semantic interaction between visual and textual feature representations, an adaptive co-attention mechanism (ACM) is introduced into cross-modal semantic feature interaction. After performing local alignment, the textual and visual feature representations are combined into the predicted entity. Cross-modal semantic feature interaction mainly includes methods such as local alignment, adaptive attention, and adaptive guided attention. ACM models the internal interactions of textual or visual feature representations, as well as the interaction relationships between them, ultimately generating multimodal features. Adaptive self-attention (ASA) and adaptive guided attention (AGA) are inspired by the scaled dot product attention mechanism. The formula for calculating attention features is: ; Among them, query ,key And value α is a learnable adaptive factor. This factor dynamically adjusts the attention score of the query matrix Q through element-wise multiplication, enhancing the model's adaptability to different inputs and allowing for more flexible allocation of attention weights. Attention Features It is obtained by weighted summation of all values ​​V based on the attention learned from Q and K.

[0042] ASA primarily consists of a multi-head attention layer, layer normalization, a feedforward network layer, and residual connections. First, the multi-head attention layer takes a set of input features as the query, key, and value. Then, the output features of the multi-head attention layer are transformed through two fully connected layers with ReLU activation and Dropout. ASA applies residual connections with layer normalization to the outputs of the multi-head attention layer and the feedforward network layer. AGA has the same architecture as ASA, but its input contains two different sets of features: text features and visual features. The text features are used as the query, and the visual features are used as the key and value.

[0043] This invention employs a multi-head attention mechanism to improve the representational power of attention features. Each head corresponds to an independent attention function with scaled dot product. Finally, the results from all heads are concatenated. The multi-head adaptive attention formula is as follows: ; ; Here, MultiHead is a multi-head attention function, whose inputs are the query vector Q, the key vector K, and the value vector V. It is the projection matrix of the j-th head, and n is the number of multi-head attention.

[0044] This invention uses ASA and AGA to construct an adaptive joint attention mechanism. The visual and text representations obtained in this way possess intra-modal contextual information and cross-modal semantic interaction information. Since visual and text features contain semantic information from two different modalities, the encoding of the text and image modalities is completed by relatively independent parts. Specifically, firstly, the text feature representation and the visual feature representation are input into separate ASAs to capture the information interaction of the inherent relationships between text or visual objects. Then, the output features of the text ASA are used as query inputs into the AGA, while the output features of the visual object ASA are used as keys and values ​​input into the AGA. Therefore, the AGA can capture the interaction relationships between the two different modalities of visual objects and text entities. The calculation process of the adaptive joint attention mechanism is as follows: ; ; ; in, and These represent the outputs of the ASA units corresponding to the text feature representation and the visual feature representation, respectively. It is the output feature of the AGA unit, which contains attention-based visual feature representations of each word in the text sequence.

[0045] Finally, the text feature representation from the adaptive attention output and the visual feature representation from the adaptive guided attention output are concatenated to generate the final fused multimodal feature representation, as shown in the formula: ; Where C represents the fused multimodal feature representation of the input text sequence, which integrates information from the text and information from visual objects in the image.

[0046] In step S7, the conditional random field decoder is trained using maximum conditional likelihood estimation and infers the optimal label sequence using the Viterbi algorithm.

[0047] This invention employs a Conditional Random Field (CRF) decoder to predict the label sequence, using the following formula: ; in, Let represent the potential function; y represents the set of all possible label sequences; It is a general input sequence of length n.

[0048] During the model training phase, the maximum conditional likelihood function is used for estimation and loss calculation, as shown in the formula: ; The objective of the maximum conditional likelihood logarithm is to maximize the parameters of the log-likelihood function, expressed as: .

[0049] During the reasoning phase, the output sequence that achieves the highest score is predicted using the following formula: .

[0050] In summary, the advantages of this invention are as follows: 1. By acquiring multimodal data including text sequences and images as input, a pre-trained language model is used to encode the text sequences to obtain text feature representations, and a pre-trained visual model is used to detect and encode visual objects in the images to obtain visual feature representations. Then, a multimodal graph is constructed based on the text and visual feature representations. Semantic alignment is performed on the text and visual feature representations in the multimodal graph by minimizing the Wasserstein distance between them. Next, an adaptive common attention mechanism is used to perform intra-modal and inter-modal interactions on the semantically aligned text and visual feature representations to generate a fused multimodal feature representation. Finally, a conditional random field decoder is used to perform sequence labeling on the fused multimodal feature representation, outputting the named entity recognition result. That is, firstly, Wasserstein... Erstein distance and cross-modal graph structure enable fine-grained semantic alignment between text sequences and visual objects in images, effectively narrowing the semantic gap between modalities and laying the foundation for precise interaction. Subsequently, an adaptive common attention mechanism is adopted to first strengthen the intra-modal feature representations within both text and visual feature representations, and then use text feature representations as a guide for directional cross-modal interaction. This allows for precise utilization of local visual cues in images to resolve ambiguities in text sequences (such as determining whether "apple" refers to a fruit or a company). This structured alignment and refined interaction mechanism enables more reliable fusion of multimodal information when faced with ambiguous expressions or missing information common in social media texts. Finally, conditional random field decoding outputs more accurate and stable entity recognition results, thereby greatly improving the accuracy and robustness of named entity recognition.

[0051] 2. Through a progressive design of “encoding → constructing a multimodal graph → semantic alignment → interactive fusion → decoding output”, deep information integration from raw data to the final recognition result is achieved. This structured process ensures that the image and text information can be fully aligned and interacted at different semantic levels, rather than superficial feature merging, thereby significantly improving the model’s ability to understand complex multimodal semantics.

[0052] 3. Employing Wasserstein distance (also known as "bulldozer distance") to minimize the difference between text and visual feature representations is a key innovation. Compared to traditional Euclidean distance or cosine similarity, Wasserstein distance can measure the overall difference between two probability distributions and is better at handling the structured distribution differences between text and image data in the representation space. By achieving semantic alignment through optimal transport theory, it can more accurately map text words and image regions describing the same entity to similar semantic spaces, fundamentally alleviating the "heterogeneous gap" problem in multimodal learning and laying a solid foundation for subsequent high-quality interactive fusion.

[0053] 4. The constructed multimodal graph (including intramodal and cross-modal edges) explicitly models the complex relationships between text units, visual objects, and text-image units, providing a structured pathway for information dissemination. Building upon this, the adaptive joint attention mechanism further enables dynamic and guided interaction. The "adaptive guided attention unit" uses text as the query and visual information as the key, allowing image information to be dynamically filtered and invoked by text semantics. The "adaptive attention unit" strengthens intramodal context modeling. This dual interaction mechanism, combining graph convolution and adaptive attention, can adaptively determine which complementary information to focus on and absorb from another modality based on the specific context, making the fusion strategy more flexible and intelligent.

[0054] 5. The use of strong foundational models such as BERT, ResNet, and Mask R-CNN ensures powerful single-modal feature extraction capabilities; the establishment of associations between noun phrases and visual objects through tools such as the Stanford parser enhances the rationality of cross-modal edge construction; layer normalization and residual connections are added to the attention mechanism (claim 8) to improve training stability; these detailed designs enable the scheme to robustly handle real-world challenges such as numerous objects in images and the incomplete direct correspondence between text descriptions and visual content, and finally outputs the globally optimal label sequence through the Conditional Random Field (CRF) decoder, improving the accuracy and boundary consistency of entity recognition.

[0055] 6. By constructing a multimodal graph structure to model the complex relationships between text and image units, and innovatively introducing Wasserstein distance for deep semantic alignment, the heterogeneity gap between text and visual modalities is effectively overcome. On this basis, an adaptive common attention mechanism is adopted to achieve dynamic and refined intramodal and cross-modal interactions, enabling the model to intelligently filter and fuse complementary information based on context. This series of collaboratively designed core modules ultimately enable the system to significantly improve the accuracy and robustness of named entity recognition when facing real-world scenarios such as ambiguous text references or ambiguous image content.

[0056] While specific embodiments of the present invention have been described above, those skilled in the art should understand that the specific embodiments described are merely illustrative and not intended to limit the scope of the present invention. Equivalent modifications and variations made by those skilled in the art in accordance with the spirit of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A multimodal named entity recognition method based on image-text semantic alignment and interaction, characterized in that: Includes the following steps: Step S1: Obtain the input multimodal data, including text sequences and images; Step S2: Encode the text sequence using a pre-trained language model to obtain text feature representations; Step S3: Use a pre-trained visual model to detect and encode visual objects in the image to obtain visual feature representations; Step S4: Construct a multimodal graph based on the text feature representations and visual feature representations; Step S5: Semantically align the text feature representation and visual feature representation in the multimodal graph by minimizing the Wasserstein distance between the text feature representation and the visual feature representation; Step S6: Adopt an adaptive joint attention mechanism to perform intra-modal and inter-modal interactions on the semantically aligned text feature representation and visual feature representation to generate a fused multimodal feature representation; Step S7: Use a conditional random field decoder to perform sequence labeling on the fused multimodal feature representation and output the named entity recognition result.

2. The multimodal named entity recognition method based on image-text semantic alignment and interaction as described in claim 1, characterized in that: In step S2, the language model adopts the BERT model, and the text features are represented as context-dependent vector sequences.

3. The multimodal named entity recognition method based on image-text semantic alignment and interaction as described in claim 1, characterized in that: In step S3, the visual model adopts the ResNet model, the visual object is detected by Mask R-CNN, the visual features are represented as region vectors and projected into the same dimensional space as the text feature representation.

4. The multimodal named entity recognition method based on image-text semantic alignment and interaction as described in claim 1, characterized in that: In step S4, the multimodal graph is an undirected graph; The nodes of the multimodal graph include text feature representations and visual feature representations; the edges include intramodal edges and cross-modal edges, which are used to represent the semantic correspondence between text feature representations and visual feature representations.

5. The multimodal named entity recognition method based on image-text semantic alignment and interaction as described in claim 4, characterized in that: In step S4, during the construction of the multimodal graph, a graph convolutional neural network is used for feature propagation to enhance the semantic representation of nodes; the construction of cross-modal edges is based on the association between noun phrases and visual objects, and is implemented through the Stanford parser and vision toolkit.

6. The multimodal named entity recognition method based on image-text semantic alignment and interaction as described in claim 1, characterized in that: In step S5, the Wasserstein distance is calculated using normalized mean square error as the distance metric, and the minimum transmission cost is solved through optimal transmission theory. The loss function is used to supervise mode alignment.

7. The multimodal named entity recognition method based on image-text semantic alignment and interaction as described in claim 1, characterized in that: In step S6, the adaptive joint attention mechanism includes: Adaptive attention unit: performs intramodal interaction on text feature representations or visual feature representations, and dynamically adjusts attention weights using a multi-head attention mechanism and adaptive factors; Adaptive Guided Attention Unit: Uses text feature representation as query and visual feature representation as key and value to perform cross-modal interaction and output attention-based visual feature representation; Multimodal feature generation unit: The text feature representation output by the adaptive attention unit is concatenated with the visual feature representation output by the adaptive guided attention unit to form the final fused multimodal feature representation.

8. The multimodal named entity recognition method based on image-text semantic alignment and interaction as described in claim 7, characterized in that: Both the adaptive attention unit and the adaptive guided attention unit include layer normalization, feedforward network and residual connection to enhance feature stability.

9. The multimodal named entity recognition method based on image-text semantic alignment and interaction as described in claim 1, characterized in that: In step S7, the conditional random field decoder is trained using maximum conditional likelihood estimation and infers the optimal label sequence using the Viterbi algorithm.

Images