A multi-modal named entity recognition method and system for multi-image scenes

Abstract

This invention discloses a multimodal named entity recognition method and system for multi-image scenes. The method includes: S1. Performing object detection on the input text sequence and several associated images to obtain a set of candidate objects in each image; S2. Extracting text features from the text sequence and object features from each candidate object; S3. Performing relevance clustering based on the similarity of object features, and selecting cross-image consistent objects related to named entities from the clustering results; S4. Inputting the text features and consistent object features into a dynamic fusion network, dynamically adjusting the modal weights through a cross-modal attention mechanism to generate a multimodal fusion representation; S5. Decoding the multimodal fusion representation to obtain the named entity recognition result. This invention solves the problems of noise interference and static modal weight allocation in traditional methods in multi-image scenes.

Description

Technical Field

[0001] This invention relates to the fields of natural language processing and multimodal computing, and specifically to a multimodal named entity recognition method and system for multi-image scenarios. Background Technology

[0002] Named entity recognition (NER) is a crucial task in information extraction, and its performance directly impacts the effectiveness of various downstream applications, including relation extraction and entity linking. With the rapid development of natural language processing technology, NER has been widely applied to various text corpora, such as news articles, encyclopedic content, and social media platforms. However, due to stylistic differences between different text types, NER presents specific challenges in various contexts.

[0003] Non-return authentication (NER) in the social media domain is particularly challenging. On the one hand, social media text is typically short, loosely structured, and riddled with non-standard language phenomena such as spelling errors, abbreviations, slang, and emojis, significantly increasing the difficulty of NER tasks. On the other hand, social media content carries a large amount of real-time information and plays a crucial role in multiple fields, such as natural disaster early warning, cyberattack detection, and emergency monitoring. Therefore, improving the performance of NER in the social media domain has significant practical implications.

[0004] Existing methods often neglect the semantic alignment issue between images and text, a problem that is particularly prominent in noisy social media environments, where irrelevant images can introduce interference. To address this issue, recent research has begun to more explicitly model the alignment relationships between text and images.

[0005] To address these issues, recent research has begun to more explicitly model the alignment relationships between text and images. For example, multi-scale fusion and improved robustness are achieved by dynamically aggregating hierarchical image features across different BERT layers. Alternatively, image features are used as cues to guide the text encoder, achieving excellent results on datasets containing single images. However, these existing methods are only applicable to single-image inputs. Their performance remains limited when dealing with complex scenarios involving multiple images, making it difficult to fully realize their advantages.

[0006] Research on multi-image MNER is still in its early stages due to the limited availability of datasets. Existing datasets, including TWITTER-2015, TWITTER-2017, and WikiDiverse, primarily consist of single images. To address this gap, MNER-MI was introduced, the first dataset specifically designed for multi-image MNER, where each instance contains 2 to 4 images. The introduction of this dataset has laid the foundation for advancing multi-image MNER methods.

[0007] Driven by the MNER-MI dataset, current research attempts to handle multi-image MNER by extending methods designed for single-image processing. Early models employed a simple multi-image feature concatenation approach, where features extracted from each image were average-pooled using ResNet and then fused with textual information. However, this fusion strategy often exacerbates noise interference. Recent research has begun to explore modeling relationships between images. For example, multiple images are treated as frames in a video, modeling the sequential dependencies between them. However, this approach neglects the semantically discontinuous correlations between images. Summary of the Invention

[0008] To address the technical problems mentioned above, this invention introduces a Consistent Object Dynamic Fusion Network (CODF). This invention designs a consistent object detection module to extract object information consistent with named entities. This module applies a similarity-based clustering method to identify semantically consistent objects that consistently appear across multiple images, while filtering out low-quality or semantically irrelevant objects. Furthermore, we propose a dynamic fusion network that progressively optimizes the attention weights for each modality by leveraging the interaction of unimodal features and multimodal representations from different network blocks, thereby achieving more effective multimodal feature integration.

[0009] To achieve the above objectives, this invention provides a multimodal named entity recognition method for multi-image scenes, comprising the following steps:

[0010] S1. Perform object detection on the input text sequence and several associated images to obtain a set of candidate objects in each image;

[0011] S2. Extract the text features of the text sequence and extract the object features of each candidate object;

[0012] S3. Based on the similarity of the object features, perform relevance clustering, and filter cross-image consistent objects related to named entities from the clustering results;

[0013] S4. Input the text features and the consistency object features into a dynamic fusion network, dynamically adjust the modal weights through a cross-modal attention mechanism, and generate a multimodal fusion representation;

[0014] S5. Decode the multimodal fusion representation to obtain the named entity recognition result.

[0015] Preferably, S1 includes: using a DETR-ResNet-50 model to detect objects in each image, and setting a detection confidence threshold to filter low-confidence candidate objects.

[0016] Preferably, S3 includes:

[0017] Calculate the cosine similarity between the feature vectors of all candidate objects, and perform correlation clustering according to the preset number of clusters K;

[0018] Calculate the central feature vector of each cluster, and select the candidate object with the highest similarity to the cluster center as the cross-image consistency object of that cluster.

[0019] Preferably, before S4 is executed, the method further includes: after concatenating the image features with the consistent object features, standardizing the concatenated features and text features respectively through a projection layer to make their dimensions consistent.

[0020] Preferably, in step S4, the dynamic fusion network comprises stacked dynamic fusion networks; each layer of dynamic fusion block performs the following operations:

[0021] Calculate the similarity between text features and image features based on a cross-modal attention mechanism;

[0022] The modality weights are dynamically updated based on similarity.

[0023] The multimodal representation of the current layer is generated by weighted fusion and then input into the next layer.

[0024] Preferably, step S5 includes: using a Conditional Random Field (CRF) decoder to perform sequence labeling on the multimodal fusion representation and predict the entity category of each word in the text sequence.

[0025] The present invention also provides a multimodal named entity recognition system for multi-image scenes. The system is used to implement the above method and includes: a detection module, an extraction module, a clustering module, a generation module, and a recognition module.

[0026] The detection module is used to perform object detection on the input text sequence and several associated images to obtain a set of candidate objects in each image.

[0027] The extraction module is used to extract the text features of the text sequence and the object features of each candidate object;

[0028] The clustering module is used to perform relevance clustering based on the similarity of the object features, and to filter cross-image consistent objects related to named entities from the clustering results;

[0029] The generation module is used to input the text features and the consistency object features into a dynamic fusion network, and dynamically adjust the modal weights through a cross-modal attention mechanism to generate a multimodal fusion representation;

[0030] The recognition module is used to obtain the named entity recognition result based on the multimodal fusion representation decoding.

[0031] Preferably, the workflow of the detection module includes: using the DETR-ResNet-50 model to detect objects in each image, and setting a detection confidence threshold to filter low-confidence candidate objects.

[0032] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0033] This invention proposes a multi-image, multimodal named entity recognition (MNER) method based on a Consistent Object Dynamic Fusion Network (CODF), primarily addressing the noise interference and static modality weight allocation issues inherent in traditional methods in multi-image scenarios. First, an innovative consistent object detection method extracts objects related to named entities from multiple images. Image feature similarity and relevance clustering effectively filters irrelevant objects, thereby enhancing the semantic alignment between image information and text. Second, this invention designs a dynamic fusion network that automatically adjusts the fusion weights of text and image modalities based on the specific circumstances of the samples, avoiding the limitation of assigning the same weights to all modalities in traditional methods, thus improving the model's flexibility and robustness. By introducing a cross-modal attention mechanism and a weight balancing module, the modality information fusion process is further optimized, enabling the model to dynamically focus on the most relevant modality information, reducing redundancy and noise impact. Finally, a Conditional Random Field (CRF) decoder is used for sequence labeling, improving the accuracy of entity recognition. Attached Figure Description

[0034] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1 The following are schematic diagrams of two cases from the MNER-MI dataset in this embodiment of the invention; wherein, (a) represents a human case; and (b) represents an animal case.

[0036] Figure 2 This is a schematic diagram of the network structure according to an embodiment of the present invention;

[0037] Figure 3 This is a schematic diagram of the dynamic fusion block framework according to an embodiment of the present invention. Detailed Implementation

[0038] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0039] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0040] As the background technology shows, multiple images provide more contextual information. However, fusing image content without filtering can lead to information redundancy and even misleading predictions. This problem is particularly pronounced when multiple images are involved; unfiltered noise may degrade performance rather than aid entity recognition.

[0041] However, existing methods typically perform simple averaging of images or assume that each image contributes equally, failing to fully utilize the information of consistent objects. For example, in Figure 1 In (a), the recurring appearance of characters across multiple images enhances the recognition of entities such as "Selena" and "Harry Styles," while irrelevant objects in multiple images, such as watches, cameras, and ornaments, may interfere with recognition accuracy. Therefore, to more effectively represent information from multiple images, it is necessary to integrate consistent object information that is strongly correlated with named entities and recurring across multiple images, while suppressing redundant information, in order to improve the accuracy of named entity recognition.

[0042] Current multimodal fusion methods typically use static approaches to assign weights between images and text. However, the correlation between text and images varies across different samples. Inappropriate weight assignment can cause the model to ignore important modal information or amplify irrelevant noise. For example, in... Figure 1 In (b), traditional methods typically assign equal weights to text and images. This approach limits the effective integration of information and may incorrectly classify the entity "Phidippus adumbrates" as a "human" entity due to the pronoun "she" in the text. However, the "spider" object in the image has a strong correlation with the entity "Phidippus adumbrates," suggesting that increasing the weight of image features may improve the accuracy of entity recognition. Therefore, dynamically adjusting modal weights based on different examples is crucial for more effective multimodal fusion.

[0043] Example 1

[0044] This embodiment provides a multimodal named entity recognition method for multi-image scenes, the steps of which include:

[0045] S1. Perform object detection on the input text sequence and several associated images to obtain a set of candidate objects in each image.

[0046] This embodiment defines the task of multimodal named entity recognition for multi-image scenarios as follows: Given a text S = {s1, s2, ..., s...} n} and its associated image set I = {I1, I2, ..., I m As input, where n represents the number of words in the text and m represents the number of images, the goal of the MNER task is to extract named entities from text a and assign them to one of the predefined entity categories. Similar to the traditional MNER method, this embodiment formulates this task as a sequence labeling problem. Let S = {s1, s2, ..., s} n Let} be the input word sequence, Y = {y1, y2, ..., y3}. n} represents the corresponding label sequence, where each y i ∈Y and Y is a predefined set of labels, using the BIO2 annotation scheme.

[0047] To extract more entity-related object information from multiple images while minimizing image noise from irrelevant objects, we propose a clustering method based on image feature similarity and relevance to select consistent objects suitable for multi-image scenarios. Specifically, this embodiment uses DETR-ResNet-50 to detect objects in multiple images. A detection confidence threshold of 0.9 is set to filter low-confidence objects and minimize noise interference.

[0048] S2. Extract text features from the text sequence and extract object features from each candidate object.

[0049] Each detected object is then processed by ViT (Visual Transformer) to generate p image patches, which are then linearly embedded to generate a representation {z1, z2, ..., z} for each patch. p A learnable special marker [CLS] is added to the beginning of these patches, forming the sequence {[CLS], z1, z2, ..., z}. p In this embodiment, the activation value of the [CLS] flag in the last layer of ViT is used to obtain the representation of the i-th object, denoted as o. i ∈R do This method obtains the representation of g objects O = {o1, o2, ..., o...} g}∈R do×g , where d oThe dimension of the object representation is represented by g, and the number of objects detected is g.

[0050] S3. Perform correlation clustering based on the similarity of object features, and filter cross-image consistent objects related to named entities from the clustering results.

[0051] Cosine similarity is used to quantify the strength of relationships between objects and correlation clustering is performed. Based on a predefined number of clusters K, each object is assigned a corresponding cluster label. Within each cluster, the object closest to the cluster center is selected as the consistent object. The cluster center is calculated by averaging the feature vectors of all objects within the cluster.

[0052]

[0053] Where k∈[1,K], c k Indicates clustering C k The center, |C k | represents cluster C k The number of objects in the middle, o i Indicates clustering C k The characteristics of each object are defined. The cosine similarity between each object and its cluster center is calculated, and the object most similar to the cluster center is selected as the consistent object for that cluster. If the number of clusters K is less than a predetermined value, the number of consistent objects is padded to K using a zero vector.

[0054]

[0055] Where k * It is the index of the object most similar to the cluster center. Ultimately, the consistent object representations of all clusters form a set.

[0056] S4. Input text features and consistent object features into a dynamic fusion network, and dynamically adjust modal weights through a cross-modal attention mechanism to generate a multimodal fusion representation.

[0057] To enhance multimodal fusion and better capture cross-modal interactions, this embodiment stacks L layers of dynamic fusion blocks. Since one or more modalities in the text or image may dominate during the fusion process, this embodiment first represents the text modality H... t H' and image modality representation v Perform an addition operation. The result of this addition serves as the input to the first dynamic fusion module.

[0058]

[0059] Multiple dynamic fusion blocks are stacked together to form a pipeline, such as Figure 2 As shown. Output of the previous layer. Single-mode representation H′ u (where u∈{t,v}) and the balancing weight α are used as inputs to the next layer module.

[0060] After calculation, the output is obtained.

[0061]

[0062] in like Figure 3 As shown, DFB is a dynamic fusion block that adaptively adjusts the weights of each modality. In this module, we first introduce a cross-modal attention mechanism (CAttn), which progressively identifies the dominant modality and evaluates the similarity between text and images. The attention query (Q) specifies the location of attention; therefore, we treat multimodal features as Q, unimodal features as K and V, and apply layer normalization (LN) for processing.

[0063]

[0064] Next, in this embodiment, the balance weight α is multiplied by the intermediate representation. Then, we will compare the obtained representation with the multimodal representation of the input. Add them together to fine-tune the contribution of each modality:

[0065]

[0066] S5. Decode the multimodal fusion representation to obtain the named entity recognition result.

[0067] final, The input is fed into a multi-head attention and feedforward neural network to generate the output of a dynamically fused block. This output is used to execute MNER.

[0068] Example 2

[0069] This invention significantly improves the accuracy and robustness of Named Entity Recognition (MNER) in the multi-image multimodal named entity recognition (MNER) task by introducing a Consistent Object Dynamic Fusion Network (CODF). In experimental evaluation, this embodiment uses two publicly available benchmark datasets: MNER-MI and MNER-MI-Plus datasets, both derived from social media content and containing multiple images and their corresponding text.

[0070] To comprehensively evaluate the effectiveness of the method of this invention, this embodiment uses three common evaluation metrics: precision (P), recall (R), and F1 score (F1). Experimental results show that, compared with existing multimodal named entity recognition methods, the CODF network of this invention performs excellently in multi-image scenarios, especially in reducing image noise interference and dynamically adjusting modality weights, significantly improving recognition performance. The experimental results are shown in Figure 1.

[0071] Table 1

[0072]

[0073] As shown in Table 1, in the experiments on the MNER-MI dataset, the method of this invention achieved an F1 score of 79.27, which is a significant improvement over traditional single-image and simple multi-image methods. On the MNER-MI-Plus dataset, the F1 score further improved to 83.50, indicating that our model can effectively handle complex multimodal inputs and fully mine the deep semantic information between text and images.

[0074] Furthermore, the method of this invention demonstrates superior capabilities in eliminating image noise and enhancing the recognition accuracy of named entities. Compared to traditional static modality fusion methods, the strategy of dynamically adjusting modality weights allows the model to flexibly adjust the image and text fusion strategy according to different input instances, thereby improving the effective utilization of cross-modal information. Through the consistent object detection module, we can accurately filter out objects related to named entities, reduce interference from irrelevant objects, and further improve the accuracy of entity recognition.

[0075] To verify the contribution of each module to the model performance in this invention, ablation experiments were conducted in this embodiment, removing the Consistent Object Detection (COD) module, the Dynamic Fusion Network (DFN) module, and the Conditional Random Field Decoder (CRF) module respectively. As shown in Table 2, by comparing the experimental results, we can evaluate the impact of each module on the accuracy of named entity recognition.

[0076] Table 2

[0077]

[0078] With the Consistent Object Detection (CODF) module removed (without COD), the model does not perform the consistent object extraction process but directly uses objects from all images. Experimental results show that removing the consistent object detection module reduces the F1 score by 1.37% and 1.24% on the MNER-MI and MNER-MI-Plus datasets, respectively. This result demonstrates that consistent object detection is crucial for filtering noisy objects and extracting objects relevant to named entities.

[0079] When the dynamic fusion network module is removed (CODF-w / o DFN), the model no longer dynamically adjusts the modality weights, but instead adopts a static weight allocation strategy. Experimental results show that the F1 score decreased by 1.01% and 0.75%, respectively. This indicates that the dynamic fusion network can flexibly adjust the contributions between modalities when processing multimodal data, improving the model's recognition performance in complex multi-image environments.

[0080] Finally, with the Conditional Random Field decoder removed (CODF-w / o CRF), we replaced the CRF decoder with a simple Softmax classifier. Experimental results show that removing the CRF reduced the F1 score by 0.41% and 0.45%, respectively. This result demonstrates that the CRF decoder can effectively model dependencies between labels, improving the accuracy of sequence labeling and entity recognition.

[0081] The combined ablation experiment results demonstrate that each module plays an indispensable role in this invention. Removing any module leads to a performance degradation, proving the effectiveness of the components of the proposed technical solution in improving the accuracy of named entity recognition.

[0082] To demonstrate CODF's capabilities more intuitively, we provide three typical cases, as shown in Table 3.

[0083] Table 3

[0084]

[0085] In Case 1 and Case 2, traditional methods, by introducing images without differentiation, often introduce noise. However, our proposed method, by selecting consistent objects from multiple images, allows the model to focus on the main object information. Therefore, CODF correctly identifies "Hoseoks" as a "PER" entity and completely identifies "Sonyejin" as a "PER" entity. In Table 3, "√" indicates correct identification, and "×" indicates incorrect identification.

[0086] In Case 3, the text "Today is the day Zophie comes back" and the two cats in the image provide crucial clues. The prominent cat features in the image make the image modality essential for accurately identifying the entity "Zophie". Traditional multimodal fusion methods typically assign equal weights to text and images without dynamically adjusting the contribution of each modality based on contextual differences between modalities. However, our model, by dynamically enhancing the weights of the image modality, accurately identifies "Zophie" as a "Miscellaneous" (MISC) entity. This approach ensures that the contribution of the image modality in multimodal named entity recognition is fully utilized, thereby significantly improving the accuracy of entity recognition.

[0087] Example 3

[0088] This embodiment also provides a multimodal named entity recognition system for multi-image scenarios, including: a detection module, an extraction module, a clustering module, a generation module, and a recognition module; the detection module is used to perform object detection on the input text sequence and several associated images to obtain a set of candidate objects in each image; the extraction module is used to extract text features from the text sequence and object features of each candidate object; the clustering module is used to perform relevance clustering based on the similarity of object features and to filter cross-image consistent objects related to named entities from the clustering results; the generation module is used to input text features and consistent object features into a dynamic fusion network, dynamically adjust modal weights through a cross-modal attention mechanism, and generate a multimodal fusion representation; the recognition module is used to decode the multimodal fusion representation to obtain the named entity recognition result.

[0089] The detection module's workflow includes: using the DETR-ResNet-50 model to detect objects in each image, and setting a detection confidence threshold to filter low-confidence candidate objects.

[0090] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A multimodal named entity recognition method for multi-image scenes, characterized by the following steps: include: S1. Perform object detection on the input text sequence and several associated images to obtain a set of candidate objects in each image; S2. Extract the text features of the text sequence and extract the object features of each candidate object; S3. Based on the similarity of the object features, perform relevance clustering, and filter cross-image consistency objects related to named entities from the clustering results; S3 includes: calculating the cosine similarity between the feature vectors of all candidate objects, performing relevance clustering according to a preset number of clusters K; calculating the central feature vector of each cluster, and selecting the candidate object with the highest similarity to the cluster center as the cross-image consistency object of that cluster, including: Where k ∈ [1,K], c k Indicates clustering C k The center, |C k | represents cluster C k The number of objects in the middle, o i Indicates clustering C k The characteristics of each object are defined; by calculating the cosine similarity between each object and the cluster center, the object most similar to the cluster center is selected as the consistent object of the cluster; if the number of clusters K is less than a predetermined value, the number of consistent objects is filled up to K using zero vectors; in It is an index of the object most similar to the cluster center; ultimately, the consistent object representations of all clusters form a set. ; S4. Input the text features and consistent object features into a dynamic fusion network, dynamically adjust the modal weights through a cross-modal attention mechanism, and generate a multimodal fusion representation; the dynamic fusion network includes stacked dynamic fusion networks; each dynamic fusion block performs the following operations: calculate the similarity between text features and image features based on the cross-modal attention mechanism; dynamically update the modal weights according to the similarity; generate the multimodal representation of the current layer through weighted fusion and input it into the next layer; specific steps include: First, let's discuss the text modal representation H. t H' and image modality representation v Perform an addition operation; the result of this addition serves as the input to the first dynamic fusion module. ; Multiple dynamic fusion blocks are stacked together to form a pipeline; the output of the upper layer Single-modal representation Where u∈{t,v} and the balancing weight α are used as inputs to the next layer module; after calculation, the output is obtained. : in DFB stands for Dynamic Fusion Block. In this module, a cross-modal attention mechanism is introduced, treating multimodal features as Q and single-modal features as K and V, and applying layer normalization for processing. ； Then the balancing weight α is multiplied by the intermediate representation The obtained representation and the input multimodal representation are then combined. Add them together to fine-tune the contribution of each modality: ； S5. Decode the multimodal fusion representation to obtain the named entity recognition result.

2. The multimodal named entity recognition method for multi-image scenes according to claim 1, characterized in that, S1 includes: using the DETR-ResNet-50 model to detect objects in each image, and setting a detection confidence threshold to filter low-confidence candidate objects.

3. The multimodal named entity recognition method for multi-image scenes according to claim 1, characterized in that, Before S4 is executed, the method further includes: after concatenating the image features with the consistent object features, the concatenated features and text features are standardized through a projection layer to make their dimensions consistent.

4. The multimodal named entity recognition method for multi-image scenes according to claim 1, characterized in that, S5 includes: using a Conditional Random Field (CRF) decoder to perform sequence labeling on the multimodal fusion representation and predicting the entity category of each word in the text sequence.

5. A multimodal named entity recognition system for multi-image scenes, the system being used to implement the method described in any one of claims 1-4, characterized in that, include: Detection module, extraction module, clustering module, generation module, and recognition module; The detection module is used to perform object detection on the input text sequence and several associated images to obtain a set of candidate objects in each image. The extraction module is used to extract the text features of the text sequence and the object features of each candidate object; The clustering module is used to perform relevance clustering based on the similarity of the object features, and to filter cross-image consistent objects related to named entities from the clustering results; The generation module is used to input the text features and the consistency object features into a dynamic fusion network, and dynamically adjust the modal weights through a cross-modal attention mechanism to generate a multimodal fusion representation; The recognition module is used to obtain the named entity recognition result based on the multimodal fusion representation decoding.

6. The multimodal named entity recognition system for multi-image scenes according to claim 5, characterized in that, The workflow of the detection module includes: using the DETR-ResNet-50 model to detect objects in each image, and setting a detection confidence threshold to filter low-confidence candidate objects.

Images