Method and system for identifying multi-modal named entities

Abstract

The invention discloses a method and a system for identifying a multi-modal named entity, and belongs to the technical field of digital data processing. In order to solve the technical problems that in the prior art, when images and text information are processed, shared information and private information are sequentially connected in series, and the shared information and the private information of visual objects in the images are directly connected in series, so that feature information confusion is caused, fine-grained alignment in visual modes is influenced, and cross-modal understanding of a GMNER system is influenced. The shared visual features and the private visual features of the image are extracted respectively, the features of the visual objects in the image and the relation features between the visual objects are distinguished, and then the images are dynamically integrated and projected to the text embedding space, so that the corresponding relation between the visual object entities and the text entities is clearer, and the text embedding efficiency is improved. And the accuracy of fine granularity alignment is improved, so that the comprehensive cross-modal understanding capability of the GMNER system is improved. The method is mainly used for multi-modal named entity recognition.

Description

Technical Field

[0001] This invention relates to the field of digital data processing technology, and specifically to a method and system for multimodal named entity recognition. Background Technology

[0002] Grounded Multimodal Named Entity Recognition (GMNER) refers to the process of identifying and classifying named entities, such as people, places, and organizations, in text-text co-occurrence scenarios by simultaneously utilizing textual semantics and visual cues. An example of a GMNER task is shown below. Figure 3 As shown. The core challenge lies in how to accurately align the "visual objects" (such as faces, buildings, and goods) in an image with the "entity references" (such as "Zhang San," "Forbidden City," and "Apple Inc.") in the text, thereby improving the accuracy of entity recognition.

[0003] In recent years, mainstream solutions have followed a "sequential concatenation + unified encoding" framework: first, convolutional networks or Transformers are used to detect images and obtain several visual object features; then, these features are sequentially concatenated after the text sequence to form a cross-modal sequence, which is then encoded by a single Transformer, and finally, an attention mechanism is used to complete the "text-visual" alignment. This method has achieved certain improvements on public datasets and is considered a relatively mature technical approach.

[0004] Shared information includes contextual knowledge formed by the interaction of an object with other entities or background, such as "person A is in front of landmark B." Private information includes unique attributes of an object, such as color, shape, and texture. Existing methods do not distinguish between "visual object features" and "relationship features between visual objects" in an image, ignoring the distinction between shared and private information. Sequentially concatenating shared and private information dilutes the private information of visual objects and contaminates the shared information between visual objects with irrelevant private information. This concatenation and mixing causes feature information confusion, affecting fine-grained alignment within visual modalities. This ambiguity is then carried into subsequent cross-modal alignment stages, further exacerbating the inherent gaps between different modalities, impacting the comprehensive cross-modal understanding of the GMNER system, and limiting its performance. Summary of the Invention

[0005] To address the technical problem that existing technologies, when processing image and text information, fail to distinguish between the "features of the visual objects themselves" and the "features of the relationships between the visual objects," directly concatenating shared and private information of visual objects in the image, resulting in feature information confusion, affecting fine-grained alignment within visual modalities, and thus impacting the cross-modal understanding of the GMNER system, this invention provides a method and system for multimodal named entity recognition.

[0006] This invention provides a method for multimodal named entity recognition, comprising:

[0007] S1: Extract global image features, local object feature sets, and image features of the image to be processed; extract text sequence features of the text corresponding to the image to be processed; the image features include: depth features, position features, and RGB features;

[0008] S2: Generate private visual features using the local object feature set and image features; generate shared visual features using the global image features and private visual features; obtain comprehensive visual features using the private visual features and shared visual features; map the comprehensive visual features to the text embedding space to generate projected visual features;

[0009] S3: Generate a structured list of entity-type-object triples based on the text sequence features and projected visual features to complete named entity recognition.

[0010] The proprietary visual features generated by this invention actively isolate global contextual information in the image, focusing only on the object's own attributes, ensuring that the unique feature information of the visual object is not averaged or diluted by other objects or background information. The generated shared visual features comprehensively utilize the independent features of each visual object, the related features between visual objects, and the related features between visual objects and the background. This allows the shared visual features to focus on the relationships between objects and the interaction between objects and the environment. The dynamically integrated comprehensive visual features contain both the details of each visual object and the contextual information of the visual objects. Projecting the comprehensive visual features onto the text embedding space makes the correspondence between visual object entities and text entities clearer, thereby improving the accuracy of fine-grained alignment. Through the projection operation, visual features and text features are mapped to a unified dimension, reducing the inherent gap between modalities and thus enhancing the cross-modal understanding capability of the GMNER system.

[0011] Furthermore, the steps for generating a structured list of entity-type-object triples include:

[0012] The hidden features are obtained by concatenating the text sequence features and the projected visual features.

[0013] Based on the hidden features and the historical token sequence accumulated before the current decoding step, the GMNER text sequence is generated autoregressively.

[0014] Entities marked as "aligned" in the GMNER text sequence are denoted as candidate visual objects. The probability distribution is calculated based on the hidden features of the candidate visual objects, and the object with the highest probability is selected as the final visual object to be identified.

[0015] A list of entity-type-object triples is generated based on the GMNER text sequence and the final recognized visual object.

[0016] This invention obtains hidden features by concatenating text sequence features and projected visual features into the same sequence. When decoding the hidden features to generate GMNER text sequences, there is no need to switch back and forth between different features.

[0017] The generated GMNER text sequence contains the alignment relationship between visual objects in the image and entities described in the text. By converting this into structured entity-type-object triples, the information is structured, facilitating storage and retrieval, and improving the overall flexibility of the system. Furthermore, the step of generating private visual features includes:

[0018] The descriptive captions of the image features and the local object features are projected onto a unified dimension through a preset weight matrix to generate expert features.

[0019] Then, based on the expert features, the preset weight matrix, and the similarity scores between the expert features and the local object features, a fused visual cue is generated;

[0020] Then, the average value of the fused visual cue and the local object features is taken as the private visual feature.

[0021] The private visual feature extraction mechanism described in this invention achieves a unified approach to semantic enhancement, noise suppression, and preservation of individual features, ultimately generating discriminative private visual features that retain their specificity.

[0022] Furthermore, the step of generating shared visual features includes: generating a feature map based on the private visual features and the global image features; and then processing the feature map, private visual features, and global image features using a two-layer graph attention transformer network to obtain shared visual features.

[0023] The shared visual feature extraction mechanism described in this invention captures semantic interaction information between various visual objects, thereby improving the visual understanding capability of the GMNER system.

[0024] Furthermore, the step of generating the feature map includes:

[0025] A node set is constructed based on the aforementioned private visual features and global image features;

[0026] Edge weights are calculated based on the cosine similarity between nodes. If the edge weights exceed a set threshold, an edge is established between the corresponding nodes. Then, a feature map is generated based on the node set and the edges between the nodes.

[0027] This invention establishes edges using cosine similarity, which can filter out low-relevance connections, reduce computational load, and suppress noise propagation. By establishing supernodes, local nodes are calibrated by the supernodes during message passing, improving the accuracy of entity-visual object alignment.

[0028] Furthermore, the loss function used during the training phase introduces a fully aligned loss;

[0029] The complete alignment loss includes global alignment loss and local alignment loss;

[0030] The global alignment loss is a two-way contrast loss between the overall text and the overall image, including text-to-image loss and image-to-text loss;

[0031] The local alignment loss is a weakly supervised contrast loss between the entity span and the visual object.

[0032] The complete loss alignment introduced in this invention during the training phase achieves connection between image and text modalities at both global and local levels. At the global level, coarse-grained alignment is achieved by minimizing the difference between the whole image representation and the corresponding sentence embedding; at the local level, each object in the image is further precisely matched with named entities in the text, achieving cascaded alignment from the whole to the details. Furthermore, the expression for the complete alignment loss is:

[0033] ;

[0034] In the formula, This represents the loss due to perfect alignment. Indicates the image-to-text loss. Indicates text-to-image loss. This represents the image-to-text features in the projected visual features, where T represents the text features. This indicates the local alignment loss.

[0035] This invention achieves semantic mutual interpretability between visual objects and text entities through bidirectional alignment constraints of image-to-text loss and text-to-image loss; it also dynamically adjusts the weight of local alignment loss using cosine similarity, making the model pay more attention to semantically related image regions and text entities.

[0036] Another aspect of the present invention provides a system for multimodal named entity recognition, comprising:

[0037] Feature extraction module: used to extract global image features, local object feature sets, and image features of the image to be processed; used to extract text sequence features of the text corresponding to the image to be processed;

[0038] Feature fusion module: used to generate private visual features using the local object feature set and image features; generate shared visual features using the global image features and private visual features; obtain comprehensive visual features using the private visual features and shared visual features; map the comprehensive visual features to the text embedding space to generate projected visual features;

[0039] Encoding-decoding module: used to generate structured entity-type-object triples based on the text sequence features and projected visual features to complete named entity recognition;

[0040] The feature fusion module includes: a dual-branch network unit, a gated fusion unit, and a projection mapping unit.

[0041] Furthermore, the feature fusion module includes: a dual-branch network unit, a gated fusion unit, and a projection mapping unit;

[0042] The dual-branch network unit includes a private network branch and a shared network branch:

[0043] The private network branch is used to generate private visual features using the local object feature set and image features;

[0044] The private network branch is used to generate private visual features using the local object feature set and image features;

[0045] The shared network branch is used to generate shared visual features using the global image features and private visual features;

[0046] The gated fusion unit is used to obtain a comprehensive visual feature by utilizing the private visual features and the shared visual features;

[0047] The projection mapping unit is used to map the integrated visual features to the text embedding space to generate projected visual features.

[0048] Furthermore, the encoding-decoding module includes: a multimodal encoder unit, a sequence-to-sequence decoder unit, a visual object prediction unit, and a result extraction unit;

[0049] The multimodal encoder unit is used to concatenate the text sequence features and the projected visual features to obtain hidden features;

[0050] The sequence-to-sequence decoder unit is used to autoregressively generate a GMNER text sequence based on the hidden features and the historical token sequence accumulated before the current decoding step.

[0051] The visual object prediction unit is used to record entities marked as "aligned" in the GMNER text sequence as candidate visual objects, calculate the probability distribution based on the hidden vectors of the candidate visual objects, and select the object with the highest probability as the final recognized visual object.

[0052] The result extraction unit is used to generate a list of entity-type-object triples based on the GMNER text sequence and the final recognized visual object.

[0053] The beneficial effects of this invention are:

[0054] Compared to existing technologies that concatenate multiple visual objects in an image, ignoring contextual information between objects and unique features of each object, this invention extracts global image features, features of each visual object in the image, and image features separately. By integrating visual object features and image features, private visual features of visual objects are obtained, actively isolating global contextual information in the image and focusing only on the object's own attributes. This prevents the unique feature information of visual objects from being averaged or diluted by other objects or background information, enhancing the distinguishability between visual objects. A feature map is constructed using private visual features and global image features, and processed through a two-layer graph attention transformer network to obtain shared visual features. These shared visual features comprehensively utilize the independent features of each visual object, the correlation features between visual objects, and the correlation features between visual objects and the background. This allows the shared visual features to focus on the relationships between objects and the interaction between objects and the environment. Then, a gating mechanism dynamically integrates private and shared visual features. The integrated visual features contain both the details of each visual object and the contextual information of the visual objects. Projecting the integrated visual features into the text embedding space makes the correspondence between visual object entities and text entities clearer, thereby improving the accuracy of fine-grained alignment. Then, the integrated visual features are projected into a feature space of the same dimension as the text features. The visual features and text features are mapped to a unified dimension, and the inherent gap between modalities is reduced, thereby enhancing the cross-modal understanding capability of the GMNER system.

[0055] Meanwhile, the method of this invention introduces both global and local alignment losses in the loss function used during the training phase. The global alignment loss aims to minimize the gap between the global image representation and the corresponding text embedding; the local alignment loss aims to align individual objects in the image with their corresponding named entities in the text. Through contrastive learning, the similarity between semantically related cross-modal pairs is enhanced, while the distance between unrelated pairs is increased. Through these processes, visual object and entity representations are effectively integrated, capturing both direct and indirect connections between objects and entities, reducing modal differences at both the global and local levels, and enhancing cross-modal understanding consistency.

[0056] This invention projects visual features directly into the text embedding space in a pluggable manner, seamlessly integrating existing methods without the need for complex cross-modal interaction modules, thus balancing performance improvement and implementation simplicity. Attached Figure Description

[0057] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0058] Figure 1 This is a schematic diagram illustrating the execution steps of a method for multimodal named entity recognition.

[0059] Figure 2 This is a schematic diagram of the structure of a system for multimodal named entity recognition;

[0060] Figure 3 An illustrative example diagram for the GMNER task;

[0061] Figure 4 This is a schematic diagram of another system for multimodal named entity recognition.

[0062] Figure 5 This is a diagram showing the data content of the witter-FMNERG dataset. Detailed Implementation

[0063] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0064] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data used can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus. Specific Implementation Method 1

[0066] To facilitate understanding of the technical solution of this application, the following section will use image-text pairs in a social media environment as the processing object to introduce the entity naming recognition method provided in this application. The method execution steps are as follows: Figure 1 .

[0067] A set of image-text pairs consists of an image I and a descriptive text S, where the text S contains named entities to be identified, and the image I contains visual objects that may correspond to the named entities.

[0068] Step 1: Extract global image features, local object feature sets, and image features from the image to be processed I; extract text sequence features from the text S corresponding to the image to be processed.

[0069] When extracting features from image I and text S, the order of operations is not required.

[0070] 1. The processing procedure for text S includes:

[0071] The text S is encoded using a pre-trained Text-to-Text Transfer Transformer (T5) model to obtain a text sequence of length L. Each token It is the sum of word embedding and position embedding, simultaneously carrying semantic and positional information; This represents the dimension of the text sequence T.

[0072] The T5 model used in this embodiment employs a 768-dimensional embedding space, a maximum sentence length of 200, and uses 3-shot examples for prompt words, enabling the model to effectively understand the semantic context.

[0073] 2. The processing procedure for image I includes:

[0074] The entire image is processed using a Vision Transformer (ViT) to generate global image features. , The dimension representing the global image features.

[0075] The Visual Features in Vision-Language (VinVL) model is used to extract the top-ranked confidence values ​​from the image. For each visual object, mean pooling is performed on the object region to obtain a local object feature set. ,in Dimensions representing the features of a local object.

[0076] Descriptive captions are generated for the first K visual objects using an image captioning tool, denoted as . ; each subtitle The T5 encoder performs sequence mean pooling to obtain the corresponding attribute representations. , Indicates the dimension of the subtitle.

[0077] Extracting image features from image I ,in, Indicates location features, Representing depth features, Represents RGB characteristics.

[0078] The global image features, using the entire image as the receptive field, capture macro-level contextual information such as scene category and spatial layout. The local object feature set focuses on salient objects with high confidence rankings, capturing micro-level information such as texture and shape. The positional features in the image features explicitly preserve spatial coordinate information; the depth features in the image features alleviate the semantic gap between 2D and 3D; and the RGB features in the image features preserve original colors and textures, enhancing appearance consistency.

[0079] The joint extraction of global features, local features, and location-depth-RGB triple image features achieves complementarity and redundancy protection of multi-level, multi-modal information at the "scene-object-pixel" level. This provides a basic representation for subsequent feature fusion operations, thereby significantly improving the overall performance of the GMNER system.

[0080] Step 2: Generate projected visual features based on the global image features, the local object feature set, image features, and text sequence features.

[0081] (a) Generate private visual features using the aforementioned local object feature set and image features; the specific steps are as follows:

[0082] (1) The descriptive captions of the image features and the local object features are projected onto a unified dimension through a preset weight matrix to generate expert features.

[0083] Specifically, image features The subtitle attributes, including positional features, depth features, RGB features, and local object features, are weighted according to preset weights. Projecting onto a unified dimension yields expert features. , where i represents the category index, and p, d, r, and c correspond to position, depth, RGB, and subtitle attribute features, respectively.

[0084] (2) For local object feature sets The k-th local object feature The position features, depth features, RGB features, and subtitle attributes are calculated and compared with the local object features, respectively. similarity score Then based on similarity scores Weight matrix and expert characteristics Receive fusion visual cues Its expression is:

[0085] .

[0086] (3) Take the fused visual cues With the local object features The average value is used to obtain the private visual features. It is represented as:

[0087] .

[0088] By calculating the similarity between position, depth, RGB, and subtitles and local object features, features with high similarity will receive greater weight. By fusing the weight matrix and expert features, it is equivalent to a re-scoring, weakening the interference of background or irrelevant objects, suppressing noise, and highlighting the unique attributes of the object. Finally, the average value of the fused visual cues and the original local features is taken, which not only preserves the original unique information such as texture and shape, but also superimposes multi-source "enhanced signals", making the unique features of the object more significant in the vector space and enhancing the distinguishability between various visual objects. Through these three steps (1)-(3), the unity of semantic enhancement, noise suppression, and preservation of individual features is achieved, and finally, discriminable private visual features that maintain specificity are generated.

[0089] (ii) Generate shared visual features using the global image features, the local object feature set, and the image features; the specific steps are as follows:

[0090] (4) Based on the private visual features of all visual objects in the image and the global image features Generate feature map G; specific steps:

[0091] Based on the aforementioned private visual features Constructing nodes based on global image features Supernodes are constructed, and nodes together with supernodes form a node set. Then, the weights of the edges are calculated based on the cosine similarity between nodes. If the cosine similarity exceeds a set threshold, an edge is established between the corresponding nodes. Then, a feature map is formed based on the node set and the edges between the nodes. ,in Represents the set of edges between nodes. Indicates the node index.

[0092] The supernode is a node that treats global image features as a separate node in the feature map G. The supernode is connected to each node but does not participate in the construction of edges between nodes.

[0093] The feature graph G is composed of nodes and edges between nodes, and is a type of node feature graph.

[0094] (5) A two-layer graph attention transformer network is used to process the feature map G to obtain shared visual features. Its expression is as follows:

[0095] ;

[0096] In the formula, This represents the enhanced features of the first K local visual objects. Enhanced features representing global image features.

[0097] Shared visual features The features of each layer of nodes are denoted as Update it to The expression for iterative updates is:

[0098] ;

[0099] In the formula, Indicates the first The feature vector of each node after being updated by the l-th layer graph attention transformer. Indicates shared visual features The node index, They are adjacent nodes; Indicates the first Layer nodes eigenvectors, Indicates the first The learnable weight matrix of the layer, Represents a non-linear activation function. This represents the attention coefficient.

[0100] The attention coefficient The calculation method is as follows:

[0101] ;

[0102] In the formula, Indicates the first The learnable attention vector of the layer, Indicates splicing, Represents a node The neighborhood of.

[0103] (4) and (5) By constructing a fine-grained object feature map G, the semantic interaction information between various visual objects is captured, which improves the visual understanding ability of the GMNER system.

[0104] (III) Dynamically integrate the private visual features and the shared visual features to obtain comprehensive visual features; the specific steps are as follows:

[0105] (6) Dynamically integrate private visual features through gating mechanisms and shared visual features To obtain comprehensive visual features The expression for the integration process is as follows:

[0106] ;

[0107] ;

[0108] In the formula, For learnable weight matrix, Represents a non-linear activation function. This represents element-wise multiplication. Indicates dynamic control of private visual features and shared visual features The dynamic equilibrium parameter, T represents the text modality.

[0109] Private visual features are controlled through a gating mechanism. and shared visual features Dynamic fusion is not a simple addition or splicing, but rather fusion on demand. For different image needs, learnable weights are used to control the weights of private and shared visual features in the fusion result, generating a comprehensive visual feature that is detailed, contextual, and balanced.

[0110] Through the decoupling and recombination operations described above, the synthesized visual features are now a clearly structured representation that has been freed from internal confusion.

[0111] (iv) Map the comprehensive visual features to the text embedding space to generate projected visual features aligned with the text sequence features:

[0112] (7) Integrate visual features Projecting onto the same dimension as the text embedding generates projected visual features. The expression for the projection process is:

[0113] ;

[0114] ;

[0115] ;

[0116] In the formula, Indicates learnable parameters; Representing comprehensive visual features The representation after the first linear transformation, where, Represents the set of features of a local object The representation after the first linear transformation Representing global image features The representation after the first linear transformation; express The representation after nonlinear activation, where express The representation after nonlinear activation. express The representation after nonlinear activation.

[0117] Projected visual features obtained by projection The splicing structure of local object features and global image features is still maintained, and the hierarchical information of the projected visual features used for alignment is not lost.

[0118] The projection process shown in this embodiment is not a simple linear transformation, but a neural network that includes a layer of nonlinear activation function. This enables the model to learn the complex, nonlinear mapping relationship between the integrated visual feature space and the text feature space. Simple linear transformations cannot cope with the complex distribution differences between modalities; nonlinear transformations greatly enhance the model's expressive power, enabling it to "shape" the diverse integrated visual features into a form that more closely resembles the distribution of text features, thereby effectively narrowing the gap between different modalities.

[0119] By mapping the comprehensive visual features to the text space to obtain projected visual features, the correspondence between visual object entities and text entities becomes clearer, improving the accuracy of fine-grained alignment. Furthermore, because the comprehensive visual features include both shared semantics that the text can describe, such as the relationships between visual objects and the actions of visual objects, and private attributes of each visual object, the distribution of projected visual features and text features in space is closer, thus effectively narrowing the cross-modal gap.

[0120] Step 3: Generate a structured list of entity-type-object triples based on the text sequence features and projected visual features to complete named entity recognition; the specific steps include (8)-(11):

[0121] (8) The text sequence features and the projected visual features By splicing, the hidden features are obtained. The expression is:

[0122] .

[0123] The concatenated hidden feature H is a structured and information-complete carrier that contains textual semantic information, visual object detail information, and global scene context information.

[0124] The concatenation operation is effective because it assumes that the combined visual features have been projected into the text embedding space, resulting in projected visual features with the same dimension as the text features. Therefore, in subsequent calculations, there is no need to switch back and forth between different features; both the visual and text modalities are presented simultaneously in the same sequence. Any visual object token and any text token in this sequence can be directly computed. Although visual and text features reside in the same space, the model can distinguish the origin of each token in the sequence through positional encoding, thereby better understanding the meaning of different modal information.

[0125] (9) Based on the hidden feature H and the current decoding step Previously accumulated historical token sequences The GMNER text sequence SEQ is generated autoregressively.

[0126] Specifically: The model first starts the generation process with a start symbol; in each decoding step, the model calculates the token representation of the current decoding step based on the accumulated historical token sequence and hidden features, and generates the distribution of the next token based on the token representation of the current decoding step, determining the next token by taking the maximum probability; if the generated token is the end symbol or reaches the maximum length K, the generation is complete, otherwise the new token is appended to the historical sequence and the iteration continues until a complete formal entity-type-object GMNER text sequence SEQ is output.

[0127] The entity-type-object text sequence ,in [SEEP] represents a structured phrase, and [SEEP] represents a separator. In this embodiment, [SEEP] is used to represent the separator, but in actual applications, the separator format can be set arbitrarily.

[0128] The entity-type-object text sequence SEQ is obtained by transforming the entity-object-semantic state sequence GE.

[0129] The sequence , Let represent the k-th aligned named entity triple. The formal definition of this triple is... , Indicates entities mentioned in the text. Represents associated visual objects in an image. Indicates the semantic category of the entity. If the text mentions... If there is a corresponding visual object in the image, then It is assigned to that object; otherwise, Setting it to "None" indicates that there is no corresponding visual object.

[0130] when At that time, Transform into structured phrases is a and "which is in the image" This represents the idx-th entity. Indicates entity type, Indicates the alignment status.

[0131] when At that time, Transform into structured phrases is a and , which is not in the image".

[0132] Representing entities In an image, there is a corresponding visual object, that is, the entity is "aligned" with the visual object; Representing entities There is no corresponding visual object in the image.

[0133] For the Each decoding step, based on the current decoding step Previously accumulated historical token sequences And the hidden feature H, calculate the first Step token represents The expression is:

[0134] .

[0135] Based on the Step token represents , generate the first Probability distribution of tokens The expression is:

[0136] ,

[0137] In the formula, , Indicates learnable parameters, Indicates the generated first Tokens.

[0138] (10) The entities marked as "aligned" in the GMNER text sequence SEQ are designated as candidate visual objects. The probability distribution of the candidate visual objects is calculated based on their hidden features, and the object with the highest probability is selected as the final visual object to be identified. Specifically:

[0139] The entity marked "aligned" refers to the structured phrase " is a and The entity corresponding to "which is in the image". During the decoding phase, when the model processes the phrase "in the image", it treats it as a textual reference to an entity. This phrase consists of three consecutive lexical units, and the hidden feature representation corresponding to the phrase is extracted as an entity representation. Specifically, let the starting position index of the phrase be... The hidden feature sequence of these three lexical units is denoted as... , , , combined as This sequence is the representation of the entity.

[0140] Entity representation The arithmetic mean vector of the hidden features of the three words is obtained by aggregation using mean pooling. Its expression is:

[0141] .

[0142] Then, the arithmetic mean vector is used. The probability distribution of the first A candidate visual objects Its expression is:

[0143] ,

[0144] In the formula, Represents the features of the first A visual objects; denoted as a probability distribution; z represents a scalar, indicating the probability that an entity will align to each visual object; d represents the feature dimension.

[0145] Then select the probability distribution The visual object with the highest probability is selected as the final visual object for identification.

[0146] (11) Generate a list of entity-type-object triples based on the GMNER text sequence and the final recognized visual object, that is, match the entities and their categories that have been identified in the GMNER text sequence SEQ with the final recognized visual object in the image to form a list, and each entry in the list is an entity-type-object triple. Specific Implementation Method Two

[0148] refer to Figure 2 and Figure 4 The system for multimodal named entity recognition described in this embodiment includes:

[0149] Feature extraction module: used to extract global image features, local object feature sets, and image features of the image to be processed; used to extract text sequence features of the text corresponding to the image to be processed;

[0150] Feature fusion module: used to generate private visual features using the local object feature set and image features; generate shared visual features using the global image features and private visual features; obtain comprehensive visual features using the private visual features and shared visual features; map the comprehensive visual features to the text embedding space to generate projected visual features;

[0151] Encoding-decoding module: used to generate structured entity-type-object triples based on the text sequence features and projected visual features to complete named entity recognition.

[0152] The feature fusion module includes: a dual-branch network unit, a gated fusion unit, and a projection mapping unit;

[0153] The dual-branch network unit includes a private network branch and a shared network branch:

[0154] The private network branch is used to generate private visual features using the local object feature set and image features;

[0155] The private network branch is used to generate private visual features using the local object feature set and image features;

[0156] The shared network branch is used to generate shared visual features using the global image features and private visual features;

[0157] The gated fusion unit is used to obtain a comprehensive visual feature by utilizing the private visual features and the shared visual features.

[0158] The projection mapping unit is used to map the integrated visual features to the text embedding space to generate projected visual features.

[0159] The encoding-decoding module includes: a multimodal encoder unit, a sequence-to-sequence decoder unit, a visual object prediction unit, and a result extraction unit;

[0160] The multimodal encoder unit is used to concatenate the text sequence features and the projected visual features to obtain hidden features;

[0161] The sequence-to-sequence decoder unit is used to autoregressively generate a GMNER text sequence based on the hidden features and the historical token sequence accumulated before the current decoding step.

[0162] The visual object prediction unit is used to record entities marked as "aligned" in the GMNER text sequence as candidate visual objects, calculate the probability distribution based on the hidden features of the candidate visual objects, and select the object with the highest probability as the final visual object to be identified.

[0163] The result extraction unit is used to generate a list of entity-type-object triples based on the GMNER text sequence and the final recognized visual object.

[0164] This system projects visual features into the text embedding space and integrates seamlessly into existing systems in a plug-and-play manner. It aims to simultaneously model the shared and private information of visual objects, effectively bridging the gap between global and local modalities.

[0165] It should be noted that the system implementation and the method implementation are the same, and their execution process and execution principle are identical, so they will not be described in detail here. Specific Implementation Method 3

[0167] This implementation aims to explain in detail how to train the models involved in Implementation Methods 1 and 2. Its core lies in introducing an improved comprehensive objective function to optimize the entity alignment capability of the model during the training process.

[0168] During training, all module parameters are optimized by minimizing the comprehensive objective function. The comprehensive objective function of the model is... It consists of three parts, namely, the perfect alignment loss. Negative log-likelihood loss Object alignment loss The expression is:

[0169] .

[0170] (a) Process a text-image pair using the methods (a) to (b) in Specific Implementation Method 1 to obtain projected visual features. Then, a perfect alignment loss is introduced. This aims to bridge the modal gap between projected visual objects and text features.

[0171] The perfect alignment loss Including global alignment loss and local alignment loss The global alignment loss Including text-to-image loss Image-to-text loss Full alignment loss The expression is:

[0172] ;

[0173] In the formula, This represents the image-to-text feature in the projected visual features, where T represents the text feature.

[0174] During the process of fully aligned loss fusion, for instances within the current training batch (Bitch), if the global alignment loss... When the threshold is exceeded, the semantic relationship between the image and the text is determined to be weak, and thus the local alignment loss is suppressed. In perfect alignment loss The contribution of [the organization / entity].

[0175] The global alignment loss By constructing positive and negative samples using data within the training batch, the overall correspondence between images and text is learned by maximizing the similarity of matching image-text pairs and minimizing the similarity of non-matching text pairs.

[0176] The positive and negative samples are defined as follows: within the current training batch Bitch, for each image-text pair, if the image and text are associated, the pair is considered a positive sample; otherwise, it is considered a negative sample.

[0177] The determination of the association between an image and text is based on the calculation of the cosine similarity between the image and text representations to measure the degree of matching between them.

[0178] Text-to-image loss The definition, taking the m-th instance being processed as an example:

[0179] ;

[0180] Image-to-text loss The calculation formula is defined, taking the m-th instance as an example:

[0181] ;

[0182] In the above two formulas, This represents the text-to-image loss for the m-th instance. Let g represent the image-to-text loss for the m-th instance, and g represent global alignment. The text representation of the m-th instance is given, B represents the dataset used in the current training batch, and n represents each pair of data in dataset B.

[0183] When there are semantically related but lacking explicit correspondences between image and text pairs, this invention enhances the model's understanding of fine-grained semantics and robustness to multimodal representations by modeling the latent semantic associations between image regions and text fragments and aligning visual objects with named entities.

[0184] The local alignment loss , is a fine-grained contrast loss used to implicitly model the latent semantic associations between visual object regions in an image and named entity parts in text.

[0185] The local alignment loss The definition, taking the m-th instance being processed as an example:

[0186] ;

[0187] In the formula, Indicates local alignment loss; Represents the set of measurement objects; This represents any local object feature set in the current training batch dataset B; Represents the similarity function; Represented as a span set, This represents the span.

[0188] The span representation refers to the text labeled as named entities. A contiguous subset is denoted as the span set. .

[0189] Then, the text is averaged and pooled based on the span set Set to obtain the span set representation E.

[0190] The similarity function The definition of is:

[0191] ;

[0192] In the formula, Represents the span-object similarity matrix. For the dimensions of local object features, Let the span set represent the dimension. They are respectively , The index.

[0193] Elements in the similarity matrix A The calculation formula is:

[0194] ;

[0195] In the formula, Indicates the first Named entity span, Representing the Local object characteristics Indicates the first The result of projecting local object features into the text embedding space.

[0196] The above content describes how alignment loss simultaneously pulls together visual and textual modalities at both the global and local levels. At the global level, it directly aligns the representation of the entire image with its corresponding text; at the local level, it matches detected objects with named entities in the text one by one. Both levels employ a contrastive learning approach, where positive samples are brought closer and other samples are pushed away. Local alignment further introduces weakly supervised attention aggregation to integrate object-entity representations. Unlike the "relevant / irrelevant" binary assumption of global alignment, object-entity relationships exhibit various strengths, such as direct and indirect; through local alignment, the model is able to capture these more nuanced semantic connections.

[0197] (b) The object alignment loss , used to train the visual object prediction unit described in Implementation Method 1.

[0198] The visual object prediction unit minimizes the object alignment loss. During training, object alignment loss The expression is:

[0199] ;

[0200] In the formula, This represents the total number of all visual entities. This represents the total number of entities that can be aligned. Represents a scalar. This represents the probability distribution of visual objects obtained using the method described in Implementation Method 1. This indicates a monitoring signal.

[0201] The monitoring signal The IoU score is the intersection-over-union (IoU) score between the predicted bounding box of a visual object and the ground truth ground truth bounding box. A higher IoU score indicates a higher degree of overlap between the predicted and ground truth bounding boxes, suggesting a more "correct" prediction. Each visual object may correspond to one or more ground truth bounding boxes; therefore, the IoU score between each predicted visual object bounding box and all ground truth bounding boxes is calculated, resulting in a set of IoU scores. The highest value is selected as the final score. If the final score of the predicted visual object bounding box exceeds a set score threshold, it is retained; otherwise, it is reset to 0. The IoU scores are then normalized to obtain the object supervision signal. In this embodiment, the score threshold is 0.5.

[0202] By analyzing the probability distribution With monitoring signals The difference between the two is measured by calculating the divergence loss. The greater the difference, the greater the loss, and the more the model needs to be adjusted. By continuously minimizing the divergence loss, that is, reducing the error between the predicted result and the true label, the visual object prediction module is taught to correctly identify objects in the image.

[0203] (c) The negative log-likelihood loss The expression is:

[0204] ;

[0205] In the formula, This indicates the number of image-text pairs within the current training batch. = The length of the GMNER text sequence SEQ. This represents the text in the currently processed image-text pair. This represents the image in the currently processed image-text pair. This represents the historical token sequence that has been accumulated before the current decoding step. This represents the probability of the token calculated in the current decoding step. , and The calculation methods all use the corresponding method of Implementation Method 1, which will not be repeated in this implementation method.

[0206] In this embodiment, the important improvements of the present invention are described in detail. The remaining steps can be carried out using conventional methods, and will not be repeated in this embodiment.

[0207] To verify the beneficial effects of the present invention, the following experiments were conducted:

[0208] Experiment 1

[0209] This experiment used two multimodal datasets to validate the method of this invention. The multimodal datasets are Twitter-GMNER and Twitter-FMNERG.

[0210] The Twitter-GMNER dataset contains four entity types: Person (PER), Organization (ORG), Location (LOC), and Other (OTHER), for text-image pair tasks.

[0211] refer to Figure 5 The Twitter-FMNERG dataset expands upon the Twitter-GMNER dataset, including 8 coarse-grained and 51 fine-grained entity types. Coarse-grained entities are still dominated by PER and ORG; in fine-grained entities, Athlete, SportsTeam, Musician, Politician, and Actor are the most common, while Software, Website, and Account account only account for 1%, showing an imbalance in entity type distribution.

[0212] In terms of image-text alignment, PER-type entities (such as athletes, musicians, and politicians) are easier to align with images; other types such as cities, sports leagues, and companies are usually difficult to align.

[0213] The invention was evaluated on the two multimodal datasets, with precision, recall, and F1 score used as evaluation metrics. The experimental results are shown in Table 1. The method of this invention achieved an F1 score of 59.41% on Twitter-GMNER and 50.0% on Twitter-FMNERG, representing an average improvement of approximately 3 percentage points over the baseline, thus validating the beneficial effects of the invention.

[0214] The single-modal and multimodal baseline methods include:

[0215] BERT-CRF-None: A BERT-CRF model based on sequence labeling that predicts all visual objects as the majority class None.

[0216] T5-Paraphrase-None: A text-based TIGER variant that models named entity recognition as a T5 paraphrase generation task.

[0217] UMT-VinVL-EVG, UMGF-VinVL-EVG, and ITA-VinVL-EVG: Three multimodal named entity recognition models based on the entity-aware visual alignment (EVG) framework.

[0218] H-Index: A BART-based multimodal model that uses a pointer mechanism to model GMNER as a sequence generation problem.

[0219] Align: A pre-trained variant of H-Index and ITA-VinVL-EVG that enhances generalization capabilities using external datasets.

[0220] TIGER: A generative model based on T5 that transforms span-type-object triples into target paraphrasing sequences.

[0221] MQSPN: A model that models GMNER as an ensemble prediction task.

[0222] To ensure a fair comparison, the experiment excluded methods that used large language models or knowledge enhancement techniques.

[0223] The experimental results are shown in Table 1. The results demonstrate that multimodal methods, by integrating image and text information, significantly outperform unimodal methods that rely solely on text (such as BERT-CRF-None and T5-Paraphrase-None), highlighting the importance of visual information in improving entity recognition accuracy and recall. Among multimodal methods, the text-image pipeline method, as an extension of unimodal methods, outperforms unimodal models but is susceptible to error propagation. In contrast, ensemble prediction (such as MQSPN) and sequence generation (such as TIGER) methods directly output GMNER triples, achieving higher metrics on both coarse-grained and fine-grained datasets. The method of this invention, based on the T5 skeleton, outperforms the core baseline TIGER by nearly 3 percentage points, attributed to its generative model effectively extracting structured information from multimodal data and accurately identifying triples in the GMNER task.

[0224] Compared to baseline methods focused on decoding strategies, Align enhances image-text understanding through image-text contrastive pre-training, but fails to address the issues of insufficient visual modeling and global-local modality gap. This invention effectively solves these problems through shared-private connections and global-local alignment. The shared network enhances visual contextual understanding by modeling semantic interactions between objects through fine-grained object graphs; the private network improves object discrimination by dynamically fusing position, depth, RGB, and caption attribute features; and global-local alignment bridges the modality gap by optimizing cross-modal alignment through contrastive learning.

[0225] Table 1 shows the experimental results on the two datasets.

[0226]

[0227] Experiment 2

[0228] Ablation studies were conducted on the Twitter-GMNER and Twitter-FMNERG datasets to evaluate the impact of each module and unit on performance metrics (precision, recall, and F1 score). Experiments were conducted to construct the following variants by removing specific modules or units, while maintaining consistent parameter settings to ensure fair comparisons:

[0229] Ours-wo-SPC: Removes the dual-branch network unit (SPC), which includes two sub-variants: Ours-wo-SC (removes the shared network sub-unit) and Ours-wo-PC (removes the private network sub-unit).

[0230] Ours-wo-CA: Remove cross-modal alignment (CMA), which includes two sub-variants: Ours-wo-GA (remove global alignment) and Ours-wo-LA (remove local alignment).

[0231] The specific experimental results are shown in Table 2. The results indicate that removing the two-branch network unit (SPC) leads to a significant performance decrease. On the Twitter-GMNER dataset, the F1 score decreased from 59.51% to 57.59%; on the Twitter-FMNERG dataset, the F1 score decreased from 50.0% to 48.26%. This demonstrates that the SPC plays a crucial role in modeling the shared and private information of objects, and is essential for achieving effective cross-modal alignment. Further analysis of subvariants shows that after removing the shared network subunit (SC), the F1 score for Twitter-GMNER is 59.34%, and for Twitter-FMNERG it is 49.56%; after removing the private network subunit (PC), the scores are 58.63% and 48.87%, respectively. Compared to removing the entire SPC, removing either the SC or PC individually results in a smaller performance decrease, indicating that both the shared and private network subunits significantly contribute to improving GMNER performance, jointly enhancing object discriminative ability and semantic interaction modeling.

[0232] Removing cross-modal alignment (CMA) resulted in a slight performance decrease, with the F1 score dropping to 58.38% on Twitter-GMNER and 49.18% on Twitter-FMNERG, highlighting the importance of CMA in bridging the gap between global and local modalities. Further removal of global alignment (GMA) or local alignment (LMA) led to a continued performance decline: on Twitter-GMNER, the F1 scores after removing GMA and LMA were 59.75% and 59.33%, respectively; on Twitter-FMNERG, they were 49.94% and 49.47%, respectively. This indicates that both global and local alignment positively contribute to overall performance, improving multimodal understanding capabilities by optimizing the semantic consistency between images and sentences and between objects and entities, respectively.

[0233] Table 2 Ablation experiments of key components of the framework

[0234]

[0235] Finally, it should be noted that in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one"... does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0236] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0237] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for multimodal named entity recognition, characterized in that, include: S1: Extract global image features, local object feature sets, and image features of the image to be processed; Extract the text sequence features of the text corresponding to the image to be processed; The image features include: depth features, position features, and RGB features; S2: Generate private visual features using the local object feature set and image features; generate shared visual features using the global image features and private visual features; obtain comprehensive visual features using the private visual features and shared visual features; map the comprehensive visual features to the text embedding space to generate projected visual features; S3: Generate a structured list of entity-type-object triples based on the text sequence features and projected visual features to complete named entity recognition.

2. The method for multimodal named entity recognition according to claim 1, characterized in that, The steps to generate a structured list of entity-type-object triples include: The hidden features are obtained by concatenating the text sequence features and the projected visual features. Based on the hidden features and the historical token sequence accumulated before the current decoding step, the GMNER text sequence is generated autoregressively. Entities marked as "aligned" in the GMNER text sequence are denoted as candidate visual objects. The probability distribution is calculated based on the hidden features of the candidate visual objects, and the object with the highest probability is selected as the final visual object to be identified. A list of entity-type-object triples is generated based on the GMNER text sequence and the final recognized visual object.

3. The method for multimodal named entity recognition according to claim 1, characterized in that, The steps for generating private visual features include: The descriptive captions of the image features and the local object features are projected onto a unified dimension through a preset weight matrix to generate expert features. Then, based on the expert features, the preset weight matrix, and the similarity scores between the expert features and the local object features, a fused visual cue is generated; Then, the average value of the fused visual cue and the local object features is taken as the private visual feature.

4. The method for multimodal named entity recognition according to claim 3, characterized in that, The step of generating shared visual features includes: generating a feature map based on the private visual features and the global image features; and then using a two-layer graph attention transformer network to process the feature map, the private visual features, and the global image features to obtain shared visual features.

5. The method for multimodal named entity recognition according to claim 4, characterized in that, The steps for generating the feature map include: A node set is constructed based on the aforementioned private visual features and global image features; Edge weights are calculated based on the cosine similarity between nodes. If the edge weights exceed a set threshold, an edge is established between the corresponding nodes. Then, a feature map is generated based on the node set and the edges between the nodes.

6. The method for multimodal named entity recognition according to claim 1, characterized in that, The loss function used during the training phase introduces a fully aligned loss. The complete alignment loss includes global alignment loss and local alignment loss; The global alignment loss is a two-way contrast loss between the overall text and the overall image, including text-to-image loss and image-to-text loss; The local alignment loss is a weakly supervised contrast loss between the entity span and the visual object.

7. The method for multimodal named entity recognition according to claim 6, characterized in that, The expression for the perfect alignment loss is: ； In the formula, This represents the loss due to perfect alignment. Indicates the image-to-text loss. Indicates text-to-image loss. This represents the image-to-text features in the projected visual features, where T represents the text features. This indicates the local alignment loss.

8. A system for multimodal named entity recognition, characterized in that, include: Feature extraction module: used to extract global image features, local object feature sets, and image features of the image to be processed; Used to extract text sequence features corresponding to the text in the image to be processed; Feature fusion module: used to generate private visual features using the local object feature set and image features; generate shared visual features using the global image features and private visual features; obtain comprehensive visual features using the private visual features and shared visual features; map the comprehensive visual features to the text embedding space to generate projected visual features; Encoding-decoding module: used to generate structured entity-type-object triples based on the text sequence features and projected visual features to complete named entity recognition; The feature fusion module includes: a dual-branch network unit, a gated fusion unit, and a projection mapping unit.

9. A system for multimodal named entity recognition according to claim 8, characterized in that, The feature fusion module includes: a dual-branch network unit, a gated fusion unit, and a projection mapping unit; The dual-branch network unit includes a private network branch and a shared network branch: The private network branch is used to generate private visual features using the local object feature set and image features; The shared network branch is used to generate shared visual features using the global image features and private visual features; The gated fusion unit is used to obtain a comprehensive visual feature by utilizing the private visual features and the shared visual features; The projection mapping unit is used to map the integrated visual features to the text embedding space to generate projected visual features.

10. A system for multimodal named entity recognition according to claim 9, characterized in that, The encoding-decoding module includes: a multimodal encoder unit, a sequence-to-sequence decoder unit, a visual object prediction unit, and a result extraction unit; The multimodal encoder unit is used to concatenate the text sequence features and the projected visual features to obtain hidden features; The sequence-to-sequence decoder unit is used to autoregressively generate a GMNER text sequence based on the hidden features and the historical token sequence accumulated before the current decoding step. The visual object prediction unit is used to record entities marked as "aligned" in the GMNER text sequence as candidate visual objects, calculate the probability distribution based on the hidden vectors of the candidate visual objects, and select the object with the highest probability as the final visual object to be recognized. The result extraction unit is used to generate a list of entity-type-object triples based on the GMNER text sequence and the final recognized visual object.

Images