A multi-modal named entity recognition method and apparatus

Abstract

This invention discloses a multimodal named entity recognition method and apparatus, relating to the field of named entity recognition technology. The method includes: inputting the text to be recognized and the image to be recognized into a trained multimodal named entity recognition model to determine the entity recognition result. The multimodal named entity recognition model includes a text extraction module, an image extraction module, a sentence-level soft gating module, a self-attention Transformer module, a cross-modal Transformer module, an MLP expert head, a visual gating module, and a CRF decoder. Based on this scheme, information loss in cross-modal interactions is reduced, and fine-grained alignment and decoding robustness are improved, thereby enhancing entity recognition performance.

Description

Technical Field

[0001] This invention relates to the field of named entity recognition technology, and in particular to a multimodal named entity recognition method and apparatus. Background Technology

[0002] With the prevalence of mixed text and image content on social media, multimodal named entity recognition (MNER) aims to identify and extract named entities by integrating modal information such as text and images. Traditional MNER methods typically extract text and image features separately, then perform simple concatenation or encoder-level fusion, and finally output named entity labels through a sequence labeler. However, this approach is prone to problems such as insufficient modal interaction, large alignment errors, and image noise interference, resulting in poor entity recognition performance. Summary of the Invention

[0003] This invention provides a multimodal named entity recognition method and apparatus, which solves the technical problem of poor entity recognition performance in existing multimodal named entity recognition methods.

[0004] The first aspect of this invention provides a multimodal named entity recognition method, comprising:

[0005] Input the text and image to be recognized into the trained multimodal named entity recognition model;

[0006] The word-level feature matrix of the text to be identified is extracted by the text extraction module, and the visual feature matrix of the image to be identified is extracted by the image extraction module.

[0007] The self-attention Transformer module is used to determine the main task text features of the word-level feature matrix, and after performing attention operation on the word-level feature matrix, a linear mapping is performed to output the auxiliary emission score;

[0008] Based on the sentence-level soft gating module, gating weights are generated according to the cosine similarity between the word-level feature matrix and the visual feature matrix, and the visual feature matrix is ​​weighted to determine the visual region features.

[0009] According to the cross-modal Transformer module, the main task text features and the visual region features are cross-attention processed to generate corresponding region alignment features and visual perception features. The main task text features are used to guide the interaction of the visual perception features to construct context-aware features.

[0010] After inputting the region alignment features and the context-aware features into the MLP expert head, the gating weights are used to perform weighted linear fusion to generate multimodal fusion features.

[0011] The visual gating module concatenates the region alignment features and the context-aware features to generate gating coefficients, and then weights the region alignment features to generate gated region alignment features.

[0012] The context-aware features, the multimodal fusion features, and the gated region alignment features are used to concatenate and output an enhanced fusion feature, which is then linearly mapped to the main emission score.

[0013] The main emission score and the auxiliary emission score are fused to output a fused emission score, and the entity recognition result is determined by the CRF decoder based on the fused emission score.

[0014] Optionally, the step of extracting the word-level feature matrix of the text to be identified through the text extraction module includes:

[0015] After adding [CLS] and [SEP] tags to the beginning and end of the text to be identified, the text is input into the RoBERTa pre-trained model, which outputs sub-word vectors.

[0016] After the sub-word vectors are randomly deactivated, repeated filtering is performed according to the first character alignment strategy to form a word-level feature matrix.

[0017] Optionally, the step of using an image extraction module to extract the visual feature matrix of the image to be identified includes:

[0018] The image to be identified is segmented using the SAM model to generate a region mask.

[0019] Each region mask is input into the CLIP model, and the corresponding mask embedding is output.

[0020] Averaging operations are performed on each of the mask embeddings to construct the corresponding region vectors;

[0021] A visual feature matrix is ​​formed by mapping the region vectors using a linear layer.

[0022] Optionally, the sentence-level soft gating module generates gating weights based on the cosine similarity between the word-level feature matrix and the visual feature matrix, and weights the visual feature matrix to determine visual region features, including:

[0023] Extract the [CLS] vector from the word-level feature matrix as a text semantic summary;

[0024] The visual feature matrix is ​​averaged to determine the overall visual image features;

[0025] Calculate the cosine similarity between the text semantic summary and the visual whole image features;

[0026] Gated weights are generated based on the cosine similarity using a sigmoid function with a learnable temperature coefficient and bias threshold.

[0027] The gating weights are broadcast by channel and multiplied with the visual feature matrix to determine the visual region features.

[0028] Optionally, the training process of the trained multimodal named entity recognition model includes:

[0029] For any associated training text and training image within the current training batch in the training set, the training word-level feature matrix, training visual feature matrix, training main task text features, and training auxiliary emission score are determined through the multimodal named entity recognition model to be trained.

[0030] The cosine similarity between the training word-level feature matrix and the training visual feature matrix is ​​standardized within the batch using the multimodal named entity recognition model to be trained, and then gating weights are generated. The training visual feature matrix is ​​then weighted to determine the training visual region features.

[0031] The training fusion emission score is determined by the multimodal named entity recognition model to be trained based on the training visual region features, the training visual region features, and the training auxiliary emission score.

[0032] The auxiliary mission path energy score of the training-assisted launch score is determined using a CRF decoder.

[0033] Calculate the CRF joint loss based on the inverse class frequency weights of the training set, using the training fusion launch score and the auxiliary mission path energy score.

[0034] By minimizing the CRF joint loss, the model parameters of the multimodal named entity recognition model to be trained are iteratively optimized until a well-trained multimodal named entity recognition model is determined.

[0035] Optionally, the CRF joint loss includes:

[0036] ;

[0037] In the formula, Indicates the joint loss of CRF. Indicates batch size, Indicates the training sample index. This represents a logarithmic function with the natural constant as its base. Indicates the first Training fusion emission score of each training sample Indicates the first The main task true label sequence of each training sample. This represents any sequence of predicted labels for the main task within the main task label space. Indicates the main task label space. Indicates the first The main task path energy score of each training sample under the true label sequence of the main task. Indicates the first The main task path energy score of each training sample under any main task predicted label sequence. Represents the natural constant. Indicates the effective length of the sentence. This represents the token time step index in the sentence. Indicates the first The training sample at the th ... The inverse class frequency weights corresponding to the true labels of the main task at each time step. This represents the weight coefficient for auxiliary tasks. This represents any sequence of auxiliary task prediction boundary labels in the auxiliary task label space. Represents the auxiliary task label space. Indicates the first The auxiliary launch score of each training sample. Indicates the first Auxiliary task ground boundary label sequence for each training sample. Indicates the first The auxiliary task path energy score of each training sample under any auxiliary task prediction boundary label sequence. Indicates the first The auxiliary task path energy score of each training sample under the true boundary label sequence of the auxiliary task.

[0038] A second aspect of the present invention provides a multimodal named entity recognition device, comprising:

[0039] The data input module is used to input the text and image to be recognized into the trained multimodal named entity recognition model;

[0040] The data extraction module is used to extract the word-level feature matrix of the text to be identified through the text extraction module, and to extract the visual feature matrix of the image to be identified by the image extraction module.

[0041] The self-attention processing module is used to determine the main task text features of the word-level feature matrix by employing the self-attention Transformer module, and to output the auxiliary emission score by linear mapping after performing attention operations on the word-level feature matrix.

[0042] The sentence-level enhancement module is used to generate gate weights based on the cosine similarity between the word-level feature matrix and the visual feature matrix, and to weight the visual feature matrix to determine the visual region features.

[0043] The cross-modal interaction module is used to perform cross-attention processing on the main task text features and the visual region features according to the cross-modal Transformer module, generate corresponding region alignment features and visual perception features, and use the main task text features to guide the interaction of the visual perception features to construct context-aware features.

[0044] The expert fusion module is used to input the region alignment features and the context-aware features into the MLP expert head respectively, and then perform weighted linear fusion using the gated weights to generate multimodal fusion features;

[0045] The visual enhancement module is used to generate gating coefficients by concatenating the region alignment features and the context-aware features through the visual gating module, and to generate gated region alignment features by weighting the region alignment features.

[0046] The scoring determination module is used to concatenate the context-aware features, the multimodal fusion features, and the gated region alignment features to output enhanced fusion features, and linearly map them to the main emission score;

[0047] The entity recognition module is used to fuse the main transmission score and the auxiliary transmission score to output a fused transmission score, and to determine the entity recognition result based on the fused transmission score by a CRF decoder.

[0048] A computer device provided in a third aspect of the present invention includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the multimodal named entity recognition method as described in any of the preceding claims.

[0049] The fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed, implements the multimodal named entity recognition method as described in any of the preceding claims.

[0050] The fifth aspect of the present invention provides a computer program product comprising a computer program / instruction, wherein the computer program / instruction, when executed by a processor, implements the multimodal named entity recognition method as described in any of the preceding claims.

[0051] As can be seen from the above technical solutions, the present invention has the following advantages:

[0052] The above-described solution of the present invention provides a multimodal named entity recognition method, comprising: inputting the text to be recognized and the image to be recognized into a trained multimodal named entity recognition model; extracting the word-level feature matrix of the text to be recognized through a text extraction module, and extracting the visual feature matrix of the image to be recognized through an image extraction module; using a self-attention Transformer module to determine the main task text features of the word-level feature matrix, and linearly mapping the word-level feature matrix after performing attention operations to output an auxiliary emission score; generating gating weights based on the cosine similarity between the word-level feature matrix and the visual feature matrix according to a sentence-level soft gating module, and weighting the visual feature matrix to determine the visual region features; and using a cross-modal Transformer module to combine the main task text features with the visual region features. The system employs cross-attention processing to generate region alignment features and visual perception features. The main task text features guide the interaction of visual perception features to construct context-aware features. Region alignment features and context-aware features are input into the MLP expert head and then weighted linearly fused using gated weights to generate multimodal fusion features. A visual gating module concatenates region alignment features and context-aware features to generate gating coefficients, and weights the region alignment features to generate gated region alignment features. The context-aware features, multimodal fusion features, and gated region alignment features are concatenated to output enhanced fusion features, which are linearly mapped to the main emission score. The main emission score and auxiliary emission score are fused to output a fused emission score, which is then decoded using a CRF decoder to determine the entity recognition result. Based on this scheme, fine-grained alignment is performed at the instance level, and information reconstruction is performed after cross-modal attention to retain key clues. Adaptive suppression of visual noise based on sentence-level image-text relevance is used while forcibly preserving the core semantics of the text. This reduces information loss during cross-modal interaction, improves the robustness of fine-grained alignment and decoding, and thus enhances entity recognition performance. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 This is a flowchart of the steps of a multimodal named entity recognition method provided in Embodiment 1 of the present invention;

[0055] Figure 2 This is an architecture diagram of a multimodal named entity recognition model provided in Embodiment 1 of the present invention;

[0056] Figure 3This is a structural block diagram of a multimodal named entity recognition device provided in Embodiment 2 of the present invention. Detailed Implementation

[0057] This invention provides a multimodal named entity recognition method and apparatus to solve the technical problem of poor entity recognition performance in existing multimodal named entity recognition methods.

[0058] To make the objectives, features, and advantages of this invention more apparent and understandable, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0059] Please see Figure 1 The present invention provides a multimodal named entity recognition method, comprising:

[0060] Step 101: Input the text to be recognized and the image to be recognized into the trained multimodal named entity recognition model.

[0061] It should be noted that, for the multimodal named entity recognition task, this embodiment uses a UMT architecture to build a system such as... Figure 2 The multimodal named entity recognition model shown includes a text extraction module, an image extraction module, a sentence-level soft gating module, a self-attention Transformer module, a cross-modal Transformer module, an MLP expert head, a visual gating module, and a CRF decoder. After training the model, a trained multimodal named entity recognition model is obtained, and feature processing is performed on the associated received text and image to be recognized input to the model.

[0062] Step 102: Extract the word-level feature matrix of the text to be identified using the text extraction module, and extract the visual feature matrix of the image to be identified using the image extraction module.

[0063] In one specific embodiment of this example, step 102 includes the following sub-steps:

[0064] After adding [CLS] and [SEP] tags to the beginning and end of the text to be recognized, input it into the RoBERTa pre-trained model and output sub-word vectors;

[0065] After randomly deactivating the sub-word vectors, repeated filtering is performed according to the first character alignment strategy to form a word-level feature matrix;

[0066] The SAM model is used to perform instance mask segmentation on the image to be recognized, generating a region mask.

[0067] Input the masks of each region into the CLIP model, and then output the corresponding mask embeddings.

[0068] Averaging is performed on each mask embedding to construct the corresponding region vector;

[0069] A visual feature matrix is ​​formed by mapping the region vectors using a linear layer.

[0070] It should be noted that in this embodiment, modal feature learning is performed through the text extraction module and the image extraction module to obtain the word-level feature matrix and the visual feature matrix. The matrix rows of the word-level feature matrix represent the sub-word context vector of each effective sub-word after alignment and filtering, and the matrix rows of the visual feature matrix represent the visual semantic vector of each instance.

[0071] The text extraction module includes a RoBERTa pre-trained model, a Dropout layer, and a word repetition filtering layer. Specifically, for the text to be recognized: to obtain context-sensitive word vectors, this embodiment uses the RoBERTa pre-trained model as the text encoder. RoBERTa is unsupervised pre-trained on a large-scale corpus and has excellent cross-domain language understanding capabilities, providing rich semantic cues for multimodal named entity recognition tasks. During the preprocessing of the text to be recognized, two special tags, start tag [CLS] and end tag [SEP], are added to the beginning and end of the text sequence, respectively, thereby obtaining... , This indicates the text to be recognized with special tags added. This indicates the [CLS] tag. This indicates the [SEP] tag. This indicates the number of tokens in the text to be recognized, helping the model better understand the contextual structure of the sequence. Inputting RoBERTa yields the context vector for each subword, i.e. , Represents sub-word vectors, Indicates the number of subwords. This refers to the RoBERTa pre-trained model. This represents the RoBERTa pre-training parameters; to enhance the model's generalization ability and suppress overfitting, this embodiment uses a Dropout layer to process the output features. A random dropout operation is applied; then, in the label alignment stage, for the segmented sub-words, the first-character alignment strategy is adopted through the sub-word filtering layer to repeatedly filter them out. That is, logically, only the sub-word vector of the first sub-word is used to represent the semantics of the complete sub-word, thereby ignoring the prediction results of subsequent sub-words in conjunction with the masking mechanism, thus ensuring the consistency between word-level features and named entity labels.

[0072] To extract visual semantic features that are highly discriminative for entity recognition tasks from images, this embodiment designs a visual feature extraction mechanism based on instance regions. The image extraction module includes a SAM model, a CLIP model, a mean operation layer, and a linear layer. Specifically, for the image to be recognized: First, the unsupervised image segmentation model SegmentAnything (SAM) is used to perform instance mask segmentation on the input image to be recognized. This can be understood as segmenting each object in the image and generating an independent pixel-level mask, thereby obtaining... Area Mask , Represents a set of region masks. Indicates the region mask index. Indicates the first A region mask, This represents the number of region masks; subsequently, each region mask is fed into the CLIP model, resulting in a mask embedding of 768-dimensional patch embeddings; furthermore, the original region vector is obtained by averaging all embeddings within the same mask using a mean operation layer. , , Indicates the first The region vector of a region mask. Indicates the first Mask embedding of a region mask, This indicates that the patch is embedded in the index. Indicates the first Each patch is embedded; and to align with the textual latent space, a trainable linear mapping is introduced through a linear layer to map the region vectors: , Indicates the first Visual semantic vectors of region masks Represents the visual weight matrix. This represents the visual bias vector, and ultimately all visual semantic vectors are stacked to form a visual feature matrix. Preferably, the mask prediction confidence scores can be stacked in the order of synchronous output from the SAM model.

[0073] Step 103: Using the self-attention Transformer module, determine the main task text features of the word-level feature matrix, and after performing attention operations on the word-level feature matrix, output the auxiliary emission score via linear mapping.

[0074] It should be noted that the self-attention Transformer module in this embodiment specifically includes a main task self-attention Transformer module and an auxiliary task self-attention Transformer module, both of which include a multi-head self-attention layer, a residual and normalization unit, and a feedforward neural network layer. The semantic context features required by the main task and the auxiliary task are obtained by using two independent self-attention Transformer modules respectively. , , Indicates the textual features of the main task. Represents the word-level feature matrix. This indicates the main task's self-attention Transformer module. Indicate textual features for auxiliary tasks, This represents the auxiliary task self-attention Transformer module; and the auxiliary task text features are mapped to the auxiliary label space through a linear layer to obtain the auxiliary emission score, which is then transferred to the main task for fusion.

[0075] Step 104: Based on the sentence-level soft gating module, generate gating weights according to the cosine similarity between the word-level feature matrix and the visual feature matrix, and weight the visual feature matrix to determine the visual region features.

[0076] In one specific embodiment of this example, step 104 includes the following sub-steps:

[0077] Extract the [CLS] vector from the word-level feature matrix as a semantic summary of the text;

[0078] The visual feature matrix is ​​averaged to determine the overall visual image features.

[0079] Calculate the cosine similarity between the text semantic summary and the visual whole image features;

[0080] Gated weights are generated based on cosine similarity using a sigmoid function with a learnable temperature coefficient and bias threshold.

[0081] The gating weights are broadcast by channel and multiplied with the visual feature matrix to determine the visual region features.

[0082] It should be noted that, to avoid forcibly injecting visual noise in scenarios with extremely low image-text relevance, this embodiment introduces sentence-level soft gating before cross-modal interaction. This allows for adaptive suppression of irrelevant visual noise at the global level without the need for manual thresholding. The sentence-level soft gating module includes a text summarization extraction module, a mean calculation layer, a similarity calculation layer, a sigmoid function, and a weighted fusion layer.

[0083] First, extract the [CLS] vector from the word-level feature matrix. As a textual semantic summary of the entire sentence, the visual end performs a comprehensive analysis of all elements in the visual feature matrix. The visual semantic vectors of each region are averaged to obtain the overall visual image features. ;

[0084] Next, the cosine similarity between the two is calculated. : , Indicates the inner product. Represents the L2 norm;

[0085] Subsequently, the cosine similarity is input into a sigmoid function with a learnable temperature coefficient and a bias threshold to generate gated weights: , Indicates the gating weight, This represents the Sigmoid function. Indicates the learnable temperature coefficient. Indicates the bias threshold;

[0086] Finally, the gating weights are broadcast channel by channel and multiplied onto all visual semantic vectors: When the sentence-graph correlation is low ( ) When the signal approaches zero, visual information is significantly weakened; when the correlation is high, the complete visual signal is preserved.

[0087] Step 105: Based on the cross-modal Transformer module, the main task text features and visual region features are cross-attention processed to generate corresponding region alignment features and visual perception features. The main task text features are used to guide the interaction of visual perception features to construct context-aware features.

[0088] It should be noted that the cross-modal Transformer module aims to achieve feature interaction between different modalities, thereby fusing effective information from each other. The cross-modal Transformer module includes a multi-head cross-attention layer, residual and normalization units, and a feedforward neural network layer:

[0089] First, through the text-to-image cross-modal Transformer module (text-to-image attention), ), based on the main task text features Mapped to query matrix Visual region features Mapped to a key matrix Sum matrix After cross-attention operations, feature processing is performed using residual and normalization units and a feedforward neural network to achieve dynamic selection and weighted fusion of visual region features in the text, generating an image-aware text representation, i.e., region alignment features. Similarly, to further enhance the ability of visual features to perceive text semantics, a cross-modal Transformer module for image-to-text attention is introduced. At this point, the visual region feature projection is used as the query matrix, and the main task text feature projection is used as the key matrix and value matrix, thus outputting the visual representation of text perception, i.e., the visual perception features: ;

[0090] After the aforementioned two steps of cross-modal interaction, both textual and visual features have incorporated effective information from the other modality. However, they still emphasize different modal perspectives. Therefore, this embodiment considers performing another text-to-text cross-modal Transformer module (text-to-text attention) with the main task text features as the query matrix and visual perception features as the key and value matrices. This allows for more detailed integration of bimodal information to obtain context-aware features. .

[0091] Step 106: After inputting the region alignment features and context-aware features into the MLP expert head, weighted linear fusion is performed using gated weights to generate multimodal fusion features.

[0092] It should be noted that after multiple cross-modal attention interactions, two complementary representations were obtained in the text modality direction, one of which is derived from... Reflux and through Context-aware features obtained by aggregation The first emphasizes text-driven global semantics and contextual consistency, and the second is based on... The generated region alignment features This approach focuses on fine-grained matching cues with instance regions. For these two branches containing rich semantic information, this embodiment sets up independent MLP expert heads for reprojection and compression, resulting in two high-level features. The two expert head networks have the same structure but independent parameters, both including linear layers, ReLU non-linear activation functions, and Dropout layers. , , This represents the input features of the MLP expert head. Indicates MLP expert head, This represents the expert head linear transformation weight matrix. This indicates the expert head bias term. Represents the ReLU nonlinear activation function. If we represent the Dropout layer, then the two outputs are denoted as follows: and , This indicates context-aware high-level features. This indicates a region-aligned MLP expert header, while the other indicates a context-aware MLP expert header. Indicates high-level features of region alignment ;

[0093] Subsequently, gating weights are generated at the sentence level based on image-text similarity gating. Multimodal fusion features are obtained by dynamically weighting and fusing the two outputs through a weighted fusion layer. : By quantifying the consistency between text and image at the sentence level through gating weights, a controllable balance is achieved between semantic dominance and regional details. This allows for adaptive reduction of the influence of region alignment branches in samples with low relevance and amplification of their contribution in samples with strong complementarity. Based on this weight-driven fusion, a significant F1 score improvement is achieved with similar computational complexity, indicating that the two expert pathways have a stable complementary effect at the information emphasis level.

[0094] Step 107: Using the visual gating module, the region alignment features and context-aware features are concatenated to generate gating coefficients, and the region alignment features are weighted to generate gating region alignment features.

[0095] It should be noted that, to suppress misaligned regions and low-confidence visual noise, this embodiment uses a visual gating module to perform token-by-token and dimension-by-dimensional gating on the two representations after cross-modal attention, under the premise of semantic dominance. This adaptively selects "when and in which dimensions" to introduce visual evidence, reducing the interference of noisy regions on sequence labeling, thereby obtaining gated region alignment features. The visual gating module includes a splicing layer, a linear layer, a sigmoid function, and an enhanced fusion layer.

[0096] Record No. The context-aware vector of each token is Its aligned first The region alignment vector of each token is After concatenating the two, the gating coefficients are obtained through linear transformation and Sigmoid operation: , Indicates the gating coefficient. Represents the weight matrix of the gated linear transformation. Indicates the gating bias term. This represents the Sigmoid function. Indicates the sequence length. Representing feature dimension Furthermore, the region alignment features are suppressed dimension-wise using gating coefficients: , This indicates the alignment characteristics of the gated region. This indicates Hadamard multiplication.

[0097] Step 108: Employ context-aware features, multimodal fusion features, and gated region alignment features to concatenate and output enhanced fusion features, and linearly map them to the main emission score.

[0098] It should be noted that context-aware features Multimodal fusion features Alignment features with gated regions By splicing layers in the feature dimension: , Indicates enhanced fusion features, This represents splicing; let the hidden feature sequence that enhances the fusion features be... , This indicates the actual number of tokens (PADs have been masked). Indicates the first One token, Representing feature dimension Through linear layers Mapping this to the main task's label space yields the main task's main emission score, which is then used for subsequent CRF decoding. , , Indicates the main launch score. This represents the linear transformation weight matrix of the main task. Indicates the main task bias. This represents the main task's tag space.

[0099] Step 109: Fuse the main emission score and the auxiliary emission score to output the fused emission score, and use the CRF decoder to decode based on the fused emission score to determine the entity recognition result.

[0100] It should be noted that, in order to mitigate the entity boundary ambiguity problem that may be introduced by the visual modality, the model will train an auxiliary entity span detection (ESD) branch in parallel. This branch only predicts simplified BIO labels (i.e., it only focuses on "whether it is an entity" and its boundaries, without distinguishing the specific type), and defines a prior transformation matrix. Used to describe the probabilistic mapping relationship from auxiliary labels (such as 'B') to main task labels (such as 'B-PER', 'B-LOC'). This represents the feature dimension of the auxiliary task. Indicates the main task feature dimension: , , Indicates the conversion score. Indicates the score of the fusion launch. The hyperparameter representing the balance between the contributions of primary and secondary features (e.g., 0.5) is then used as input in the fused emission score for decoding in the CRF decoder, specifically, in conjunction with the label transition matrix. (in Indicates from the label Move to label The probabilities of these probabilities form a first-order linear chain CRF, which is used for a given decoded input feature. and predicted label sequence Its path energy score is , Indicates the path energy score. Indicates token index , Indicates the actual number of tokens. Indicates the first One predicted label, Indicates the first One predicted label, Indicates from predicted labels Transfer to predictive labels The transfer score, The score representing the fusion launch score is... Each token was predicted as a label. The launch score is determined by using the Viterbi algorithm based on... The highest-scoring label sequence is decoded as the entity recognition result.

[0101] In one specific implementation of this embodiment, the training process of the trained multimodal named entity recognition model includes:

[0102] For any associated training text and training image within the current training batch in the training set, the training word-level feature matrix, training visual feature matrix, training main task text features, and training auxiliary emission score are determined through the multimodal named entity recognition model to be trained.

[0103] The cosine similarity between the training word-level feature matrix and the training visual feature matrix is ​​standardized within the batch using the multimodal named entity recognition model to be trained, and then gating weights are generated. The training visual feature matrix is ​​then weighted to determine the training visual region features.

[0104] The training fusion emission score is determined by using the multimodal named entity recognition model to be trained based on the training visual region features, the training visual region features, and the training auxiliary emission score.

[0105] The auxiliary mission path energy score is determined using a CRF decoder to train the auxiliary launch score.

[0106] Calculate the CRF joint loss based on the training fusion launch score and the auxiliary mission path energy score;

[0107] By minimizing the CRF joint loss, the model parameters of the multimodal named entity recognition model to be trained are iteratively optimized until the trained multimodal named entity recognition model is determined.

[0108] It should be noted that the feature processing during model training is similar to steps 101-108 mentioned above. Only a brief overview of the general process is provided here; the specific determination process for each feature can be understood by referring to the relevant content in the preceding steps, and will not be elaborated upon here. Two main optimization processes are proposed during model training:

[0109] On the one hand, in order to eliminate the training instability caused by the difference in sample similarity distribution between different training batches and to prevent the subsequent Sigmoid activation function from entering the saturation region, this embodiment modifies the original cosine similarity within the batch. After standardization (e.g., Z-score normalization), gated weights are generated using standardized cosine similarity.

[0110] ;

[0111] In the formula, Represents the standardized cosine similarity. This represents the mean of the cosine similarity within a batch. The standard deviation of cosine similarity within a batch. Represents the minimum value to prevent division by zero errors (e.g.) );

[0112] On the other hand, due to the highly uneven distribution of entity categories in social media data, i.e., the long-tail distribution problem, entity boundary determination is prone to jitter. Even when using sequence labelers (such as CRF), insufficient recall is likely to occur. This embodiment considers designing a sentence-level reweighted CRF joint loss based on inverse category frequency weights, including:

[0113] ;

[0114] In the formula, Indicates the joint loss of CRF. Indicates batch size, Indicates the training sample index. This represents a logarithmic function with the natural constant as its base. Indicates the first Training fusion emission score of each training sample Indicates the first The main task true label sequence of each training sample. This represents any sequence of predicted labels for the main task within the main task label space. Indicates the main task label space. Indicates the first The main task path energy score of each training sample under the true label sequence of the main task. Indicates the first The main task path energy score of each training sample under any main task predicted label sequence. Represents the natural constant. Indicates the effective length of the sentence. This represents the token time step index in the sentence. Indicates the first The training sample at the th ... The inverse class frequency weights corresponding to the true labels of the main task at each time step. This represents the weight coefficient for auxiliary tasks. This represents any sequence of auxiliary task prediction boundary labels in the auxiliary task label space. Represents the auxiliary task label space. Indicates the first The auxiliary launch score of each training sample. Indicates the first Auxiliary task ground boundary label sequence for each training sample. Indicates the first The auxiliary task path energy score of each training sample under any auxiliary task prediction boundary label sequence. Indicates the first The auxiliary task path energy score of each training sample under the true boundary label sequence of the auxiliary task.

[0115] Understandably, the auxiliary task rebranch only predicts simplified BIO labels, so it is supervised by another independent linear chain CRF; the training objective of the model is to minimize the joint loss of the main task loss and the auxiliary task loss for the entire batch for end-to-end optimization.

[0116] In a more specific implementation of this embodiment, the calculation process of the inverse class frequency weight includes:

[0117] ;

[0118] In the formula, Represents the inverse class frequency weight. Represents the total number of tokens. Indicates the total number of label categories. Indicates label category Total frequency of occurrence;

[0119] It should be noted that, in order to balance the contributions of high-frequency classes (such as 'O') and rare entity classes (such as 'B-MISC'), the contributions of each label class are statistically analyzed on the training set. The total frequency of occurrence is used to calculate the inverse class frequency weight, which ensures that the weight of rare classes is amplified while the weight of high-frequency classes is suppressed.

[0120] To illustrate the technical effectiveness of this embodiment, experiments were conducted on the publicly available Twitter-2015 and Twitter-2017 image-text datasets. Both datasets provide paired text-image samples and BIO sequence annotations, covering various entity types. Commonly used metrics include Precision / Recall / F1. The experimental results are shown in Table 1.

[0121] Table 1 Experimental Results

[0122]

[0123] Experimental results show that, compared with the fusion method that only uses global image vectors or uniform grid patches, this embodiment achieves significant improvements in overall F1 and recall of minority classes such as ORG / MISC. In scenarios with weak correlation between images and text or missing images, sentence-level gating enables the model to robustly degenerate into a text-dominated path, effectively reducing the false detection rate and verifying the effectiveness and generalization ability of the proposed method in complex social media environments.

[0124] In this embodiment of the invention, fine-grained alignment is performed for instance-level regions, information reconstruction is performed after cross-modal attention to retain key clues, visual noise is adaptively suppressed according to sentence-level image-text relevance while forcibly retaining the core semantics of the text, and class imbalance constraints of boundary assistance and inverse class frequency weights are combined to improve minority class recall and sequence boundary stability, thereby further enhancing the performance of named entity recognition.

[0125] Please see Figure 3 The present invention provides a multimodal named entity recognition device according to embodiment two, comprising:

[0126] Data input module 301 is used to input the text to be recognized and the image to be recognized into the trained multimodal named entity recognition model;

[0127] The data extraction module 302 is used to extract the word-level feature matrix of the text to be identified through the text extraction module, and to extract the visual feature matrix of the image to be identified by the image extraction module.

[0128] The self-attention processing module 303 is used to use the self-attention Transformer module to determine the main task text features of the word-level feature matrix, and to perform attention operations on the word-level feature matrix and then linearly map and output the auxiliary emission score.

[0129] The sentence-level enhancement module 304 is used to generate gate weights based on the cosine similarity between the word-level feature matrix and the visual feature matrix, and to determine the visual region features by weighting the visual feature matrix based on the sentence-level soft gate module.

[0130] The cross-modal interaction module 305 is used to perform cross-attention processing on the main task text features and visual region features according to the cross-modal Transformer module, generate corresponding region alignment features and visual perception features, and use the main task text features to guide the interaction of visual perception features to construct context-aware features.

[0131] The expert fusion module 306 is used to input the region alignment features and context-aware features into the MLP expert head respectively, and then perform weighted linear fusion using gated weights to generate multimodal fusion features;

[0132] The visual enhancement module 307 is used to generate gating coefficients by concatenating region alignment features and context-aware features through the visual gating module, and to generate gated region alignment features by weighting the region alignment features.

[0133] The scoring determination module 308 is used to concatenate context-aware features, multimodal fusion features and gated region alignment features to output enhanced fusion features, and linearly map them to the main emission score;

[0134] The entity recognition module 309 is used to fuse the main transmission score and the auxiliary transmission score to output a fused transmission score, and to determine the entity recognition result by decoding the fused transmission score through the CRF decoder.

[0135] In one specific embodiment of this implementation, a model training module is also included, for:

[0136] For any associated training text and training image within the current training batch in the training set, the training word-level feature matrix, training visual feature matrix, training main task text features, and training auxiliary emission score are determined through the multimodal named entity recognition model to be trained.

[0137] The cosine similarity between the training word-level feature matrix and the training visual feature matrix is ​​standardized within the batch using the multimodal named entity recognition model to be trained, and then gating weights are generated. The training visual feature matrix is ​​then weighted to determine the training visual region features.

[0138] The training fusion emission score is determined by using the multimodal named entity recognition model to be trained based on the training visual region features, the training visual region features, and the training auxiliary emission score.

[0139] The auxiliary mission path energy score is determined using a CRF decoder to train the auxiliary launch score.

[0140] Calculate the CRF joint loss based on the training fusion launch score and the auxiliary mission path energy score;

[0141] By minimizing the CRF joint loss, the model parameters of the multimodal named entity recognition model to be trained are iteratively optimized until the trained multimodal named entity recognition model is determined.

[0142] Embodiment 3 of the present invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program; when the computer program is executed by the processor, the processor performs the steps of the multimodal named entity recognition method as described in Embodiment 1 of the present invention.

[0143] Embodiment 4 of the present invention also provides a computer-readable storage medium storing a computer program / instructions thereon, which, when executed by a processor, implements the steps of the multimodal named entity recognition method as described in Embodiment 1 of the present invention.

[0144] Embodiment 5 of the present invention also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the multimodal named entity recognition method as described in Embodiment 1 of the present invention.

[0145] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described device and module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0146] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0147] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0148] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0149] If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0150] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal named entity recognition method, characterized in that, include: Input the text and image to be recognized into the trained multimodal named entity recognition model; The word-level feature matrix of the text to be identified is extracted by the text extraction module, and the visual feature matrix of the image to be identified is extracted by the image extraction module. The self-attention Transformer module is used to determine the main task text features of the word-level feature matrix, and after performing attention operation on the word-level feature matrix, a linear mapping is performed to output the auxiliary emission score; Based on the sentence-level soft gating module, gating weights are generated according to the cosine similarity between the word-level feature matrix and the visual feature matrix, and the visual feature matrix is ​​weighted to determine the visual region features. According to the cross-modal Transformer module, the main task text features and the visual region features are cross-attention processed to generate corresponding region alignment features and visual perception features. The main task text features are used to guide the interaction of the visual perception features to construct context-aware features. After inputting the region alignment features and the context-aware features into the MLP expert head, the gating weights are used to perform weighted linear fusion to generate multimodal fusion features. The visual gating module concatenates the region alignment features and the context-aware features to generate gating coefficients, and then weights the region alignment features to generate gated region alignment features. The context-aware features, the multimodal fusion features, and the gated region alignment features are used to concatenate and output an enhanced fusion feature, which is then linearly mapped to the main emission score. The main emission score and the auxiliary emission score are fused to output a fused emission score, and the entity recognition result is determined by the CRF decoder based on the fused emission score.

2. The multimodal named entity recognition method according to claim 1, characterized in that, The step of extracting the word-level feature matrix of the text to be identified through the text extraction module includes: After adding [CLS] and [SEP] tags to the beginning and end of the text to be identified, the text is input into the RoBERTa pre-trained model, which outputs sub-word vectors. After the sub-word vectors are randomly deactivated, repeated filtering is performed according to the first character alignment strategy to form a word-level feature matrix.

3. The multimodal named entity recognition method according to claim 1, characterized in that, The step of extracting the visual feature matrix of the image to be identified using the image extraction module includes: The image to be identified is segmented using the SAM model to generate a region mask. Each region mask is input into the CLIP model, and the corresponding mask embedding is output. Averaging operations are performed on each of the mask embeddings to construct the corresponding region vectors; A visual feature matrix is ​​formed by mapping the region vectors using a linear layer.

4. The multimodal named entity recognition method according to claim 1, characterized in that, The sentence-level soft gating module generates gating weights based on the cosine similarity between the word-level feature matrix and the visual feature matrix, and weights the visual feature matrix to determine visual region features, including: Extract the [CLS] vector from the word-level feature matrix as a text semantic summary; The visual feature matrix is ​​averaged to determine the overall visual image features; Calculate the cosine similarity between the text semantic summary and the visual whole image features; Gated weights are generated based on the cosine similarity using a sigmoid function with a learnable temperature coefficient and bias threshold. The gating weights are broadcast by channel and multiplied with the visual feature matrix to determine the visual region features.

5. The multimodal named entity recognition method according to claim 1, characterized in that, The training process of the trained multimodal named entity recognition model includes: For any associated training text and training image within the current training batch in the training set, the training word-level feature matrix, training visual feature matrix, training main task text features, and training auxiliary emission score are determined through the multimodal named entity recognition model to be trained. The cosine similarity between the training word-level feature matrix and the training visual feature matrix is ​​standardized within the batch using the multimodal named entity recognition model to be trained, and then gating weights are generated. The training visual feature matrix is ​​then weighted to determine the training visual region features. The training fusion emission score is determined by the multimodal named entity recognition model to be trained based on the training visual region features, the training visual region features, and the training auxiliary emission score. The auxiliary mission path energy score of the training-assisted launch score is determined using a CRF decoder. Calculate the CRF joint loss based on the inverse class frequency weights of the training set, using the training fusion launch score and the auxiliary mission path energy score. By minimizing the CRF joint loss, the model parameters of the multimodal named entity recognition model to be trained are iteratively optimized until a well-trained multimodal named entity recognition model is determined.

6. The multimodal named entity recognition method according to claim 5, characterized in that, The CRF joint loss includes: ； In the formula, Indicates the joint loss of CRF. Indicates batch size, Indicates the training sample index. This represents a logarithmic function with the natural constant as its base. Indicates the first Training fusion emission score of each training sample Indicates the first The main task true label sequence of each training sample. This represents any sequence of predicted labels for the main task within the main task label space. Indicates the main task label space. Indicates the first The main task path energy score of each training sample under the true label sequence of the main task. Indicates the first The main task path energy score of each training sample under any main task predicted label sequence. Represents the natural constant. Indicates the effective length of the sentence. This represents the token time step index in the sentence. Indicates the first The training sample at the th ... The inverse class frequency weights corresponding to the true labels of the main task at each time step. This represents the weight coefficient for auxiliary tasks. This represents any sequence of auxiliary task prediction boundary labels in the auxiliary task label space. Represents the auxiliary task label space. Indicates the first The auxiliary launch score of each training sample. Indicates the first Auxiliary task ground boundary label sequence for each training sample. Indicates the first The auxiliary task path energy score of each training sample under any auxiliary task prediction boundary label sequence. Indicates the first The auxiliary task path energy score of each training sample under the true boundary label sequence of the auxiliary task.

7. A multimodal named entity recognition device, characterized in that, include: The data input module is used to input the text and image to be recognized into the trained multimodal named entity recognition model; The data extraction module is used to extract the word-level feature matrix of the text to be identified through the text extraction module, and to extract the visual feature matrix of the image to be identified by the image extraction module. The self-attention processing module is used to determine the main task text features of the word-level feature matrix by employing the self-attention Transformer module, and to output the auxiliary emission score by linear mapping after performing attention operations on the word-level feature matrix. The sentence-level enhancement module is used to generate gate weights based on the cosine similarity between the word-level feature matrix and the visual feature matrix, and to weight the visual feature matrix to determine the visual region features. The cross-modal interaction module is used to perform cross-attention processing on the main task text features and the visual region features according to the cross-modal Transformer module, generate corresponding region alignment features and visual perception features, and use the main task text features to guide the interaction of the visual perception features to construct context-aware features. The expert fusion module is used to input the region alignment features and the context-aware features into the MLP expert head respectively, and then perform weighted linear fusion using the gated weights to generate multimodal fusion features; The visual enhancement module is used to generate gating coefficients by concatenating the region alignment features and the context-aware features through the visual gating module, and to generate gated region alignment features by weighting the region alignment features. The scoring determination module is used to concatenate the context-aware features, the multimodal fusion features, and the gated region alignment features to output enhanced fusion features, and linearly map them to the main emission score; The entity recognition module is used to fuse the main transmission score and the auxiliary transmission score to output a fused transmission score, and to determine the entity recognition result based on the fused transmission score by a CRF decoder.

8. A computer device, characterized in that, The method includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor causes the processor to perform the steps of the multimodal named entity recognition method as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the multimodal named entity recognition method as described in any one of claims 1-6.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the multimodal named entity recognition method as described in any one of claims 1-6.

Images