A multimodal named entity recognition method and apparatus

Abstract

The present invention discloses a multimodal named entity recognition method and device, which are used to solve the technical problem that the existing multimodal named entity recognition methods lead to poor accuracy of multimodal named entity recognition. The method comprises the following steps: obtaining text data to be detected and image data to be detected; using the text data to be detected and the image data to be detected as inputs of a text image description extraction module, and outputting text modal features and image description text modal features; using an image feature extraction module to extract image features based on the image data to be detected, and outputting image modal features; performing feature fusion on the text modal features and the image description text modal features through a hybrid expert fusion module to generate fused text modal features; and using a multimodal attention fusion module to perform attention fusion on the fused text modal features and image modal features, and outputting a target general feature recognition result.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a multimodal named entity recognition method and apparatus. Background Technology

[0002] With the rapid development of the field of artificial intelligence, more and more application scenarios need to integrate multiple modal data (such as text, images, audio, video, etc.). Data of different modalities can complement each other and make up for the shortcomings of a single modality in information transmission.

[0003] Currently, multimodal named entity recognition (MNER) of text and its accompanying images has attracted widespread attention. MNER enables the exchange and transformation of semantic information between different modalities by fusing features from text and images, accurately identifying and labeling important concepts, terms, and entities (such as names of people, places, and organizations) contained in the target modality. In modern education, streaming media technology has promoted the diversification of teaching content, generating various types of data such as text, video, audio, and images. The fusion of these different modalities can provide complementary information, eliminating ambiguities that exist in single-modal data. For example, images and audio in a video can help understand the meaning of text, thereby improving the ability to recognize complex scenes, providing more comprehensive contextual information, and contributing to the accuracy and robustness of entity recognition tasks. Therefore, feature extraction and fusion of information from multiple modalities has become the main method for current multimodal named entity recognition tasks.

[0004] Most existing multimodal named entity recognition methods utilize entity-related information from image modalities to assist in the recognition of text named entities. However, the distribution of text and visual features in the feature space is usually significantly different, and the meanings of text and its accompanying image do not always match. This makes it difficult for the fused features to effectively capture the correlation between the two, and the semantic features are incomplete, resulting in poor accuracy of multimodal named entity recognition. Summary of the Invention

[0005] This invention provides a multimodal named entity recognition method and apparatus to solve the technical problem that existing multimodal named entity recognition methods result in poor accuracy in multimodal named entity recognition.

[0006] The first aspect of this invention provides a multimodal named entity recognition method, comprising:

[0007] The system acquires text data and image data to be detected, and inputs the text data and image data to be detected into a preset multimodal named entity recognition model. The preset multimodal named entity recognition model includes a text image description extraction module, an image feature extraction module, a hybrid expert fusion module, and a multimodal attention fusion module.

[0008] The text data to be detected and the image data to be detected are used as inputs to the text-image description extraction module, which outputs text modal features and image description text modal features.

[0009] The image feature extraction module is used to extract image features from the image data to be detected and output image modal features;

[0010] The hybrid expert fusion module performs feature fusion on the text modal features and the image description text modal features to generate fused text modal features;

[0011] The multimodal attention fusion module is used to perform attention fusion on the fused text modal features and the image modal features, and outputs the target general feature recognition result.

[0012] Optionally, the text-image description extraction module includes a convolutional submodule, an autoregressive decoder, a natural language processing model encoder, and a natural language processing model decoder; the step of using the text data to be detected and the image data to be detected as input to the text-image description extraction module, and outputting text modal features and image description text modal features, includes:

[0013] A convolutional submodule is used to extract high-dimensional features from the image data to be detected, generating high-dimensional image features.

[0014] The high-dimensional features of the image are decoded using an autoregressive decoder to output image description text;

[0015] The image description text and the text data to be detected are respectively input into the natural language processing model encoder for encoding, generating the identifier sequence and attention mask corresponding to the image description text, and the identifier sequence and attention mask corresponding to the text data to be detected;

[0016] The identifier sequence corresponding to the image description text and the identifier sequence corresponding to the text data to be detected are marked respectively to generate the identifier mark sequence corresponding to the image description text and the identifier mark sequence corresponding to the text data to be detected.

[0017] The identifier tag sequence and attention mask corresponding to the image description text, and the identifier tag sequence and attention mask corresponding to the text data to be detected are respectively input into the natural language processing model decoder for decoding to generate text modal features and image description text modal features.

[0018] Optionally, the step of using the image feature extraction module to extract image features based on the image data to be detected and outputting image modal features includes:

[0019] The image data to be detected is pixel-adjusted to generate an adjusted pixel image;

[0020] The image feature extraction module extracts image features from the adjusted pixel image and outputs a visual representation of the image.

[0021] A linear transformation is performed on the visual representation of the image to output the image modal features.

[0022] Optionally, the hybrid expert fusion module includes a gating layer, an expert layer, and an attention layer; the step of fusing the text modal features and the image description text modal features through the hybrid expert fusion module to generate fused text modal features includes:

[0023] Based on the text modal features and the image description text modal features, weighted input features are determined;

[0024] The gating layer performs probability calculations based on the weighted input features to determine the selection probability distribution of multiple experts in the expert layer.

[0025] Select any expert whose selection probability distribution is greater than a preset probability threshold as the activated expert;

[0026] The weighted input features are calculated by multiple activation experts, and the expert calculation results corresponding to each activation expert are output.

[0027] An attention layer is used to perform weighted calculations on the expert calculation results and selection probability distributions corresponding to each activated expert to determine the hybrid expert weighted result.

[0028] The hybrid expert weighted results and the weighted input features are normalized and fused to generate fused text modal features.

[0029] Optionally, the multimodal attention fusion module includes a multimodal attention layer and a normalized feedforward network submodule; the step of using the multimodal attention fusion module to perform attention fusion on the fused text modal features and the image modal features, and outputting a target general feature recognition result, includes:

[0030] The multimodal attention layer calculates multi-head attention weights based on the fused text modal features and the image modal features, and outputs multiple first text image attention weights and multiple second text image attention weights.

[0031] The attention weights of the first text image and the attention weights of the second text image are concatenated to generate the first text image attention concatenation vector and the second text image attention concatenation vector.

[0032] Linear transformations are performed on the first text image attention concatenation vector and the second text image attention concatenation vector, respectively, to output the first multimodal attention fusion representation and the second multimodal attention fusion representation;

[0033] The normalized feedforward network submodule is used to perform normalized superposition based on the first multimodal attention fusion representation and the image modal features to generate the first normalized superposition feature;

[0034] The normalized feedforward network submodule is used to normalize and superimpose the second multimodal attention fusion representation and the fused text modal features to generate the second normalized superimposed feature;

[0035] The multimodal attention layer calculates multi-head attention weights based on the fused text modal features and the first normalized superposition features, and outputs multiple text normalized attention weights.

[0036] The normalized attention weights of the multiple texts are concatenated and linearly transformed to output a third multimodal attention fusion representation;

[0037] The target text representation is generated by normalizing and superimposing the third multimodal attention fusion representation and the fused text modality features using a normalized feedforward network submodule.

[0038] The second normalized overlay feature and the target text representation are fused to generate a target visual representation;

[0039] The visual representation and text representation of the target are concatenated and linearly transformed to generate a general feature recognition result for the target.

[0040] Optionally, the training process of the pre-set multimodal named entity recognition model includes:

[0041] When the text data and image data to be trained are received, the text data and image data to be trained are used as input to the initial multimodal named entity recognition model, and the general feature recognition result to be trained is output.

[0042] Calculate the probability distribution to be trained based on the recognition results of the general features to be trained;

[0043] Substitute the probability distribution to be trained into the preset loss function and take its derivative to determine the model gradient;

[0044] The initial multimodal named entity recognition model is updated using the model gradient to determine the intermediate multimodal named entity recognition model, and the number of model iterations is counted in real time.

[0045] Determine whether the number of model iterations has reached the preset number of training iterations;

[0046] If this is achieved, the intermediate multimodal named entity recognition model will be used as the pre-trained multimodal named entity recognition model.

[0047] A second aspect of the present invention provides a multimodal named entity recognition device, comprising:

[0048] The acquisition module is used to acquire the text data to be detected and the image data to be detected, and input the text data to be detected and the image data to be detected into a preset multimodal named entity recognition model. The preset multimodal named entity recognition model includes a text image description extraction module, an image feature extraction module, a hybrid expert fusion module, and a multimodal attention fusion module.

[0049] The input module is used to take the text data to be detected and the image data to be detected as input to the text image description extraction module, and output text modal features and image description text modal features;

[0050] The output module is used to extract image features based on the image data to be detected using the image feature extraction module, and output image modal features;

[0051] The generation module is used to perform feature fusion on the text modal features and the image description text modal features through the hybrid expert fusion module to generate fused text modal features;

[0052] The fusion module is used to perform attention fusion on the fused text modal features and the image modal features using the multimodal attention fusion module, and output the target general feature recognition result.

[0053] A computer device provided in a third aspect of the present invention includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the multimodal named entity recognition method as described in any of the preceding claims.

[0054] The fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed, implements the steps of the multimodal named entity recognition method as described in any of the preceding claims.

[0055] The fifth aspect of the present invention provides a computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, wherein, when the program instructions are executed by a computer, the computer performs the steps of the multimodal named entity recognition method as described in any of the preceding claims.

[0056] As can be seen from the above technical solutions, the present invention has the following advantages:

[0057] The above-described technical solution of the present invention provides a multimodal named entity recognition method. First, it acquires text data and image data to be detected, and inputs these data into a pre-set multimodal named entity recognition model. This model includes a text-image description extraction module, an image feature extraction module, a hybrid expert fusion module, and a multimodal attention fusion module. Next, the text data and image data to be detected are used as input to the text-image description extraction module, which outputs text modal features and image description text modal features. The image feature extraction module extracts image features from the image data to be detected, outputting image modal features. Finally, the hybrid expert fusion module fuses the text modal features and image description text modal features to generate fused text. Modal features; finally, a multimodal attention fusion module is used to perform attention fusion on the fused text modal features and image modal features, and output the target general feature recognition result; based on the above scheme, the text image description extraction module and the image feature extraction module are used to process the acquired text data and image data to be detected, and the generated text modal features and image description text modal features are fused by the hybrid expert fusion module to generate fused text modal features. Then, the multimodal attention fusion module is used to perform attention fusion on the fused text modal features and image modal features to output the target general feature recognition result. This process can greatly enhance the semantic information of the text features, making the semantic features more complete and consistent in the transmission process, thereby improving the accuracy of multimodal named entity recognition. Attached Figure Description

[0058] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0059] Figure 1 This is a flowchart of the steps of a multimodal named entity recognition method provided in Embodiment 1 of the present invention;

[0060] Figure 2 This is a schematic diagram of the structure of the pre-set multimodal named entity recognition model provided in Embodiment 1 of the present invention;

[0061] Figure 3 This is a flowchart illustrating the steps of the model training process for the pre-set multimodal named entity recognition model provided in Embodiment 2 of the present invention.

[0062] Figure 4 This is a training framework diagram of the pre-set multimodal named entity recognition model provided in Embodiment 2 of the present invention;

[0063] Figure 5 This is a structural block diagram of a multimodal named entity recognition device provided in Embodiment 3 of the present invention. Detailed Implementation

[0064] This invention provides a multimodal named entity recognition method and apparatus to solve the technical problem that existing multimodal named entity recognition methods have poor accuracy in multimodal named entity recognition.

[0065] To make the objectives, features, and advantages of this invention more apparent and understandable, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0066] Please see Figure 1 , Figure 1 This is a flowchart illustrating the steps of a multimodal named entity recognition method provided in Embodiment 1 of the present invention.

[0067] This invention provides a multimodal named entity recognition method, comprising:

[0068] Step 101: Obtain the text data and image data to be detected, and input the text data and image data to be detected into the preset multimodal named entity recognition model. The preset multimodal named entity recognition model includes a text image description extraction module, an image feature extraction module, a hybrid expert fusion module, and a multimodal attention fusion module.

[0069] It should be noted that you should refer to [link / reference]. Figure 2The pre-built multimodal named entity recognition model includes a text image description extraction module, an image feature extraction module, a hybrid expert fusion module, a multimodal attention fusion module, and a CRF entity labeling module. The text image description extraction module includes a text feature extraction module and an image description generation module. The text feature extraction module includes a natural language processing model encoder and a natural language processing model decoder. The image description generation module includes a convolutional submodule and an autoregressive decoder. The CRF entity labeling module is used for model training to obtain the trained pre-built multimodal named entity recognition model.

[0070] Step 102: Use the text data to be detected and the image data to be detected as input to the text image description extraction module, and output the text modal features and the image description text modal features.

[0071] The text image description extraction module includes a convolutional submodule, an autoregressive decoder, a natural language processing model encoder, and a natural language processing model decoder.

[0072] Specifically, the process of using the text data to be detected and the image data to be detected as input to the text-image description extraction module, and outputting text modal features and image description text modal features, can be achieved by executing the following sub-steps S11 to S15:

[0073] Step S21: Use a convolutional submodule to extract high-dimensional features from the image data to be detected, and generate high-dimensional image features;

[0074] Step S22: Decode the high-dimensional features of the image using an autoregressive decoder and output the image description text;

[0075] It should be noted that this invention uses a pre-trained BLIP (Bootstrapping Language-Image Pre-training) model for image description generation, which can efficiently generate a natural language description of an image that incorporates its context. This model combines visual and linguistic information, using multimodal learning to understand image content and generate relevant text descriptions. The BLIP model first extracts features from the input image, using its built-in convolutional network through a convolutional submodule to extract high-dimensional features (high-dimensional image features). These features effectively capture information such as the image's structure, color, and texture, forming a deep understanding of the image content.

[0076] Furthermore, after feature extraction, BLIP employs an autoregressive decoder to progressively generate descriptive text, i.e., image description text. The decoder predicts the next most likely word based on the current image features and already generated words. The generation process begins with a starting marker and proceeds word-by-word until an ending marker is encountered or the maximum length limit is reached. The generated descriptions are typically natural and fluent. BLIP is pre-trained on large-scale image-text pairs, giving it excellent generalization capabilities and enabling it to adapt to different types of images and scenes, generating highly accurate and rich text descriptions. Compared to traditional methods, BLIP not only reduces additional feature engineering work but also makes the entire image description generation process smoother and more efficient.

[0077] Step S23: Input the image description text and the text data to be detected into the natural language processing model encoder for encoding, and generate the identifier sequence and attention mask corresponding to the image description text and the identifier sequence and attention mask corresponding to the text data to be detected.

[0078] Step S24: Mark the identifier sequence corresponding to the image description text and the identifier sequence corresponding to the text data to be detected, respectively, to generate the identifier mark sequence corresponding to the image description text and the identifier mark sequence corresponding to the text data to be detected;

[0079] Step S25: Input the identifier tag sequence and attention mask corresponding to the image description text, and the identifier tag sequence and attention mask corresponding to the text data to be detected into the natural language processing model decoder for decoding, to generate text modal features and image description text modal features.

[0080] It should be noted that multimodal named entity recognition typically involves two main modalities of data: text (S) and image (I). This invention introduces another text modality, namely image description text (C). Therefore, this invention uses... To represent the input feature data and its dimensions for the three modalities, where, The input modalities include images, text, and image description text. The sequence length corresponding to the input mode. The feature dimension corresponding to the input modality.

[0081] Furthermore, the modal data requiring text feature extraction in this invention are of two types: text and text-image descriptions. To process the text data, this invention employs a pre-trained BERT model to extract text features. BERT is a pre-trained model based on the Transformer architecture. It is first pre-trained on a large amount of unlabeled text data, and then fine-tuned on labeled data for specific tasks to better adapt to different application scenarios. Using the pre-trained BERT-base-cased model, the tokenizer also preserves word case information, which is crucial for named entity recognition tasks. The tokenizer segments the original text into sentences and words, processes punctuation, and cleans the text to remove irrelevant characters and formatting.

[0082] Specifically, the image description text and the text to be detected are input into the BERT-base-cased model's natural language processing encoder for encoding. Each input text is converted into an ID sequence containing a corresponding vocabulary index. Simultaneously, an attention mask is generated to indicate which words are actual input words and which are padding positions. Next, the identifier sequences corresponding to the image description text and the text to be detected are labeled, adding two special labels to mark the beginning and end of sentences to help the model better understand the contextual structure of the sequence. Typically, a start label ([CLS]) is used as the sentence start label, and a end label ([SEP]) is used as the sentence end label. Subsequently, the converted ID sequences and masks are input into the BERT model's natural language processing decoder. Multiple Transformer encoder layers stacked with a self-attention mechanism capture the relationships between words in the input sequence, ultimately obtaining the contextual embedding for each input word, thus generating text modal features and image description text modal features.

[0083] Furthermore, assuming that the dimension of the text (S) in the Twitter2015 and Twitter2017 datasets is... ,in , The dimension of the corresponding descriptive text generated based on the images in the dataset is... ,in , The final text features (text modal features) are S=(s1+s2+...+s...). 128 The image generation description features (image description text modal features) are C=(c1+c2+...+c 128); where s1 is the feature embedding of the first text word in the text sequence, s2 is the feature embedding of the second text word in the text sequence, and s 128 c1 is the feature embedding of the 128th text word in the text sequence, c2 is the feature embedding of the 1st text description in the image description text sequence, c3 is the feature embedding of the 2nd text description in the image description text sequence, and c4 is the feature embedding of the 1st text description in the image description text sequence. 128 Feature embedding for the 128th text description in the image description text sequence.

[0084] Step 103: Use the image feature extraction module to extract image features from the image data to be detected and output the image modal features.

[0085] Specifically, the process of extracting image features from the image data to be detected using the image feature extraction module and outputting the image modal features can be achieved by executing the following sub-steps S31 to S32:

[0086] Step S31: Adjust the pixels of the image data to be detected to generate an adjusted pixel image;

[0087] Step S32: Extract image features from the adjusted pixel image using the image feature extraction module and output the image visual representation;

[0088] Step S33: Perform a linear transformation on the visual representation of the image to output the image modal features.

[0089] The image feature extraction module adopts a residual network (ResNet) structure.

[0090] It should be noted that, as one of the most advanced CNN models for image recognition, Residual Networks (ResNet) have demonstrated their ability to extract meaningful feature representations of input images at deep layers. Therefore, this invention preserves the output of the last convolutional layer of a pre-trained 152-layer ResNet and uses it to represent each image. This effectively segments each input image into 49 visual patches of equal size, and represents each patch with a 2048-dimensional vector.

[0091] Specifically, given an input image I (the image data to be detected), its size is first resized to 224×224 pixels, and then input into ResNet to obtain its visual representation (image visual representation), represented as I=(i1+i2+...+i...). 49 ), where i u I is the representation of the u-th visual block in a 2048-dimensional vector, and i1 is the representation of the 1-th visual block in a 2048-dimensional vector. To project the visual representation onto the same distribution space as the text representation, this invention further uses a linear transformation to convert I: ,in It is a weight matrix. The value is 768. Therefore, V = (v1 + v2 + ... + v 49 The generated image features (image modal features) are the generated image features.

[0092] Step 104: Use the hybrid expert fusion module to fuse text modal features and image description text modal features to generate fused text modal features.

[0093] The hybrid expert fusion module includes a gating layer, an expert layer, and an attention layer.

[0094] It should be noted that the use of a hybrid expert model to dynamically and adaptively select the most suitable expert network to fuse text and image to generate descriptions captures the correlation between different modalities, narrows the distribution gap between text features and visual features, enhances the consistency of text semantic features in the network transmission process, and improves the performance of multimodal named entity recognition.

[0095] Specifically, the process of fusing text modal features and image description text modal features through the hybrid expert fusion module to generate fused text modal features can be achieved by executing the following sub-steps S41 to S46:

[0096] Step S41: Determine the weighted input features based on text modal features and image description text modal features;

[0097] It should be noted that when receiving text features and image description features (text modality features and image description text modality features) as input, the text generated from the image may be more blurry or noisy than direct text input. By adaptively adjusting the contribution weights of the two inputs through learning parameters, the influence of poor-quality inputs on the final result can be reduced, while relying more on high-quality inputs. Therefore, the processing of weighted inputs (weighted input features) can be represented as:

[0098] ;

[0099] in, X is the first learnable parameter; S is the weighted input feature; C is the text modality feature; and C is the image description text modality feature. This is the second learnable parameter.

[0100] Step S42: The gating layer performs probability calculations based on the weighted input features to determine the selection probability distribution of multiple experts in the expert layer;

[0101] It should be noted that, through the routing in the module, i.e., the gating layer, the selection probability distribution of each expert is calculated for each input, that is, the selection probability distribution corresponding to each expert in the expert layer is calculated. The processing of the selection probability distribution can be represented as follows:

[0102] ;

[0103] Where Route represents the selection probability distribution; It is the softmax activation function; This is the weight matrix corresponding to the gated layer; X represents the bias term corresponding to the gated layer; X represents the weighted input feature.

[0104] Step S43: Select any expert whose selection probability distribution is greater than the preset probability threshold as the activated expert;

[0105] It should be noted that any expert corresponding to a selection probability distribution greater than a preset probability threshold is selected as an activated expert, while experts corresponding to selection probability distributions less than or equal to the preset probability threshold are not activated; the preset probability threshold can be set as needed, and this invention is not limited thereto.

[0106] Step S44: Perform expert calculations on the weighted input features using multiple activation experts, and output the expert calculation results corresponding to each activation expert;

[0107] It should be noted that each activation expert independently calculates the weighted input and produces the corresponding output (expert calculation result). The processing of expert calculation results can be represented as follows:

[0108] ;

[0109] in, The expert calculation result corresponding to the i-th activated expert; This is the second weight matrix corresponding to the i-th activation expert; It is the ReLU activation function; Let be the first weight matrix corresponding to the i-th activation expert; This is the first bias term corresponding to the i-th activated expert; X represents the second bias term corresponding to the i-th activation expert; X is the weighted input feature.

[0110] Step S45: Use the attention layer to perform weighted calculations on the expert calculation results and selection probability distributions corresponding to each activated expert to determine the weighted result of the mixed experts;

[0111] It should be noted that, based on the selection probability distribution, the outputs of all experts are weighted. Specifically, an attention layer is used to weight the expert calculations and selection probability distributions for each activated expert, determining the mixed expert weighted result. The processing procedure for the mixed expert weighted result is as follows:

[0112] ;

[0113] in, The results are weighted by a combination of experts. Let i be the selection probability distribution corresponding to the i-th activated expert; The expert calculation result corresponding to the i-th activated expert; To determine the number of experts activated.

[0114] Step S46: Normalize and fuse the hybrid expert weighted results and weighted input features to generate fused text modal features.

[0115] It's worth noting that to mitigate the vanishing and exploding gradient problems and allow the model to learn more complex representations without losing useful information from the original input, residual connections were added to the model. This allows the network to rely not only on deep, abstract features but also to retain some information from the original input. Furthermore, LayerNorm normalizes the output of each layer, maintaining input stability and thus improving training speed and the model's generalization ability.

[0116] Furthermore, the process of fusing text modal features can be represented as follows:

[0117] ;

[0118] in, To fuse text modal features; X represents weighted input features; The results are weighted by a combination of experts. The mean of the layer normalized input; The standard deviation of the layer normalized input; For learnable scaling parameters; The offset parameter is learnable.

[0119] Step 105: Use a multimodal attention fusion module to perform attention fusion on the fused text modal features and image modal features, and output the target general feature recognition result.

[0120] The multimodal attention fusion module includes a multimodal attention layer and a normalized feedforward network submodule; the normalized feedforward network submodule includes a normalization layer and a feedforward layer, and the structure of the feedforward layer is a feedforward network.

[0121] It should be noted that, to make the fusion of image and text two-modal data more effective, this invention has enhanced the semantic features of the text in the hybrid expert module. This can alleviate the contextual bias problem that may exist when fusing visual features. Therefore, the multimodal attention fusion module only needs to combine the image modality features V with the fused text modality Output enhanced with semantic features. moeThe fusion process was then performed. Furthermore, to further enhance the learning of better feature representations during the fusion, a total of three multimodal attention fusions were conducted.

[0122] Specifically, the process of using a multimodal attention fusion module to perform attention fusion on the fused text modal features and image modal features, and outputting the target general feature recognition result, can be achieved by executing the following sub-steps S51 to S510:

[0123] Step S51: Calculate multi-head attention weights based on the fused text modal features and image modal features through a multimodal attention layer, and output multiple first text image attention weights and multiple second text image attention weights;

[0124] It should be noted that, in order to better integrate visual features with text features, the visual representation first guides the learning of text representation, learning text features with better visual context. This allows the model to refer to relevant image information when processing text, thereby improving word representation capabilities. The image features V input to the module are used as the query, and the text features are used as the output. moe Using these as keys and values, calculate the attention weights for the first text image corresponding to the current attention head for both modal data.

[0125] Furthermore, to obtain a visual representation related to each token in the text, each word is aligned with its closely related visual blocks, and high or low attention weights are assigned to its related or unrelated visual blocks. A multimodal attention fusion mechanism is then used again to output the text features. moe As a query, image features V are used as keys and values ​​to calculate the second text image attention weights for the two modalities at the current attention head.

[0126] The calculation process of the first text image attention weight can be expressed as follows:

[0127] ;

[0128] in, Let be the attention weight for the i-th first text image, and let represent the i-th attention head; The weight matrix corresponding to the query matrix; This is the weight matrix corresponding to the key-value matrix; To fuse text modal features; d represents the image modal features; d is the text dimension (768); m is the number of attention heads. is the weight matrix corresponding to the value matrix; T is the transpose.

[0129] It is worth mentioning that the processing of the second text image attention weight is the same as the calculation process of the first text image attention weight, and will not be elaborated further in this invention.

[0130] Step S52: Concatenate multiple first text image attention weights and multiple second text image attention weights respectively to generate first text image attention concatenation vector and second text image attention concatenation vector;

[0131] Step S53: Perform linear transformations on the first text image attention concatenation vector and the second text image attention concatenation vector respectively, and output the first multimodal attention fusion representation and the second multimodal attention fusion representation;

[0132] It should be noted that concatenating the attention weights calculated by each attention head together can capture the diverse attention that different heads give to the input. Finally, the concatenated result is multiplied by a weight matrix (such as W') to perform a linear transformation, learning the relationships between different attention heads, reducing potentially redundant information after concatenation, and fusing the features of the multi-head outputs to generate the final multimodal attention fusion representation.

[0133] The processing of the first multimodal attention fusion representation can be expressed as follows:

[0134] ;

[0135] in, This is the first multimodal attention fusion representation; The first text image attention weight represents the first attention head; Let be the attention weight for the m-th first text image, and let represent the m-th attention head. This is the weight matrix; To fuse text modal features; These are image modal features.

[0136] It is worth mentioning that the processing procedure for the second multimodal attention fusion representation is the same as that for the first multimodal attention fusion representation, and will not be described in detail here.

[0137] Step S54: The normalized feedforward network submodule is used to perform normalization and superposition based on the first multimodal attention fusion representation and image modal features to generate the first normalized superposition feature;

[0138] It should be noted that this multimodal attention fusion representation is input into three transformer sublayers (normalized feedforward network submodules). This provides richer and more meaningful feature representations, enabling the model to converge to the optimal solution more quickly.

[0139] The processing of the first normalized superposition feature can be expressed as follows:

[0140] ;

[0141] in, Normalized features; For layer normalization; For a feedforward network; P is the first normalized superposition feature, P=(p1+p2+...+p 49 ), p1 is, p2 is, p 49 for; To fuse text modal features; These are image modal features.

[0142] Step S55: The normalized feedforward network submodule is used to normalize and superimpose the second multimodal attention fusion representation and the fused text modality features to generate the second normalized superimposed features;

[0143] It should be noted that the second normalized superposition feature Q = (q1 + q2 + ... + q n The processing flow of the first normalized superposition feature is the same as that of the first normalized superposition feature, and will not be described in detail here.

[0144] Step S56: Calculate multi-head attention weights based on the fused text modal features and the first normalized superposition features through the multimodal attention layer, and output multiple text normalized attention weights;

[0145] Step S57: Concatenate and linearly transform the normalized attention weights of multiple texts to output the third multimodal attention fusion representation;

[0146] Step S58: The normalized feedforward network submodule is used to normalize and superimpose the third multimodal attention fusion representation and the fused text modality features to generate the target text representation;

[0147] It should be noted that, due to the fact that each generated vector p i This corresponds to the i-th visual block, not the i-th input word. Therefore, P needs to undergo another multimodal attention fusion to fuse the text modality features. moe As a query, the first normalized superimposed feature P serves as the key and value, generating the final target text representation T=(t1+t2+...+t...). n The processing flow of text normalization attention weight, third multimodal attention fusion representation, and target text representation is the same as that of first text image attention weight, first multimodal attention fusion representation, and first normalized superimposed feature, and will not be elaborated further in this invention.

[0148] Step S59: Fuse the second normalized overlay features and the target text representation to generate the target visual representation;

[0149] It should be noted that visual gating is used to dynamically control the contribution of visual features to obtain the final visual representation, i.e., the target visual representation Q'. The processing flow of the target visual representation can be represented as follows:

[0150] ;

[0151] in, For the visual representation of the target; The sigmoid activation function is used; T represents the target text representation. This is a transpose operation; This is the second weight matrix; Q is the second normalized superimposed feature. This is the first weight matrix.

[0152] Step S510: Concatenate and linearly transform the visual representation and text representation of the target to generate a target general feature recognition result.

[0153] It should be noted that the target visual representation Q' and the target text representation T are concatenated and linearly transformed to obtain the fused general feature (target general feature recognition result) M. The processing of the target general feature recognition result can be represented as follows:

[0154] ;

[0155] Where M represents the target general feature recognition result; T represents the target text representation; For the visual representation of the target; This is the third weight matrix.

[0156] Furthermore, the scores obtained by the present invention on the processed dataset are shown in Table 1 below. Through experimental verification, the present invention has achieved certain results in the multimodal named entity recognition task.

[0157] Table 1 Experimental Verification

[0158]

[0159] As a comparison of technical effects, existing technologies can be used as a reference. With the rapid development of artificial intelligence, more and more application scenarios require the integration of multiple modalities of data (such as text, images, audio, and video). Data from different modalities can complement each other, compensating for the shortcomings of a single modality in information transmission. As social media posts become increasingly multimodal, multimodal named entity recognition (MNER) of text and its accompanying images has attracted widespread attention. MNER enables the exchange and transformation of semantic information between modalities by fusing multimodal features such as text and images, accurately identifying and labeling important concepts, terms, and entities (such as names of people, places, and organizations) contained in the target modality. In modern education, streaming media technology has promoted the diversification of teaching content, generating various types of data such as text, video, audio, and images. The fusion of these different modalities of data can provide complementary information, eliminating ambiguities that exist in single-modal data. For example, images and audio in videos can help understand the meaning of text, thereby improving the ability to recognize complex scenes, providing more comprehensive contextual information, and contributing to the accuracy and robustness of entity recognition tasks. Therefore, feature extraction and fusion of information from multiple modalities has become the main method for current multimodal named entity recognition tasks.

[0160] However, this approach of combining multimodal data for modal named entity recognition faces two main problems. First, although existing work can generate word-aware visual representations, these representations are insensitive to visual context, ignoring potential biases introduced by visual context and leading to inaccurate information delivery. Second, the distribution of textual and visual features in the feature space often differs significantly, and the meanings of text and its accompanying image do not always match, making it difficult for the fused features to effectively capture the correlation between the two, resulting in incomplete semantic features.

[0161] To address the aforementioned problems, this invention proposes a multimodal named entity recognition method. This invention incorporates an image description generation task, aiming to extract features from the input image and generate a concise and accurate text description, providing contextual information and effectively mitigating biases caused by visual representation. Furthermore, to address the semantic loss caused by the mismatch between text and image meanings, this invention employs a hybrid expert model. The hybrid expert model adaptively selects the most suitable expert network based on the text features and the image-generated descriptive text features, capturing subtle differences between the text and image-generated text, capturing correlations between different modalities, narrowing the gap between features of different modalities in the representation space, and significantly enhancing the semantic information of the text features, making the semantic features more complete and consistent during transmission. This invention effectively solves the problem of excessively large differences in the distribution of features between text and image modalities, adaptively enhancing the semantic information of text features through image-generated text, mitigating the visual bias problem caused by the fusion of image and text features, and achieving excellent performance in multimodal named entity recognition tasks.

[0162] In summary, multimodal named entity recognition (MID) tasks capture entities present in the target modality based on both text and image data. Traditional MID methods primarily focus on modality fusion, interaction modeling, and noise filtering, employing attention and gating mechanisms to enhance the synergy between text and visual information. However, this approach, which fuses features from different modalities through multimodal attention mechanisms, may suffer from insufficient intermodal information interaction, missing semantic features, and noise interference. To address the issue of word-aware visual representations being insensitive to visual context, this invention introduces an image description generation task, generating text related to the image's visual features as supplementary information. Furthermore, considering the informal and short texts found in social media datasets, a hybrid expert model dynamically and adaptively selects the most suitable expert network to fuse text and image to generate descriptions. This captures the correlation between different modalities, narrows the distribution gap between text and visual features, enhances the consistency of text semantic features during network transmission, and improves the performance of MID. We conducted experiments on the Twitter2015 and Twitter2017 datasets to evaluate the stability and effectiveness of our proposed method. Both the Twitter2015 and Twitter2017 datasets are collected from tweets on the Twitter platform, primarily including multimodal user posts published on Twitter during 2014-2015 and 2016-2017, focusing mainly on tweets with images. However, compared to Twitter2015, the Twitter2017 dataset contains richer content and more samples. Both datasets typically use the BIO2 annotation method, which can effectively handle various types of named entities. Furthermore, this invention fuses text and image-generated descriptive features using a hybrid expert model. This allows the language text features to be supplemented with image-based contextual representations before being fused with image features, enhancing the consistency of semantic features during information flow and thus improving the accuracy and robustness of named entity recognition.

[0163] In this embodiment of the invention, a multimodal named entity recognition method is provided. First, text data and image data to be detected are acquired and input into a pre-set multimodal named entity recognition model. The pre-set multimodal named entity recognition model includes a text-image description extraction module, an image feature extraction module, a hybrid expert fusion module, and a multimodal attention fusion module. Next, the text data and image data to be detected are used as input to the text-image description extraction module, which outputs text modal features and image description text modal features. The image feature extraction module extracts image features from the image data to be detected, outputting image modal features. Finally, the hybrid expert fusion module fuses the text modal features and image description text modal features to generate a fused text. This modal feature; finally, a multimodal attention fusion module is used to perform attention fusion on the fused text modal features and image modal features, and output the target general feature recognition result; based on the above scheme, the text image description extraction module and the image feature extraction module are used to process the acquired text data and image data to be detected, and the generated text modal features and image description text modal features are fused by the hybrid expert fusion module to generate fused text modal features. Then, the multimodal attention fusion module is used to perform attention fusion on the fused text modal features and image modal features to output the target general feature recognition result. This process can greatly enhance the semantic information of the text features, making the semantic features more complete and consistent in the transmission process, thereby improving the accuracy of multimodal named entity recognition.

[0164] For better explanation, refer to Figure 3 The flowchart of the model training process of the pre-set multimodal named entity recognition model provided in Embodiment 2 of the present invention includes the following steps:

[0165] Step 301: When the text data and image data to be trained are received, the text data and image data to be trained are used as the input of the initial multimodal named entity recognition model, and the general feature recognition result to be trained is output.

[0166] It should be noted that you should refer to [link / reference]. Figure 3 The training framework of the pre-built multimodal named entity recognition model proposed in this invention specifically includes: encoding the input text data, extracting features, and obtaining the text features (text modal features) as S=(s1+s2+...+s...). nBy processing the input image using a pre-trained BLIP (Bootstrapping Language-Image Pre-training) model, captions (image description text) can be obtained. After encoding and feature extraction, the image generation description features (image description text modal features) can be obtained as C=(c1+c2+...+c n The input image is feature extracted using ResNet to obtain a visual representation, denoted as I = (i1 + i2 + ... + i49). A hybrid expert model is used to dynamically and adaptively select the most suitable expert network to fuse text and image to generate a description, ultimately outputting fused text modality features. moe =(o1+o2+...+o n By combining the image features output from the linear transformation of the image visual representation, the final general feature recognition result M=(m1+m2+...+m n Finally, the result of this general feature recognition is M=(m1+m2+...+m n The initial multimodal named entity recognition model is trained to obtain a pre-trained multimodal named entity recognition model.

[0167] Furthermore, Conditional Random Fields (CRFs) can achieve higher labeling accuracy in sequence labeling tasks because they consider the correlation between labels in the neighborhood. For example, in part-of-speech tagging tasks, adjectives are more likely to be followed by nouns than verbs, and in named entity recognition with standard BIO2 annotations, I-PER cannot follow B-LOC. Therefore, this invention does not decode each label independently, but rather uses a Conditional Random Field to jointly model them. This module uses M to represent the general input sequence, representing all possible label sequences of the sentence.

[0168] Step 302: Calculate the probability distribution to be trained based on the recognition results of the general features to be trained;

[0169] It should be noted that, for a given sequence, i.e., the general feature recognition result M to be trained, all possible label sequences (the probability distribution to be trained) can be calculated using the following equation:

[0170] ;

[0171] in, The probability distribution to be trained; Let y be a latent function, representing the label y. i-1 The relationship with M; This is the (i-1)th real label; Let Y be the i-th true label; M be the general feature recognition result to be trained; Y be the sequence label set; and T be the length of the sequence. This is the (i-1)th predicted label; Let be the i-th predicted label.

[0172] Step 303: Substitute the probability distribution to be trained into the preset loss function and perform differentiation to determine the model gradient;

[0173] It should be noted that after obtaining the probability distribution to be trained, maximum conditional likelihood estimation is used to train the CRF model, where the preset loss function can be expressed as:

[0174] ;

[0175] in, This is the loss value; Let be the probability distribution to be trained.

[0176] Step 304: Update the initial multimodal named entity recognition model using model gradient, determine the intermediate multimodal named entity recognition model, and count the number of model iterations in real time;

[0177] Step 305: Determine whether the model iteration count has reached the preset training count;

[0178] Step 306: If the condition is met, use the intermediate multimodal named entity recognition model as the pre-trained multimodal named entity recognition model.

[0179] It should be noted that if the number of model iterations does not reach the preset number of training iterations, the intermediate multimodal named entity recognition model is used as the new initial multimodal named entity recognition model, and the process jumps to step 301 until the number of model iterations reaches the preset number of training iterations. The intermediate multimodal named entity recognition model determined when the number of model iterations reaches the preset number of training iterations is used as the trained preset multimodal named entity recognition model. The preset number of training iterations can be set as needed, and this invention is not limited to this.

[0180] Please see Figure 5 , Figure 5 This is a structural block diagram of a multimodal named entity recognition device provided in Embodiment 3 of the present invention.

[0181] The present invention provides a multimodal named entity recognition device, comprising:

[0182] The acquisition module 501 is used to acquire the text data to be detected and the image data to be detected, and input the text data to be detected and the image data to be detected into a preset multimodal named entity recognition model. The preset multimodal named entity recognition model includes a text image description extraction module, an image feature extraction module, a hybrid expert fusion module and a multimodal attention fusion module.

[0183] The input module 502 is used to take the text data to be detected and the image data to be detected as input to the text image description extraction module, and output the text modal features and the image description text modal features;

[0184] Output module 503 is used to extract image features from the image data to be detected using the image feature extraction module and output image modal features;

[0185] The generation module 504 is used to perform feature fusion on text modal features and image description text modal features through the hybrid expert fusion module to generate fused text modal features;

[0186] The fusion module 505 is used to perform attention fusion on the text modal features and image modal features using a multimodal attention fusion module, and output the target general feature recognition result.

[0187] Furthermore, the text image description extraction module includes a convolutional submodule, an autoregressive decoder, a natural language processing model encoder, and a natural language processing model decoder; the input module 502 is specifically used for:

[0188] A convolutional submodule is used to extract high-dimensional features from the image data to be detected, generating high-dimensional image features.

[0189] The high-dimensional features of the image are decoded by an autoregressive decoder, and the image description text is output.

[0190] The image description text and the text data to be detected are respectively input into the encoder of the natural language processing model for encoding, generating the identifier sequence and attention mask corresponding to the image description text, and the identifier sequence and attention mask corresponding to the text data to be detected;

[0191] The identifier sequences corresponding to the image description text and the identifier sequences corresponding to the text data to be detected are marked respectively to generate the identifier marking sequences corresponding to the image description text and the identifier marking sequences corresponding to the text data to be detected.

[0192] The identifier tag sequence and attention mask corresponding to the image description text, and the identifier tag sequence and attention mask corresponding to the text data to be detected are respectively input into the natural language processing model decoder for decoding to generate text modal features and image description text modal features.

[0193] Furthermore, the output module 503 is specifically used for:

[0194] The pixel data of the image to be detected is adjusted to generate an adjusted pixel image;

[0195] The image feature extraction module extracts image features from the adjusted pixel image and outputs a visual representation of the image.

[0196] Perform a linear transformation on the visual representation of the image to output the image modal features.

[0197] Furthermore, the hybrid expert fusion module includes a gating layer, an expert layer, and an attention layer; the generation module 504 is specifically used for:

[0198] Weighted input features are determined based on text modal features and image description text modal features;

[0199] The gating layer performs probability calculations based on weighted input features to determine the selection probability distribution of multiple experts in the expert layer.

[0200] Select any expert whose selection probability distribution is greater than a preset probability threshold as the activated expert;

[0201] The weighted input features are calculated by multiple activation experts, and the expert calculation results corresponding to each activation expert are output.

[0202] An attention layer is used to weight the expert calculation results and selection probability distribution corresponding to each activated expert to determine the weighted result of the mixed experts.

[0203] The weighted results from the hybrid experts and the weighted input features are normalized and fused to generate fused text modal features.

[0204] Furthermore, the multimodal attention fusion module includes a multimodal attention layer and a normalized feedforward network submodule; the fusion module 505 is specifically used for:

[0205] The multimodal attention layer calculates multi-head attention weights based on the fused text modal features and image modal features, and outputs multiple first text image attention weights and multiple second text image attention weights.

[0206] The attention weights of multiple first text images and multiple second text images are concatenated to generate the first text image attention concatenation vector and the second text image attention concatenation vector.

[0207] Linear transformations are performed on the first text image attention concatenation vector and the second text image attention concatenation vector, respectively, to output the first multimodal attention fusion representation and the second multimodal attention fusion representation;

[0208] The normalized feedforward network submodule is used to perform normalized superposition based on the first multimodal attention fusion representation and image modal features to generate the first normalized superposition feature;

[0209] The normalized feedforward network submodule is used to normalize and superimpose the second multimodal attention fusion representation and the fused text modal features to generate the second normalized superimposed features;

[0210] The multimodal attention layer calculates multi-head attention weights based on the fused text modal features and the first normalized superposition features, and outputs multiple text normalized attention weights.

[0211] The normalized attention weights of multiple texts are concatenated and linearly transformed to output a third multimodal attention fusion representation;

[0212] The target text representation is generated by normalizing and superimposing the third multimodal attention fusion representation and the fused text modality features using a normalized feedforward network submodule.

[0213] The second normalized overlay feature and the target text representation are fused to generate a target visual representation;

[0214] The visual representation and text representation of the target are concatenated and linearly transformed to generate a general feature recognition result for the target.

[0215] In one alternative embodiment, the apparatus further includes:

[0216] The first module is used to take the text data to be trained and the image data to be trained as input to the initial multimodal named entity recognition model when it receives the text data to be trained and the image data to be trained, and output the general feature recognition result to be trained.

[0217] The second module is used to calculate the probability distribution to be trained based on the recognition results of the general features to be trained.

[0218] The third module is used to substitute the probability distribution to be trained into the preset loss function and perform differentiation to determine the model gradient.

[0219] The fourth module is used to update the initial multimodal named entity recognition model using model gradients, determine the intermediate multimodal named entity recognition model, and count the number of model iterations in real time.

[0220] The fifth module is used to determine whether the number of model iterations has reached the preset number of training iterations;

[0221] The sixth module is used to, if necessary, use the intermediate multimodal named entity recognition model as a pre-trained multimodal named entity recognition model.

[0222] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described device and module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0223] This invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program; when the computer program is executed by the processor, the processor performs the steps of the multimodal named entity recognition method as described in any of the above embodiments.

[0224] This invention also provides a computer-readable storage medium storing a computer program / instructions thereon, which, when executed by a processor, implements the steps of the multimodal named entity recognition method as described in any of the above embodiments.

[0225] This invention also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the multimodal named entity recognition method as described in any of the above embodiments.

[0226] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0227] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0228] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal named entity recognition method, characterized in that, include: The system acquires text data and image data to be detected, and inputs the text data and image data to be detected into a preset multimodal named entity recognition model. The preset multimodal named entity recognition model includes a text image description extraction module, an image feature extraction module, a hybrid expert fusion module, and a multimodal attention fusion module. The text data to be detected and the image data to be detected are used as inputs to the text-image description extraction module, which outputs text modal features and image description text modal features. The image feature extraction module is used to extract image features from the image data to be detected and output image modal features; The hybrid expert fusion module performs feature fusion on the text modal features and the image description text modal features to generate fused text modal features; The multimodal attention fusion module is used to perform attention fusion on the fused text modal features and the image modal features, and outputs the target general feature recognition result; The hybrid expert fusion module includes a gating layer, an expert layer, and an attention layer; The step of fusing the text modal features and the image description text modal features through the hybrid expert fusion module to generate fused text modal features includes: Based on the text modal features and the image description text modal features, weighted input features are determined; The gating layer performs probability calculations based on the weighted input features to determine the selection probability distribution of multiple experts in the expert layer. Select any expert whose selection probability distribution is greater than a preset probability threshold as the activated expert; The weighted input features are calculated by multiple activation experts, and the expert calculation results corresponding to each activation expert are output. An attention layer is used to perform weighted calculations on the expert calculation results and selection probability distributions corresponding to each activated expert to determine the hybrid expert weighted result. The hybrid expert weighted results and the weighted input features are normalized and fused to generate fused text modal features.

2. The multimodal named entity recognition method according to claim 1, characterized in that, The text-image description extraction module includes a convolutional submodule, an autoregressive decoder, a natural language processing model encoder, and a natural language processing model decoder; the step of taking the text data to be detected and the image data to be detected as inputs to the text-image description extraction module and outputting text modal features and image description text modal features includes: A convolutional submodule is used to extract high-dimensional features from the image data to be detected, generating high-dimensional image features. The high-dimensional features of the image are decoded using an autoregressive decoder to output image description text; The image description text and the text data to be detected are respectively input into the natural language processing model encoder for encoding, generating the identifier sequence and attention mask corresponding to the image description text, and the identifier sequence and attention mask corresponding to the text data to be detected; The identifier sequence corresponding to the image description text and the identifier sequence corresponding to the text data to be detected are marked respectively to generate the identifier mark sequence corresponding to the image description text and the identifier mark sequence corresponding to the text data to be detected. The identifier tag sequence and attention mask corresponding to the image description text, and the identifier tag sequence and attention mask corresponding to the text data to be detected are respectively input into the natural language processing model decoder for decoding to generate text modal features and image description text modal features.

3. The multimodal named entity recognition method according to claim 1, characterized in that, The step of using the image feature extraction module to extract image features based on the image data to be detected and outputting image modal features includes: The image data to be detected is pixel-adjusted to generate an adjusted pixel image; The image feature extraction module extracts image features from the adjusted pixel image and outputs a visual representation of the image. A linear transformation is performed on the visual representation of the image to output the image modal features.

4. The multimodal named entity recognition method according to claim 1, characterized in that, The multimodal attention fusion module includes a multimodal attention layer and a normalized feedforward network submodule; the multimodal attention fusion module is used to perform attention fusion on the fused text modal features and the image modal features, and outputs a target general feature recognition result, including: The multimodal attention layer calculates multi-head attention weights based on the fused text modal features and the image modal features, and outputs multiple first text image attention weights and multiple second text image attention weights. The attention weights of the first text image and the attention weights of the second text image are concatenated to generate the first text image attention concatenation vector and the second text image attention concatenation vector. Linear transformations are performed on the first text image attention concatenation vector and the second text image attention concatenation vector, respectively, to output the first multimodal attention fusion representation and the second multimodal attention fusion representation; The normalized feedforward network submodule is used to perform normalized superposition based on the first multimodal attention fusion representation and the image modal features to generate the first normalized superposition feature; The normalized feedforward network submodule is used to normalize and superimpose the second multimodal attention fusion representation and the fused text modal features to generate the second normalized superimposed feature; The multimodal attention layer calculates multi-head attention weights based on the fused text modal features and the first normalized superposition features, and outputs multiple text normalized attention weights. The normalized attention weights of the multiple texts are concatenated and linearly transformed to output a third multimodal attention fusion representation; The target text representation is generated by normalizing and superimposing the third multimodal attention fusion representation and the fused text modality features using a normalized feedforward network submodule. The second normalized overlay feature and the target text representation are fused to generate a target visual representation; The visual representation and text representation of the target are concatenated and linearly transformed to generate a general feature recognition result for the target.

5. The multimodal named entity recognition method according to claim 1, characterized in that, The training process of the pre-set multimodal named entity recognition model includes: When the text data and image data to be trained are received, the text data and image data to be trained are used as input to the initial multimodal named entity recognition model, and the general feature recognition result to be trained is output. Calculate the probability distribution to be trained based on the recognition results of the general features to be trained; Substitute the probability distribution to be trained into the preset loss function and take its derivative to determine the model gradient; The initial multimodal named entity recognition model is updated using the model gradient to determine the intermediate multimodal named entity recognition model, and the number of model iterations is counted in real time. Determine whether the number of model iterations has reached the preset number of training iterations; If this is achieved, the intermediate multimodal named entity recognition model will be used as the pre-trained multimodal named entity recognition model.

6. A multimodal named entity recognition device, applied to the multimodal named entity recognition method of claim 1, characterized in that, include: The acquisition module is used to acquire the text data to be detected and the image data to be detected, and input the text data to be detected and the image data to be detected into a preset multimodal named entity recognition model. The preset multimodal named entity recognition model includes a text image description extraction module, an image feature extraction module, a hybrid expert fusion module, and a multimodal attention fusion module. The input module is used to take the text data to be detected and the image data to be detected as input to the text image description extraction module, and output text modal features and image description text modal features; The output module is used to extract image features based on the image data to be detected using the image feature extraction module, and output image modal features; The generation module is used to perform feature fusion on the text modal features and the image description text modal features through the hybrid expert fusion module to generate fused text modal features; The fusion module is used to perform attention fusion on the fused text modal features and the image modal features using the multimodal attention fusion module, and output the target general feature recognition result.

7. A computer device, characterized in that, The method includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor causes the processor to perform the steps of the multimodal named entity recognition method as described in any one of claims 1-5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed, it implements the multimodal named entity recognition method as described in any one of claims 1-5.

9. A computer program product, characterized in that, The computer program product includes a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions, wherein when the program instructions are executed by a computer, the computer performs the multimodal named entity recognition method as described in any one of claims 1-5.

Images