Multi-modal named entity identification method and device

Abstract

The invention discloses a multi-modal named entity identification method and device, which are used for solving the technical problem of poor accuracy of multi-modal named entity identification caused by the existing multi-modal named entity identification method. The method comprises the steps of obtaining to-be-detected text data and to-be-detected image data; the to-be-detected text data and the to-be-detected image data serve as input of a text image description extraction module, and text modal features and image description text modal features are output; an image feature extraction module is adopted to perform image feature extraction according to the to-be-detected image data, and image modal features are output; performing feature fusion on the text modal features and the image description text modal features through a hybrid expert fusion module to generate fused text modal features; and carrying out attention fusion on the fused text modal features and the image modal features by adopting a multi-modal attention fusion module, and outputting a target general feature recognition result.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a multimodal named entity recognition method and device. Background Art

[0002] With the rapid development of the field of artificial intelligence, more and more application scenarios require the integration of multi-modal data (such as text, images, audio, video, etc.). Data in different modalities can complement each other and make up for the shortcomings of a single modality in information transmission.

[0003] At present, multimodal named entity recognition (MNER) of text and its accompanying images has attracted widespread attention. Multimodal named entity recognition enables semantic information to be exchanged and converted between modalities by fusing multimodal features such as text and images, and accurately identifies and annotates important concepts, terms and entities (such as names, place names, institutions, etc.) contained in the target modality. In modern education, streaming media technology has promoted the diversification of teaching content, generating a variety of data such as text, video, audio and image. These different modal data can provide complementary information after being fused, eliminating the ambiguity of entities in single-modal data. For example, images and audio in videos can help understand the meaning of text, thereby improving the ability to recognize complex scenes, providing more comprehensive contextual information, and helping to improve the accuracy and robustness of entity recognition tasks. Therefore, feature extraction and fusion of information from multiple modalities has become the main method for current multimodal named entity recognition tasks.

[0004] Most existing multimodal named entity recognition methods use entity-related information from the image modality to assist in the recognition of textual named entities. However, there are usually significant differences in the distribution of text and visual features in the feature space, and the meanings of the text and its accompanying image do not always match, making it difficult for the fused features to effectively capture the correlation between the two. In addition, the semantic features are incomplete, resulting in poor accuracy in multimodal named entity recognition. Summary of the invention

[0005] The present invention provides a multimodal named entity recognition method and device, which are used to solve the technical problem that the accuracy of multimodal named entity recognition is poor due to the existing multimodal named entity recognition method.

[0006] A first aspect of the present invention provides a multimodal named entity recognition method, comprising:

[0007] Acquire text data to be detected and image data to be detected, and input the text data to be detected and the image data to be detected into a preset multimodal named entity recognition model, wherein the preset multimodal named entity recognition model includes a text image description extraction module, an image feature extraction module, a hybrid expert fusion module and a multimodal attention fusion module;

[0008] Using the to-be-detected text data and the to-be-detected image data as inputs of a text image description extraction module, and outputting text modality features and image description text modality features;

[0009] The image feature extraction module is used to extract image features according to the image data to be detected, and output image modal features;

[0010] The text modality feature and the image description text modality feature are subjected to feature fusion by the hybrid expert fusion module to generate a fused text modality feature;

[0011] The multimodal attention fusion module is used to perform attention fusion on the fused text modal features and the image modal features, and output a target common feature recognition result.

[0012] Optionally, the text image description extraction module includes a convolution submodule, an autoregressive decoder, a natural language processing model encoder and a natural language processing model decoder; the step of taking the to-be-detected text data and the to-be-detected image data as inputs of the text image description extraction module and outputting text modality features and image description text modality features includes:

[0013] Using a convolution submodule to extract high-dimensional features from the image data to be detected, and generate high-dimensional features of the image;

[0014] Decoding the high-dimensional features of the image through an autoregressive decoder and outputting image description text;

[0015] Input the image description text and the text data to be detected into a natural language processing model encoder for encoding, respectively, to generate an identifier sequence and an attention mask corresponding to the image description text, and an identifier sequence and an attention mask corresponding to the text data to be detected;

[0016] Marking the identifier sequence corresponding to the image description text and the identifier sequence corresponding to the text data to be detected respectively, to generate an identifier marking sequence corresponding to the image description text and an identifier marking sequence corresponding to the text data to be detected;

[0017] The identifier tag sequence and attention mask corresponding to the image description text and the identifier tag sequence and attention mask corresponding to the text data to be detected are respectively input into the natural language processing model decoder for decoding to generate text modality features and image description text modality features.

[0018] Optionally, the using the image feature extraction module to extract image features according to the image data to be detected and outputting image modality features includes:

[0019] Performing pixel adjustment on the image data to be detected to generate an adjusted pixel image;

[0020] Performing image feature extraction on the adjusted pixel image by the image feature extraction module, and outputting an image visual representation;

[0021] Perform a linear transformation on the image visual representation and output image modality features.

[0022] Optionally, the hybrid expert fusion module includes a gating layer, an expert layer, and an attention layer; the step of fusing the text modality feature and the image description text modality feature through the hybrid expert fusion module to generate a fused text modality feature includes:

[0023] Determining weighted input features based on the text modality features and the image description text modality features;

[0024] Performing probability calculation according to the weighted input features through the gating layer to determine the selection probability distribution corresponding to the multiple experts in the expert layer;

[0025] Select any expert corresponding to the selection probability distribution greater than the preset probability threshold as the activated expert;

[0026] Perform expert calculations on the weighted input features respectively by the plurality of activated experts, and output expert calculation results corresponding to the activated experts;

[0027] Using an attention layer to perform a weighted operation on the expert calculation results and selection probability distribution corresponding to each of the activated experts to determine a mixed expert weighted result;

[0028] The mixed expert weighted result and the weighted input feature are normalized and fused to generate a fused text modality feature.

[0029] Optionally, the multimodal attention fusion module includes a multimodal attention layer and a normalized feedforward network submodule; the multimodal attention fusion module is used to perform attention fusion on the fused text modality features and the image modality features, and output a target general feature recognition result, including:

[0030] Performing multi-head attention weight calculation according to the fused text modality feature and the image modality feature through a multimodal attention layer, and outputting a plurality of first text-image attention weights and a plurality of second text-image attention weights;

[0031] Respectively splicing a plurality of the first text image attention weights and a plurality of the second text image attention weights to generate a first text image attention splicing vector and a second text image attention splicing vector;

[0032] Performing linear transformation on the first text image attention splicing vector and the second text image attention splicing vector respectively, and outputting a first multimodal attention fusion representation and a second multimodal attention fusion representation;

[0033] Using a normalized feedforward network submodule to perform normalized superposition according to the first multimodal attention fusion representation and the image modality feature to generate a first normalized superposition feature;

[0034] Using a normalized feedforward network submodule to perform normalized superposition according to the second multimodal attention fusion representation and the fused text modality feature to generate a second normalized superposition feature;

[0035] Performing multi-head attention weight calculation according to the fused text modality feature and the first normalized superposition feature through a multimodal attention layer, and outputting multiple text normalized attention weights;

[0036] Concatenate and linearly change the normalized attention weights of the multiple texts, and output a third multimodal attention fusion representation;

[0037] Using a normalized feedforward network submodule to perform normalized superposition according to the third multimodal attention fusion representation and the fused text modality features to generate a target text representation;

[0038] fusing the second normalized superposition feature and the target text representation to generate a target visual representation;

[0039] The target visual representation and the target text representation are concatenated and linearly transformed to generate a target general feature recognition result.

[0040] Optionally, the training process of the preset multimodal named entity recognition model includes:

[0041] When receiving the text data to be trained and the image data to be trained, using the text data to be trained and the image data to be trained as inputs of an initial multimodal named entity recognition model, and outputting a general feature recognition result to be trained;

[0042] Calculating the probability distribution to be trained according to the recognition result of the general feature to be trained;

[0043] Substituting the probability distribution to be trained into a preset loss function and performing derivation to determine the model gradient;

[0044] The initial multimodal named entity recognition model is updated using the model gradient, an intermediate multimodal named entity recognition model is determined, and the number of model iterations is counted in real time;

[0045] Determine whether the number of model iterations reaches a preset number of training times;

[0046] If achieved, the intermediate multimodal named entity recognition model is used as the trained preset multimodal named entity recognition model.

[0047] A second aspect of the present invention provides a multimodal named entity recognition device, comprising:

[0048] An acquisition module, used for acquiring text data to be detected and image data to be detected, and inputting the text data to be detected and the image data to be detected into a preset multimodal named entity recognition model, wherein the preset multimodal named entity recognition model includes a text image description extraction module, an image feature extraction module, a hybrid expert fusion module and a multimodal attention fusion module;

[0049] An input module, used to use the to-be-detected text data and the to-be-detected image data as inputs of a text image description extraction module, and output text modality features and image description text modality features;

[0050] An output module, used to extract image features according to the image data to be detected by using the image feature extraction module, and output image modal features;

[0051] A generating module, used for performing feature fusion on the text modality feature and the image description text modality feature through the hybrid expert fusion module to generate a fused text modality feature;

[0052] A fusion module is used to use the multimodal attention fusion module to perform attention fusion on the fused text modal features and the image modal features, and output a target common feature recognition result.

[0053] A third aspect of the present invention provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the multimodal named entity recognition method as described in any one of the above items.

[0054] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed, the steps of the multimodal named entity recognition method as described in any one of the above items are implemented.

[0055] A fifth aspect of the present invention provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions, wherein when the program instructions are executed by a computer, the computer performs the steps of the multimodal named entity recognition method as described in any one of the above items.

[0056] It can be seen from the above technical solutions that the present invention has the following advantages:

[0057] The above technical solution of the present invention provides a multimodal named entity recognition method, which first obtains text data to be detected and image data to be detected, and inputs the text data to be detected and the image data to be detected into a preset multimodal named entity recognition model, wherein the preset multimodal named entity recognition model includes a text image description extraction module, an image feature extraction module, a hybrid expert fusion module and a multimodal attention fusion module; then, the text data to be detected and the image data to be detected are used as inputs of the text image description extraction module, and text modal features and image description text modal features are output; an image feature extraction module is used to extract image features according to the image data to be detected, and image modal features are output; and feature fusion of the text modal features and the image description text modal features is performed through the hybrid expert fusion module to generate a fused text modal features; finally, a multimodal attention fusion module is used to fuse the fused text modal features and image modal features, and the target general feature recognition result is output; based on the above scheme, a text image description extraction module and an image feature extraction module are used to process the acquired text data to be detected and the image data to be detected, and the generated text modal features and image description text modal features are fused by a hybrid expert fusion module to generate fused text modal features, and then the fused text modal features and image modal features are fused by a multimodal attention fusion module to output the target general feature recognition result. This process can greatly enhance the semantic information of the text features, making the semantic features more complete and consistent during the transmission process, thereby improving the accuracy of multimodal named entity recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0059] Figure 1 A flowchart of a multimodal named entity recognition method provided in Example 1 of the present invention;

[0060] Figure 2 A schematic diagram of the structure of a preset multimodal named entity recognition model provided in the first embodiment of the present invention;

[0061] Figure 3 A flowchart of the steps of the model training process of the preset multimodal named entity recognition model provided in the second embodiment of the present invention;

[0062] Figure 4 A training framework diagram of a preset multimodal named entity recognition model provided in the second embodiment of the present invention;

[0063] Figure 5 A structural block diagram of a multimodal named entity recognition device provided in Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0064] The embodiments of the present invention provide a multimodal named entity recognition method and device, which are used to solve the technical problem that the accuracy of multimodal named entity recognition is poor due to the existing multimodal named entity recognition method.

[0065] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0066] See also Figure 1 , Figure 1 A flowchart of the steps of a multimodal named entity recognition method provided in Example 1 of the present invention.

[0067] The present invention provides a multimodal named entity recognition method, comprising:

[0068] Step 101, obtain the text data to be detected and the image data to be detected, and input the text data to be detected and the image data to be detected into a preset multimodal named entity recognition model, which includes a text image description extraction module, an image feature extraction module, a hybrid expert fusion module and a multimodal attention fusion module.

[0069] Please note that Figure 2The preset multimodal named entity recognition model includes a text image description extraction module, an image feature extraction module, a hybrid expert fusion module, a multimodal attention fusion module and a CRF entity labeling module; the text image description extraction module includes a text feature extraction module and an image description generation module, the text feature extraction module includes a natural language processing model encoder and a natural language processing model decoder, and the image description generation module includes a convolution submodule and an autoregressive decoder; the CRF entity labeling module is used for model training to obtain a trained preset multimodal named entity recognition model.

[0070] Step 102: Use the to-be-detected text data and the to-be-detected image data as inputs of a text image description extraction module, and output text modal features and image description text modal features.

[0071] The text image description extraction module includes a convolution submodule, an autoregressive decoder, a natural language processing model encoder, and a natural language processing model decoder.

[0072] Specifically, the process of using the to-be-detected text data and the to-be-detected image data as the input of the text image description extraction module and outputting the text modality features and the image description text modality features can be implemented by executing the following sub-steps S11 to S15:

[0073] Step S21, using a convolution submodule to extract high-dimensional features from the image data to be detected, to generate high-dimensional features of the image;

[0074] Step S22, decoding the high-dimensional features of the image through an autoregressive decoder, and outputting image description text;

[0075] It should be noted that the present invention uses a pre-trained BLIP (Bootstrapping Language-Image Pre-training) model to perform image description generation tasks, which can efficiently generate a natural language description for an image that combines its context. The model combines visual and language information, understands the image content and generates relevant text descriptions through multimodal learning. The BLIP model first extracts features from the input image, and uses its built-in convolutional network through the convolution submodule to extract high-dimensional features of the image (image high-dimensional features). These features can effectively capture information such as the structure, color, and texture of the image, forming a deep understanding of the image content.

[0076] Furthermore, after feature extraction, BLIP uses an autoregressive decoder (autoregressive decoder) to gradually generate description text, i.e., image description text. The decoder predicts the next most likely word based on the current image features and the generated words. The generation process starts from a start tag and generates word by word until the end tag is encountered or the maximum length limit is reached. The generated description is usually natural and fluent. BLIP is pre-trained on large-scale image-text pairs, so that it has good generalization ability, can adapt to different types of images and scenes, and generate highly accurate and rich text descriptions. Compared with traditional methods, BLIP not only reduces additional feature engineering work, but also makes the entire image description generation process smoother and more efficient.

[0077] Step S23, inputting the image description text and the text data to be detected into the natural language processing model encoder for encoding, generating an identifier sequence and an attention mask corresponding to the image description text, and an identifier sequence and an attention mask corresponding to the text data to be detected;

[0078] Step S24, marking the identifier sequence corresponding to the image description text and the identifier sequence corresponding to the text data to be detected respectively, to generate an identifier marking sequence corresponding to the image description text and an identifier marking sequence corresponding to the text data to be detected;

[0079] Step S25: input the identifier tag sequence and attention mask corresponding to the image description text, and the identifier tag sequence and attention mask corresponding to the text data to be detected into the natural language processing model decoder for decoding, and generate text modal features and image description text modal features.

[0080] It should be noted that multimodal named entity recognition usually involves two main modal data, text (S) and image (I). In the present invention, another text modal data, image description text (C), is introduced. To represent the input feature data and dimensions of the three modes, where The input modality includes images, text, and image description text. is the sequence length corresponding to the input modality, is the feature dimension corresponding to the input modality.

[0081] Furthermore, there are two types of modal data that need to be extracted for text features in the present invention, namely text and text image description. In order to process text data, the present invention adopts a pre-trained BERT model to extract text features. BERT is a pre-trained model based on the Transformer architecture. It is first pre-trained on a large amount of unlabeled text data, and then fine-tuned on the labeled data of a specific task to better adapt to different application scenarios. Using the pre-trained bert-base-cased model, the word segmenter will also retain the case information of the word, which is very important for the named entity recognition task. The word segmenter will split the original text into sentences and words, process punctuation, and clean the text to eliminate irrelevant characters and formats.

[0082] Specifically, the image description text and the text data to be detected are respectively input into the natural language processing model encoder of the bert-base-cased model for encoding. Each input text will be converted into an ID sequence containing the corresponding vocabulary index. At the same time, the corresponding attention mask is also generated to indicate which words are actual input words and which are padding positions. Then, the identifier sequence corresponding to the image description text and the identifier sequence corresponding to the text data to be detected are marked respectively, and two special tags are added to mark the beginning and end of the sentence respectively to help the model better understand the context structure of the sequence. The start tag ([CLS]) is usually used as the start tag of the sentence, and the end tag ([SEP]) is used as the end tag of the sentence. Subsequently, the text-converted ID sequence and mask are input into the natural language processing model decoder of the BERT model. The self-attention mechanism of multiple Transformer encoder layers is stacked to capture the relationship between words in the input sequence, and finally the contextual embedding of each input word is obtained, that is, the text modality features and image description text modality features are generated.

[0083] Furthermore, assume that in the Twitter2015 and Twitter2017 datasets, the dimension of text (S) is ,in , The dimension of the corresponding description text generated based on the images in the dataset is ,in , The final text feature (text modality feature) is S=(s1+s2+...+s 128 ), the image generation description feature (image description text modality feature) is C=(c1+c2+...+c 128), where s1 is the feature embedding of the first text word in the text sequence, s2 is the feature embedding of the second text word in the text sequence, and s 128 is the feature embedding of the 128th text word in the text sequence, c1 is the feature embedding of the first text description in the image description text sequence, c2 is the feature embedding of the second text description in the image description text sequence, and c 128 Feature embedding for the 128th text description in the image description text sequence.

[0084] Step 103: Use an image feature extraction module to extract image features based on the image data to be detected, and output image modal features.

[0085] Specifically, the process of using the image feature extraction module to extract image features according to the image data to be detected and outputting the image modal features can be achieved by executing the following sub-steps S31 to S32:

[0086] Step S31, performing pixel adjustment on the image data to be detected to generate an adjusted pixel image;

[0087] Step S32, extracting image features from the adjusted pixel image through an image feature extraction module, and outputting an image visual representation;

[0088] Step S33: perform linear transformation on the image visual representation and output image modal features.

[0089] The image feature extraction module adopts the residual network (ResNet) structure.

[0090] It should be noted that as one of the most advanced image recognition CNN models, the residual network (ResNet) has demonstrated its ability to extract meaningful feature representations of input images at a deep level. Therefore, the present invention retains the output of the last convolutional layer of the pre-trained 152-layer ResNet to represent each image. This actually divides each input image into 49 visual blocks of the same size and represents each block with a 2048-dimensional vector.

[0091] Specifically, given an input image I (image data to be detected), first resize it to 224×224 pixels and input it into ResNet to obtain its visual representation (image visual representation), expressed as I=(i1+i2+...+i 49 ), where i u is the representation of the uth visual block of the 2048-dimensional vector, and i1 is the representation of the 1st visual block of the 2048-dimensional vector. In order to project the visual representation into the same distribution space of the text representation, the present invention further transforms I using a linear transformation: ,in is the weight matrix, The value is 768. Therefore, V=(v1+v2+...+v 49 ) is the generated image feature (image modality feature).

[0092] Step 104: Fusing the text modality features and the image description text modality features through a hybrid expert fusion module to generate fused text modality features.

[0093] The hybrid expert fusion module includes a gating layer, an expert layer, and an attention layer.

[0094] It should be noted that the hybrid expert model is used to dynamically and adaptively select the most appropriate expert network to fuse text and image to generate descriptions, capture the correlation between different modalities, narrow the distribution gap between text features and visual features, enhance the consistency of text semantic features during network transmission, and improve the performance of multimodal named entity recognition.

[0095] Specifically, the process of fusing the text modality features and the image description text modality features through the hybrid expert fusion module to generate the fused text modality features can be achieved by executing the following sub-steps S41 to S46:

[0096] Step S41, determining weighted input features based on text modality features and image description text modality features;

[0097] It should be noted that the received text features and image description features (text modality features and image description text modality features) are used as inputs. Since the text generated by the image may be more blurred or noisy than the direct text input, the contribution weights of the two inputs can be adaptively adjusted by learning parameters to reduce the impact of poor quality input on the final result and rely more on high quality input. Therefore, the processing of weighted input (weighted input features) can be expressed as:

[0098] ;

[0099] in, is the first learnable parameter; X is the weighted input feature; S is the text modality feature; C is the image description text modality feature; is the second learnable parameter.

[0100] Step S42: performing probability calculation according to the weighted input features through the gating layer to determine the selection probability distribution corresponding to the multiple experts in the expert layer;

[0101] It should be noted that the selection probability distribution of each expert is calculated for each input through the routing in the module, that is, the gating layer calculates the selection probability distribution corresponding to each expert in the expert layer. The processing process of the selection probability distribution can be expressed as:

[0102] ;

[0103] Among them, Route is the selection probability distribution; is the softmax activation function; is the weight matrix corresponding to the gating layer; is the bias term corresponding to the gating layer; X is the weighted input feature;

[0104] Step S43, selecting any expert corresponding to a selection probability distribution greater than a preset probability threshold as an activated expert;

[0105] It should be noted that any expert corresponding to a selection probability distribution greater than a preset probability threshold is regarded as an activated expert, while an expert corresponding to a selection probability distribution less than or equal to the preset probability threshold is not activated; wherein, the preset probability threshold can be set as needed, and the present invention is not limited thereto.

[0106] Step S44, performing expert calculations on the weighted input features respectively through multiple activated experts, and outputting expert calculation results corresponding to each activated expert;

[0107] It should be noted that each activated expert will independently calculate the weighted input and generate the corresponding output (expert calculation result) , where the processing process of expert calculation results can be expressed as:

[0108] ;

[0109] in, The expert calculation result corresponding to the i-th activated expert; is the second weight matrix corresponding to the i-th activated expert; is the ReLU activation function; is the first weight matrix corresponding to the i-th activated expert; is the first bias term corresponding to the i-th activated expert; is the second bias term corresponding to the i-th activated expert; X is the weighted input feature.

[0110] Step S45: Use the attention layer to perform weighted operations on the expert calculation results and selection probability distributions corresponding to each activated expert to determine the mixed expert weighted results;

[0111] It should be noted that according to the selection probability distribution, the outputs of all experts are weighted, that is, the expert calculation results and the selection probability distribution corresponding to each activated expert are weighted by the attention layer to determine the mixed expert weighted result; the processing process of the mixed expert weighted result is specifically as follows:

[0112] ;

[0113] in, Weighting results for hybrid experts; is the selection probability distribution corresponding to the i-th activated expert; The expert calculation result corresponding to the i-th activated expert; is the number of activated experts.

[0114] Step S46: normalize and fuse the mixed expert weighted results and the weighted input features to generate fused text modal features.

[0115] It should be noted that in order to alleviate the gradient vanishing and gradient exploding problems and allow the model to learn more complex representations without losing useful information of the original input, residual connections are added to the model, allowing the network to not only rely on deep abstract features but also retain some information of the original input. In addition, the output of each layer is normalized through LayerNorm to maintain the stability of the input, thereby improving the training speed and the generalization ability of the model.

[0116] Furthermore, the process of fusing text modal features can be expressed as:

[0117] ;

[0118] in, is the fusion text modality feature; X is the weighted input feature; Weighting results for hybrid experts; Normalize the input to the layer to mean; The standard deviation of the normalized input to the layer; is a learnable scaling parameter; is a learnable offset parameter.

[0119] Step 105: Use a multimodal attention fusion module to perform attention fusion on the fused text modal features and image modal features, and output the target common feature recognition result.

[0120] The multimodal attention fusion module includes a multimodal attention layer and a normalized feedforward network submodule; the normalized feedforward network submodule includes a normalization layer and a feedforward layer, and the structure of the feedforward layer is a feedforward network.

[0121] It should be noted that in order to make the fusion of image and text modal data more effective, the present invention has enhanced the semantic features of text in the hybrid expert module, which can alleviate the contextual bias problem that may exist when fusing visual features. Therefore, the multimodal attention fusion module only needs to combine the image modal feature V with the fused text modal Output with enhanced semantic features. moeIn addition, in order to enhance the fusion again and learn better feature representation, a total of three multimodal attention fusions were performed.

[0122] Specifically, the process of using the multimodal attention fusion module to perform attention fusion on the fused text modal features and the image modal features and outputting the target general feature recognition result can be achieved by executing the following sub-steps S51 to S510:

[0123] Step S51: Calculate multi-head attention weights according to the fused text modality features and image modality features through a multimodal attention layer, and output a plurality of first text image attention weights and a plurality of second text image attention weights;

[0124] It should be noted that in order to better integrate visual features with text features, let the visual representation guide the learning of text representation first, learn better text features with visual context, and allow the model to refer to related image information when processing text, thereby improving the representation ability of words. The image feature V input to the module is used as the query, and the text feature Output moe As the key and value, calculate the first text image attention weight corresponding to the current attention head of the two modal data,

[0125] Furthermore, in order to obtain a visual representation related to each token in the text, each word is aligned with its closely related visual blocks, and high or low attention weights are assigned to its related or unrelated visual blocks. The multimodal attention fusion mechanism is used again to align the text feature Output moe As the query, the image feature V is used as the key and value, and the second text image attention weight corresponding to the two modal data in the current attention head is calculated.

[0126] Among them, the calculation process of the first text image attention weight can be expressed as:

[0127] ;

[0128] in, is the attention weight of the i-th first text image, representing the i-th attention head; is the weight matrix corresponding to the query matrix; is the weight matrix corresponding to the key-value matrix; To integrate text modality features; is the image modality feature; d is the dimension of the text 768; m is the number of attention heads; is the weight matrix corresponding to the value matrix; T is the transpose.

[0129] It is worth mentioning that the processing process of the second text image attention weight is consistent with the calculation process of the first text image attention weight, and the present invention will not go into details.

[0130] Step S52: splicing the plurality of first text image attention weights and the plurality of second text image attention weights respectively to generate a first text image attention splicing vector and a second text image attention splicing vector;

[0131] Step S53, performing linear transformation on the first text image attention splicing vector and the second text image attention splicing vector respectively, and outputting a first multimodal attention fusion representation and a second multimodal attention fusion representation;

[0132] It should be noted that concatenating the attention weights calculated by each attention head can capture the diverse attention of different heads to the input. Finally, the concatenated result is multiplied by a weight matrix (such as W') to perform a linear transformation, learn the relationship between different attention heads, reduce the redundant information that may be generated after concatenation, fuse the features of multiple head outputs, and generate the final multimodal attention fusion representation.

[0133] Among them, the processing process of the first multimodal attention fusion representation can be expressed as:

[0134] ;

[0135] in, It is the first multimodal attention fusion representation; is the first text image attention weight, indicating the first attention head; is the mth first text image attention weight, representing the mth attention head; is the weight matrix; To integrate text modality features; is the image modality feature.

[0136] It is worth mentioning that the processing process of the second multimodal attention fusion representation is consistent with the processing process of the first multimodal attention fusion representation, and the present invention will not go into details.

[0137] Step S54: using a normalized feedforward network submodule to perform normalized superposition according to the first multimodal attention fusion representation and the image modality feature to generate a first normalized superposition feature;

[0138] It should be noted that this multimodal attention fusion representation is input into three transformer sub-layers (normalized feedforward network sub-modules), providing a richer and more meaningful feature representation, allowing the model to converge to the optimal solution faster.

[0139] Among them, the processing process of the first normalized superposition feature can be expressed as:

[0140] ;

[0141] in, is the normalized feature; Normalize the layer; is a feedforward network; P is the first normalized superposition feature, P=(p1+p2+...+p 49 ), p1 is, p2 is, p 49 for; To integrate text modality features; is the image modality feature.

[0142] Step S55, using a normalized feedforward network submodule to perform normalized superposition according to the second multimodal attention fusion representation and the fused text modality feature to generate a second normalized superposition feature;

[0143] It should be noted that the second normalized superposition feature Q=(q1+q2+...+q n ) is consistent with the processing flow of the first normalized superposition feature processing process, and the present invention will not go into details.

[0144] Step S56: Calculate multi-head attention weights according to the fused text modal features and the first normalized superposition features through the multimodal attention layer, and output multiple text normalized attention weights;

[0145] Step S57: concatenate and linearly change the normalized attention weights of multiple texts, and output a third multimodal attention fusion representation;

[0146] Step S58, using a normalized feedforward network submodule to perform normalized superposition according to the third multimodal attention fusion representation and the fused text modality features to generate a target text representation;

[0147] It should be noted that since each generated vector p i corresponds to the i-th visual block rather than the i-th input word. Therefore, it is necessary to perform multimodal attention fusion on P again to fuse the text modality feature Output moe As the query, the first normalized superposition feature P is used as the key and value to generate the final target text representation T = (t1 + t2 + ... + t n ), wherein the processing flow of the text normalized attention weight, the third multimodal attention fusion representation, and the target text representation is consistent with the processing flow of the first text image attention weight, the first multimodal attention fusion representation, and the first normalized superposition feature, and the present invention will not go into details.

[0148] Step S59: fusing the second normalized superposition feature and the target text representation to generate a target visual representation;

[0149] It should be noted that the visual gate is used to dynamically control the contribution of visual features to obtain the final visual representation, that is, the target visual representation Q', where the processing flow of the target visual representation can be expressed as:

[0150] ;

[0151] in, is a visual representation of the target; is the sigmoid activation function; T is the target text representation; is the transpose operation; is the second weight matrix; Q is the second normalized superposition feature; is the first weight matrix.

[0152] Step S510: concatenate and linearly transform the target visual representation and the target text representation to generate a target general feature recognition result.

[0153] It should be noted that the target visual representation Q' and the target text representation T are concatenated and linearly transformed to obtain the fused general feature (target general feature recognition result) M, where the processing process of the target general feature recognition result can be expressed as:

[0154] ;

[0155] Among them, M is the target general feature recognition result; T is the target text representation; is a visual representation of the target; is the third weight matrix.

[0156] Furthermore, the scores measured by the present invention on the processed data set are shown in Table 1 below. After experimental verification, the present invention has achieved certain results in the multimodal named entity recognition task.

[0157] Table 1 Experimental verification

[0158]

[0159] As a comparison of technical effects, it can be combined with existing technologies for reference. With the rapid development of the field of artificial intelligence, more and more application scenarios need to integrate multi-modal data (such as text, images, audio, video, etc.). Data of different modalities can complement each other and make up for the shortcomings of a single modality in information transmission. As social media posts tend to be multimodal, multimodal named entity recognition (MNER) of text and its accompanying images has attracted widespread attention. Multimodal named entity recognition enables semantic information to be exchanged and converted between modalities by fusing multimodal features such as text and images, and accurately identifies and annotates important concepts, terms and entities (such as names, place names, institutions, etc.) contained in the target modality. In modern education, streaming media technology has promoted the diversification of teaching content and generated a variety of data such as text, video, audio and image. These different modal data can provide complementary information after being fused with each other, eliminating the ambiguity of entities in single-modal data. For example, images and audio in videos can help understand the meaning in text, thereby improving the ability to recognize complex scenes, providing more comprehensive contextual information, and helping to improve the accuracy and robustness of entity recognition tasks. Therefore, feature extraction and fusion of information from multiple modalities has become the main method for current multimodal named entity recognition tasks.

[0160] However, this approach of combining multimodal data for modal named entity recognition tasks faces two major problems. First, although existing work can generate word-aware visual representations, these representations are insensitive to visual context and ignore the bias that may be brought by visual context, resulting in inaccurate information transfer. Second, there are usually significant differences in the distribution of text and visual features in the feature space, and the meaning of text and its accompanying image do not always match, making it difficult for the fused features to effectively capture the correlation between the two, resulting in incomplete semantic features.

[0161] In view of the above problems, the present invention proposes a multimodal named entity recognition method. The present invention adds an image description generation task, which aims to extract features from the input image, generate a concise and accurate text description, provide certain contextual information, and effectively alleviate the bias caused by visual representation. In addition, in order to solve the semantic loss problem caused by the mismatch between the meaning of text and image, the present invention adopts a hybrid expert model. The hybrid expert model adaptively selects the most suitable expert network based on the description text features generated by the text features and the image, captures the subtle differences between the text and the image generated text, captures the correlation between different modalities, narrows the gap between different modal features in the representation space, and greatly enhances the semantic information of the text features, making the semantic features more complete and consistent during the transmission process. The present invention can effectively solve the problem of too large a distribution gap between the two modal features of text and image, adaptively enhance the semantic information of text features by generating text from images, alleviate the visual bias problem caused by the fusion of image and text features, and can achieve excellent performance in the multimodal named entity recognition task.

[0162] In summary, in the multimodal named entity recognition task, the entities existing in the target modality are captured based on the two modal data of text and image. Traditional multimodal named entity recognition methods mainly focus on modal fusion, interaction modeling and noise filtering, and introduce attention mechanism and gating mechanism to enhance the synergy between text and visual information. However, this method of fusing different modal features through multimodal attention mechanism may have problems such as insufficient information interaction between modalities, missing semantic feature information and noise influence. In order to solve the problem that the visual representation of word perception is insensitive to the visual context, the present invention introduces the image description generation task to generate a text related to the visual features of the image as supplementary information of the visual features. In addition, considering the informal and short text in the social media dataset, the hybrid expert model is used to dynamically and adaptively select the most appropriate expert network to fuse text and image to generate descriptions, capture the correlation between different modalities, narrow the distribution gap between text features and visual features, enhance the consistency of text semantic features in the network transmission process, and improve the performance of multimodal named entity recognition. We conducted experiments on Twitter2015 and Twitter2017 datasets to evaluate the stability and effectiveness of our proposed method. Both Twitter2015 and Twitter2017 datasets are tweets collected from the Twitter platform, mainly including multimodal user posts published on Twitter during 2014-2015 and 2016-2017, focusing on tweets with pictures. However, compared with Twitter2015, the Twitter2017 dataset contains richer content and more samples. These two datasets usually use the BIO2 annotation method, which can effectively process various types of named entities. In addition, the present invention fuses text and image generation description features through a hybrid expert model, so that the language text features are supplemented with the image-based context representation before being fused with the image features, and the consistency of semantic features in the information flow process is enhanced, thereby improving the accuracy and robustness of named entity recognition.

[0163] In an embodiment of the present invention, the present invention provides a multimodal named entity recognition method, firstly, obtaining text data to be detected and image data to be detected, and inputting the text data to be detected and the image data to be detected into a preset multimodal named entity recognition model, wherein the preset multimodal named entity recognition model comprises a text image description extraction module, an image feature extraction module, a hybrid expert fusion module and a multimodal attention fusion module; then, the text data to be detected and the image data to be detected are used as inputs of the text image description extraction module, and text modal features and image description text modal features are output; an image feature extraction module is used to extract image features according to the image data to be detected, and image modal features are output; and feature fusion of the text modal features and the image description text modal features is performed through the hybrid expert fusion module to generate a fused text image. this modal feature; finally, a multimodal attention fusion module is used to perform attention fusion on the fused text modal features and image modal features, and the target general feature recognition result is output; based on the above scheme, a text image description extraction module and an image feature extraction module are used to process the acquired text data to be detected and the image data to be detected, and the generated text modal features and image description text modal features are combined through a hybrid expert fusion module to perform feature fusion to generate a fused text modal feature, and then the fused text modal features and image modal features are performed attention fusion by a multimodal attention fusion module to output the target general feature recognition result. This process can greatly enhance the semantic information of the text features, making the semantic features more complete and consistent during the transmission process, thereby improving the accuracy of multimodal named entity recognition.

[0164] For better explanation, refer to Figure 3 , a flowchart of the steps of the model training process of the preset multimodal named entity recognition model provided in the second embodiment of the present invention, the process may include the following steps:

[0165] Step 301: When receiving the text data to be trained and the image data to be trained, use the text data to be trained and the image data to be trained as inputs of an initial multimodal named entity recognition model, and output a general feature recognition result to be trained;

[0166] Please note that Figure 3 The training framework of the preset multimodal named entity recognition model proposed in the present invention specifically includes: encoding the input text data (encoder), extracting features, and obtaining text features (text modality features) as S=(s1+s2+...+s n), the input image is processed by the pre-trained BLIP (Bootstrapping Language-Image Pre-training) model to obtain the subtitles (image description text), and then after encoding and feature extraction, the image generation description feature (image description text modality feature) can be obtained as C=(c1+c2+...+c n ), extract features from the input image through ResNet, and obtain the visual representation of the image, expressed as I=(i1+i2+...+i49). The hybrid expert model dynamically and adaptively selects the most appropriate expert network to fuse text and image to generate descriptions, and finally outputs the fused text modality feature Output moe =(o1+o2+...+o n ), and combined with the image features output by linear transformation of the image visual representation, the final universal feature recognition result M=(m1+m2+...+m n ); Finally, the universal feature recognition result M=(m1+m2+...+m n ), train the initial multimodal named entity recognition model to obtain a trained preset multimodal named entity recognition model.

[0167] Furthermore, conditional random fields (CRFs) can produce higher tagging accuracy in sequence tagging tasks because CRFs take into account the correlation between tags in the neighborhood. For example, in part-of-speech tagging tasks, adjectives are more likely to be followed by nouns than verbs, and in named entity recognition with standard BIO2 annotations, I-PER cannot be followed by B-LOC. Therefore, instead of decoding each tag independently, the present invention jointly models them using conditional random fields. This module uses M to represent the universal input sequence, representing all possible tag sequences for a sentence.

[0168] Step 302: Calculate the probability distribution to be trained according to the recognition result of the general feature to be trained;

[0169] It should be noted that for a given sequence, that is, the general feature recognition result M to be trained, all possible label sequences (probability distribution to be trained) can be calculated by the following equation:

[0170] ;

[0171] in, is the probability distribution to be trained; is the potential function, representing the label y i-1 Relationship with M; is the i-1th true label; is the i-th true label; M is the general feature recognition result to be trained; Y is the sequence label set; T is the length of the sequence; is the i-1th predicted label; is the i-th predicted label.

[0172] Step 303: Substitute the probability distribution to be trained into the preset loss function and perform derivation to determine the model gradient;

[0173] It should be noted that after obtaining the probability distribution to be trained, the maximum conditional likelihood estimation is used to train the CRF model, where the preset loss function can be expressed as:

[0174] ;

[0175] in, is the loss value; is the probability distribution to be trained.

[0176] Step 304: using the model gradient to update the initial multimodal named entity recognition model, determining the intermediate multimodal named entity recognition model, and counting the number of model iterations in real time;

[0177] Step 305: determine whether the number of model iterations reaches the preset number of training times;

[0178] Step 306: If reached, use the intermediate multimodal named entity recognition model as the trained preset multimodal named entity recognition model.

[0179] It should be noted that if the number of model iterations reaches the preset number of training times in the future, the intermediate multimodal named entity recognition model will be used as the new initial multimodal named entity recognition model, and the process will jump to step 301 until the number of model iterations reaches the preset number of training times, and the intermediate multimodal named entity recognition model determined when the number of model iterations reaches the preset number of training times will be used as the trained preset multimodal named entity recognition model; wherein, the preset number of training times can be set as needed, and the present invention is not limited to this.

[0180] See also Figure 5 , Figure 5 A structural block diagram of a multimodal named entity recognition device provided in Embodiment 3 of the present invention.

[0181] The present invention provides a multimodal named entity recognition device, comprising:

[0182] An acquisition module 501 is used to acquire text data to be detected and image data to be detected, and input the text data to be detected and the image data to be detected into a preset multimodal named entity recognition model, wherein the preset multimodal named entity recognition model includes a text image description extraction module, an image feature extraction module, a hybrid expert fusion module and a multimodal attention fusion module;

[0183] An input module 502, used to use the to-be-detected text data and the to-be-detected image data as inputs of a text image description extraction module, and output text modality features and image description text modality features;

[0184] An output module 503 is used to extract image features according to the image data to be detected by using an image feature extraction module, and output image modal features;

[0185] A generating module 504, configured to perform feature fusion on the text modality feature and the image description text modality feature through a hybrid expert fusion module to generate a fused text modality feature;

[0186] The fusion module 505 is used to use the multimodal attention fusion module to perform attention fusion on the fused text modal features and the image modal features, and output the target general feature recognition result.

[0187] Furthermore, the text image description extraction module includes a convolution submodule, an autoregressive decoder, a natural language processing model encoder and a natural language processing model decoder; an input module 502, specifically used for:

[0188] The convolution submodule is used to extract high-dimensional features of the image data to be detected and generate high-dimensional features of the image;

[0189] Decode the high-dimensional features of the image through an autoregressive decoder and output the image description text;

[0190] The image description text and the text data to be detected are respectively input into the natural language processing model encoder for encoding, and an identifier sequence and an attention mask corresponding to the image description text and an identifier sequence and an attention mask corresponding to the text data to be detected are generated;

[0191] Marking the identifier sequence corresponding to the image description text and the identifier sequence corresponding to the text data to be detected respectively, to generate an identifier marking sequence corresponding to the image description text and an identifier marking sequence corresponding to the text data to be detected;

[0192] The identifier token sequence and attention mask corresponding to the image description text and the identifier token sequence and attention mask corresponding to the text data to be detected are respectively input into the natural language processing model decoder for decoding to generate text modality features and image description text modality features.

[0193] Furthermore, the output module 503 is specifically used for:

[0194] Perform pixel adjustment on the image data to be detected to generate an adjusted pixel image;

[0195] Performing image feature extraction on the adjusted pixel image through an image feature extraction module, and outputting an image visual representation;

[0196] Perform linear transformation on the image visual representation and output image modality features.

[0197] Furthermore, the hybrid expert fusion module includes a gating layer, an expert layer, and an attention layer; a generation module 504 is specifically used for:

[0198] Determine weighted input features based on text modality features and image description text modality features;

[0199] The gating layer performs probability calculation based on weighted input features to determine the selection probability distribution corresponding to multiple experts in the expert layer;

[0200] Select any expert corresponding to the selection probability distribution greater than the preset probability threshold as the activated expert;

[0201] Perform expert calculations on weighted input features through multiple activated experts, and output expert calculation results corresponding to each activated expert;

[0202] The attention layer is used to perform weighted operations on the expert calculation results and selection probability distribution corresponding to each activated expert to determine the mixed expert weighted results;

[0203] The hybrid expert weighted results and weighted input features are normalized and fused to generate fused text modality features.

[0204] Furthermore, the multimodal attention fusion module includes a multimodal attention layer, a normalized feedforward network submodule; and a fusion module 505, which is specifically used for:

[0205] A multi-head attention weight calculation is performed by a multi-modal attention layer according to the fused text modality features and the image modality features, and a plurality of first text image attention weights and a plurality of second text image attention weights are output;

[0206] Respectively splicing a plurality of first text image attention weights and a plurality of second text image attention weights to generate a first text image attention splicing vector and a second text image attention splicing vector;

[0207] Performing linear transformation on the first text image attention splicing vector and the second text image attention splicing vector respectively, and outputting a first multimodal attention fusion representation and a second multimodal attention fusion representation;

[0208] A normalized feedforward network submodule is used to perform normalized superposition according to the first multimodal attention fusion representation and the image modality feature to generate a first normalized superposition feature;

[0209] A normalized feedforward network submodule is used to perform normalized superposition according to the second multimodal attention fusion representation and the fused text modality features to generate a second normalized superposition feature;

[0210] The multi-head attention weights are calculated by the multimodal attention layer according to the fused text modal features and the first normalized superposition features, and multiple text normalized attention weights are output;

[0211] Concatenate and linearly transform the normalized attention weights of multiple texts to output the third multimodal attention fusion representation;

[0212] A normalized feedforward network submodule is used to perform normalized superposition based on the third multimodal attention fusion representation and the fused text modality features to generate the target text representation;

[0213] The second normalized superposition feature and the target text representation are fused to generate a target visual representation;

[0214] The target visual representation and target text representation are concatenated and linearly transformed to generate the target common feature recognition result.

[0215] In an optional embodiment, the device further comprises:

[0216] The first module is used for, when receiving the text data to be trained and the image data to be trained, using the text data to be trained and the image data to be trained as inputs of the initial multimodal named entity recognition model, and outputting the recognition results of the common features to be trained;

[0217] The second module is used to calculate the probability distribution to be trained based on the recognition results of the general features to be trained;

[0218] The third module is used to substitute the probability distribution to be trained into the preset loss function and perform derivation to determine the model gradient;

[0219] The fourth module is used to update the initial multimodal named entity recognition model using the model gradient, determine the intermediate multimodal named entity recognition model, and count the number of model iterations in real time;

[0220] The fifth module is used to determine whether the number of model iterations reaches the preset number of training times;

[0221] The sixth module is used to use the intermediate multimodal named entity recognition model as a trained preset multimodal named entity recognition model if achieved.

[0222] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0223] An embodiment of the present invention further provides a computer device, including a memory and a processor, wherein a computer program is stored in the memory; when the computer program is executed by the processor, the processor executes the steps of the multimodal named entity recognition method as described in any of the above embodiments.

[0224] An embodiment of the present invention further provides a computer-readable storage medium having a computer program / instruction stored thereon. When the computer program / instruction is executed by a processor, the steps of the multimodal named entity recognition method as described in any of the above embodiments are implemented.

[0225] An embodiment of the present invention further provides a computer program product, including a computer program / instruction, which implements the steps of the multimodal named entity recognition method as described in any of the above embodiments when the computer program / instruction is executed by a processor.

[0226] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0227] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0228] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal named entity recognition method, characterized in that: include: Acquire text data to be detected and image data to be detected, and input the text data to be detected and the image data to be detected into a preset multimodal named entity recognition model, wherein the preset multimodal named entity recognition model includes a text image description extraction module, an image feature extraction module, a hybrid expert fusion module and a multimodal attention fusion module; Using the to-be-detected text data and the to-be-detected image data as inputs of a text image description extraction module, and outputting text modality features and image description text modality features; The image feature extraction module is used to extract image features according to the image data to be detected, and output image modal features; The text modality feature and the image description text modality feature are subjected to feature fusion by the hybrid expert fusion module to generate a fused text modality feature; The multimodal attention fusion module is used to perform attention fusion on the fused text modal features and the image modal features, and output a target common feature recognition result.

2. The multimodal named entity recognition method according to claim 1, characterized in that: The text image description extraction module includes a convolution submodule, an autoregressive decoder, a natural language processing model encoder and a natural language processing model decoder; the text data to be detected and the image data to be detected are used as inputs of the text image description extraction module, and the text modality features and the image description text modality features are output, including: Using a convolution submodule to extract high-dimensional features from the image data to be detected, and generate high-dimensional features of the image; Decoding the high-dimensional features of the image through an autoregressive decoder and outputting image description text; Input the image description text and the text data to be detected into a natural language processing model encoder for encoding, respectively, to generate an identifier sequence and an attention mask corresponding to the image description text, and an identifier sequence and an attention mask corresponding to the text data to be detected; Marking the identifier sequence corresponding to the image description text and the identifier sequence corresponding to the text data to be detected respectively, to generate an identifier marking sequence corresponding to the image description text and an identifier marking sequence corresponding to the text data to be detected; The identifier tag sequence and attention mask corresponding to the image description text and the identifier tag sequence and attention mask corresponding to the text data to be detected are respectively input into the natural language processing model decoder for decoding to generate text modality features and image description text modality features.

3. The multimodal named entity recognition method according to claim 1, characterized in that: The step of using the image feature extraction module to extract image features according to the image data to be detected and outputting image modality features includes: Performing pixel adjustment on the image data to be detected to generate an adjusted pixel image; Performing image feature extraction on the adjusted pixel image by the image feature extraction module, and outputting an image visual representation; Perform a linear transformation on the image visual representation and output image modality features.

4. The multimodal named entity recognition method according to claim 1, characterized in that: The hybrid expert fusion module includes a gating layer, an expert layer, and an attention layer; the hybrid expert fusion module performs feature fusion on the text modality feature and the image description text modality feature to generate a fused text modality feature, including: Determining weighted input features based on the text modality features and the image description text modality features; Performing probability calculation according to the weighted input features through the gating layer to determine the selection probability distribution corresponding to the multiple experts in the expert layer; Select any expert corresponding to the selection probability distribution greater than the preset probability threshold as the activated expert; Perform expert calculations on the weighted input features respectively by the plurality of activated experts, and output expert calculation results corresponding to the activated experts; Using an attention layer to perform a weighted operation on the expert calculation results and selection probability distribution corresponding to each of the activated experts to determine a mixed expert weighted result; The mixed expert weighted result and the weighted input feature are normalized and fused to generate a fused text modality feature.

5. The multimodal named entity recognition method according to claim 1, characterized in that: The multimodal attention fusion module includes a multimodal attention layer and a normalized feedforward network submodule; the multimodal attention fusion module is used to perform attention fusion on the fused text modality feature and the image modality feature, and output a target general feature recognition result, including: Performing multi-head attention weight calculation according to the fused text modality feature and the image modality feature through a multimodal attention layer, and outputting a plurality of first text-image attention weights and a plurality of second text-image attention weights; Respectively splicing a plurality of the first text image attention weights and a plurality of the second text image attention weights to generate a first text image attention splicing vector and a second text image attention splicing vector; Performing linear transformation on the first text image attention splicing vector and the second text image attention splicing vector respectively, and outputting a first multimodal attention fusion representation and a second multimodal attention fusion representation; Using a normalized feedforward network submodule to perform normalized superposition according to the first multimodal attention fusion representation and the image modality feature to generate a first normalized superposition feature; Using a normalized feedforward network submodule to perform normalized superposition according to the second multimodal attention fusion representation and the fused text modality feature to generate a second normalized superposition feature; Performing multi-head attention weight calculation according to the fused text modality feature and the first normalized superposition feature through a multimodal attention layer, and outputting multiple text normalized attention weights; Concatenate and linearly change the normalized attention weights of the multiple texts, and output a third multimodal attention fusion representation; Using a normalized feedforward network submodule to perform normalized superposition according to the third multimodal attention fusion representation and the fused text modality features to generate a target text representation; fusing the second normalized superposition feature and the target text representation to generate a target visual representation; The target visual representation and the target text representation are concatenated and linearly transformed to generate a target general feature recognition result.

6. The multimodal named entity recognition method according to claim 1, characterized in that: The training process of the preset multimodal named entity recognition model includes: When receiving the text data to be trained and the image data to be trained, using the text data to be trained and the image data to be trained as inputs of an initial multimodal named entity recognition model, and outputting a general feature recognition result to be trained; Calculating the probability distribution to be trained according to the recognition result of the general feature to be trained; Substituting the probability distribution to be trained into a preset loss function and performing derivation to determine the model gradient; The initial multimodal named entity recognition model is updated using the model gradient, an intermediate multimodal named entity recognition model is determined, and the number of model iterations is counted in real time; Determine whether the number of model iterations reaches a preset number of training times; If achieved, the intermediate multimodal named entity recognition model is used as the trained preset multimodal named entity recognition model.

7. A multimodal named entity recognition device, characterized in that: include: An acquisition module, used for acquiring text data to be detected and image data to be detected, and inputting the text data to be detected and the image data to be detected into a preset multimodal named entity recognition model, wherein the preset multimodal named entity recognition model includes a text image description extraction module, an image feature extraction module, a hybrid expert fusion module and a multimodal attention fusion module; An input module, used to use the to-be-detected text data and the to-be-detected image data as inputs of a text image description extraction module, and output text modality features and image description text modality features; An output module, used to extract image features according to the image data to be detected by using the image feature extraction module, and output image modal features; A generating module, used for performing feature fusion on the text modality feature and the image description text modality feature through the hybrid expert fusion module to generate a fused text modality feature; A fusion module is used to use the multimodal attention fusion module to perform attention fusion on the fused text modal features and the image modal features, and output a target common feature recognition result.

8. A computer device, characterized in that: It includes a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the multimodal named entity recognition method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed, the multimodal named entity recognition method according to any one of claims 1 to 6 is implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program stored on a non-transitory computer-readable storage medium, wherein the computer program comprises program instructions, wherein when the program instructions are executed by a computer, the computer is caused to execute the multimodal named entity recognition method as described in any one of claims 1-6.

Images