Multimodal named entity recognition method based on entity semantics and visual textualization

Abstract

The present invention discloses a multimodal named entity recognition method based on entity semantics and visual textualization. Step 1: Obtain a social media image and text dataset; Step 2: Process the input text S and input image I separately to obtain the input text representation B and the textual image representation V; Step 3: Preset the entity location task, input the input text features H into another BiLSTM network, and pass the output of the other BiLSTM network through a CRF decoder, and use an average pooling operation to obtain the entity representation E of the input text; Step 4: Use a cross-attention mechanism to interact the textual image representation V with the entity representation E and the input text representation B to obtain a multimodal fusion representation M; Step 5: Use the conditional random field (CRF) decoder to perform label prediction on the final multimodal fusion representation M to obtain the optimal label sequence and complete recognition. This method solves the problem of image features incorrectly guiding non-entity words. It also solves the problem of feature differences between modalities.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal named entity recognition, and in particular to a multimodal named entity recognition method based on entity semantics and visual textualization. Background Art

[0002] Named entity recognition, a core task in natural language processing, aims to identify and classify specific entities in text, including names of people, places, and organizations. In social media analysis, multimodal named entity recognition is particularly critical. It combines visual and textual inputs to accurately identify and determine the type of entity. Since text on social media is often informal, concise, and polysemic, image data can serve as effective auxiliary information to help resolve ambiguity in text understanding. Multimodal feature extraction is a key step in multimodal named entity recognition. Its goal is to convert multimodal information into an effective feature representation. The effectiveness of feature extraction directly affects the effectiveness of subsequent tasks.

[0003] However, current multimodal feature extraction methods still have two problems, making it difficult for the information extracted by the model to be used in subsequent processes:

[0004] (1) Image features can easily mislead non-entity words. Since prominent visual objects in an image may be overemphasized, if each word in the input text is directly fused with the image representation, some non-entity words may be mistakenly identified as entities due to the misleading image features. For example, given a sentence "This is what the sunset looked like over Lake Michigan on August 7th." and an image containing the visual object "sun," the model will easily identify the word "sunset" used to describe the scene when the sun sets as an entity, which is obviously wrong. The reason for this problem is that the model does not locate the location of the entity during the feature extraction stage and pass this information to the feature fusion module.

[0005] (2) The problem of feature differences between modalities. Traditional methods tend to use encoders for text and images separately to generate their respective representations. However, even when describing the same thing, there are often significant differences between the features generated by text and images. Research shows that this feature difference between modalities is widely present within multimodal models, causing the text and image embeddings in multimodal datasets to often be isolated from each other in the model's embedding space. This phenomenon hinders the model from establishing an effective connection between the two types of information. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of the existing technology and propose a multimodal named entity recognition method based on entity semantics and visual textualization, which solves the problem of image features misleading non-entity words and solves the problem of feature differences between modalities.

[0007] In order to solve the above technical problems, the technical solution of the present invention is:

[0008] A multimodal named entity recognition method based on entity semantics and visual textualization includes the following steps:

[0009] Step 1: Obtain a social media image and text dataset, which includes input text S and input image I;

[0010] Step 2: Process the input text S and the input image I to obtain the input text representation B and the textual image representation V respectively;

[0011] The method to obtain the input text representation B is:

[0012] In the input text S = {s1, s2, ..., s n ) word embedding adds the part-of-speech information of the input text, and then the BERT pre-trained language model is used as a text encoder to extract the corresponding input text features H = {h0, h1, ..., h n+1},in, Indicates s i The corresponding context-aware text feature representation, d is the word vector dimension of BERT output. ; Input the input text feature H to the BiLSTM network to obtain the input text representation B;

[0013] The method to obtain the textual image representation V is:

[0014] By inputting the image description, image label, and optical character generated by the image, the image description, image label, and optical character are mapped to the text space respectively to obtain a comprehensive summary of the image semantic information in the form of text. The image semantic information in these three text forms is spliced together to obtain the textual image representation V;

[0015] Step 3: Preset the entity localization task, input the input text feature H into another BiLSTM network, and pass the output of the BiLSTM network through the CRF decoder to predict the entity location in the text to obtain the entity prediction result, and use the average pooling operation to obtain the entity representation E of the input text;

[0016] Step 4: Input the input text representation B, textual image representation V and entity representation E into Figure 3In the entity-level multimodal graph fusion module shown in the figure, the cross-attention mechanism is used to interact the textual image representation V with the entity representation E and the input text representation B respectively to update the entity representation E, input text representation B, and textual image representation V of each layer, and finally obtain a multimodal fusion representation M;

[0017] Step 5: The final multimodal fusion representation M in step 5 is subjected to label prediction through the conditional random field (CRF) decoder to obtain the optimal label sequence and complete the recognition.

[0018] Preferably, in step 2, Stanford CoreNLP is used as a part-of-speech tagging tool to identify the part-of-speech information of each word in the input text.

[0019] Preferably, the method for embedding the part-of-speech information is:

[0020] Using the same mapping method as token embedding, segment embedding, and position embedding in BERT embedding, the part-of-speech category sequence of the input text is mapped to a fixed-size dense vector, recorded as part-of-speech embedding, to indicate the part-of-speech information of the word. The token embedding, segment embedding, position embedding, and part-of-speech embedding are merged through vector addition to obtain the final embedding representation Embeddings of the input text.

[0021] Preferably, in step 2, the object detection part of VinVL is used as the image label generation model to obtain the image label V attribute ={a1,o1,a2,o2,…,a k , o k The VinVL model fine-tuned on the image description on MS-COCO is used as the image description model. The COCO dataset and the corresponding image description annotations are used as the fine-tuning dataset. The fine-tuning parameters include: learning rate 3e-6, batch size 16, and iteration rounds 64. The corresponding image description V caption ; Use Tesseract OCR as the OCR model to detect possible text in the image and obtain the optical character V ocr .

[0022] As a preferred method, in step 2, the method of converting the image description, image label and optical character into text form is: using a special separation mark "[X]" to splice the image label V attribute , Image Description V caption , Optical Character V ocr, get the image information in text form, keep the word length of the spliced text image information consistent with the maximum length n of the input text, and obtain the text image representation output by BERT through word embedding processing and BERT text feature extraction operation

[0023] Preferably, the specific steps of step 3 include:

[0024] Step 3.1: Pass the input text feature H from step 2.4 through another bidirectional LSTM layer to obtain the text context feature representation R containing part-of-speech information.

[0025] Step 3.2: Input R into the CRF decoder for label prediction. CRF can convert R into the optimal label sequence in Z is a set consisting of "B", "I", and "O". Specifically, "B" represents the start word of an entity, "I" represents the non-start word of an entity, and "O" represents the word that does not belong to any entity. Therefore, according to z * You can get the predicted entity positions in the input text, and extract the words at these positions from the text S to get the entity prediction results;

[0026] Step 3.3: For an input sentence, since the entity predicted by the present invention may contain multiple words, and each word corresponds to a feature representation, the present invention uses an average pooling operation to extract the entity representation from the multiple words that constitute the entity. Figure 2 As shown in the figure, the present invention performs average pooling processing on the text representation R containing part-of-speech information output by the bidirectional LSTM network according to the entity prediction results. Specifically, first, based on the entity prediction results, the present invention can extract all m entities identified by the entity localization task in the input text S. Each entity Each of them consists of r consecutive words in the input text, where 1≤m≤n, 1≤i≤m, and 1≤r≤n. The word representation of these r words can be obtained by extracting the input text representation B by the text feature extraction module, which is recorded as Next, this module will T i The element features in are concatenated into a matrix, where each row corresponds to the representation vector of a word. For each column of the matrix, that is, each dimension, the average value of all the eigenvalues of the column is taken out, thereby obtaining a vector of length d, that is, the word vector after the average pooling layer, to represent the feature average of all the constituent words of this entity. This feature average can be used as the representation of one of the entities in the input text. After the average pooling operation is performed on m entities, the representation of each entity can be obtained. The representations of all entities are concatenated. If the number of entities m is less than n+2, the vector corresponding to the [CLS] tag and the vector corresponding to the [SEP] tag obtained from the input text representation B are first used as the first and last vectors of the entity representation. Then, the entity representation is supplemented to n+2 vectors with a zero vector. The entity representation of the input text can be obtained, which is recorded as

[0027] Preferably, the specific steps of step 4 include:

[0028] Step 4.1: The text input to the first graph fusion layer of the entity-level multimodal graph fusion module is represented as B (l) , the entity is represented by E (l) , the image is represented by V (l) First, the three feature representations are subjected to homomodal feature interaction, and each of them is subjected to self-attention operation through the standard Transformer layer to interact with the homomodal edge and obtain the updated text representation. Entity Representation Image Representation

[0029] Step 4.2: Image representation V′ (l) With the text representation B′ (l) The feature interaction of the text-aware image representation The text representation after integrating visual information is obtained through the following formula:

[0030]

[0031]

[0032]

[0033] in, Represents the ratio of cross-modal fusion of text representation and image representation, w1 (l) 、w2 (l) represents the learnable weight matrix, b1 (l) 、b2 (l) Indicates the bias value, That is, the text representation after integrating visual information;

[0034] Step 4.3: Entity representation E′ (l) With image representation V′ (l) The feature interaction adopts the same cross-attention mechanism-based method as the text and image interaction to obtain entity-aware image representation Then, by V (l) and V″′(l) The image representation that incorporates both text representation and entity representation is obtained through the following formula

[0035]

[0036]

[0037]

[0038] in, represents the fusion ratio of text-aware image representation and entity-aware image representation, w3 (l) 、w4 (l) represents the learnable weight matrix, b3 (l) 、b4 (l) Indicates the bias value;

[0039] Step 4.4: The entity representation E is updated to the current text representation B″ using the average pooling operation in step 3.3 based on the entity position obtained in step 3.2. (l) The corresponding entity representation in

[0040] Step 4.5: Text representation B″ of the output of layer l (l) , entity representation E″ (l) , image representation V″″ (l) , and use it as the feature input of the next graph fusion layer, repeat steps 4.1 to 4.4, and after L graph fusion layers, the final feature representations of text, entity, and image are obtained, which are B final 、E final 、V final , B final With V final Splicing them together, we can get the final multimodal fusion representation

[0041] Preferably, the final loss function of the model composed of steps 1 to 4 is trained The specific calculation process is as follows:

[0042]

[0043] Among them, λ EL It is a hyperparameter used to control the contribution of the entity localization task, and the loss function CRF is used as the decoder for text-based entity localization and recognition tasks.

[0044] Given a training sample set Loss Function The definition of is as follows:

[0045]

[0046] Here, N is the batch size, that is, the number of samples used in each iteration of the model during training.

[0047] The present invention has the following characteristics and beneficial effects:

[0048] Using the above technical solution, the present invention proposes a multimodal named entity recognition method based on entity semantics and visual textualization. By introducing an entity localization task based on part-of-speech information and performing an average pooling operation on the entities identified by this task, entity features are acquired. In subsequent steps, an entity-level multimodal graph fusion method is used to fuse text, entity, and image representations, rather than simply fusing all words and images. This solves the problem of image features misleading non-entity words. The input image is converted to text space, that is, image descriptions, image labels, and optical characters are obtained from the input image to comprehensively reflect the image semantic information. These three textual forms of image information are connected and used as image information for subsequent interaction with text features, solving the problem of feature differences between modalities. Experimental results show that the F1 score achieved by this method on the Twitter-2017 dataset is higher than that of other models, fully verifying the effectiveness of this method in solving the problem of features being difficult to utilize in subsequent steps in multimodal feature extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0050] Figure 1 The overall architecture diagram of the model of the present invention;

[0051] Figure 2 Schematic diagram of entity representation obtained by the average pooling layer of the present invention.

[0052] Figure 3 Structural diagram of the entity-level multimodal graph fusion module of the present invention. DETAILED DESCRIPTION

[0053] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0054] Reference Figure 1This is the overall architecture diagram of the model of the present invention. This embodiment discloses a multimodal named entity recognition method based on entity semantics and visual textualization. The present invention is further described below with reference to the accompanying drawings, and includes the following steps:

[0055] Step 1: Obtain a social media image and text dataset, and obtain the input text S and input image I.

[0056] Specifically, we use the Twitter-2017 public dataset for the multimodal named entity recognition task and divide the dataset into training set, validation set, and test set to provide basic data support for subsequent experiments.

[0057] Step 2: For the input text S, add the part-of-speech information of the input text to its word embedding, use the BERT pre-trained language model as the text encoder to extract the corresponding input text features H, and input the BERT output into the BiLSTM network to obtain the input text representation B.

[0058] Furthermore, the specific steps of step 2 include:

[0059] Using the BERT pre-trained model as a text encoder, the BERT pre-trained model can simultaneously consider the contextual information of words and has achieved good performance in multiple natural language processing tasks. For a given input text sequence S = {s1, s2, ..., s n}, add [CLS] tag at the beginning for subsequent classification tasks, and add [SEP] tag at the end to separate sentences. Let S′={s0,s1,…,s n+1} is the input text sequence after the above tags are inserted, where s0 and s n+1 Indicates the two inserted tokens. Before being input into the BERT model, the input text is processed with word embedding, including token embedding, segment embedding, and position embedding, which respectively indicate the vector representation of words, the relationship between different sentences, and the position information of words in sentences.

[0060] Stanford CoreNLP is used as a part-of-speech tagging tool to identify the part-of-speech information of each word in the input text. The present invention selects four parts of speech as output categories, namely noun, verb, adjective, and other.

[0061] Using the same mapping method as token embedding, segment embedding, and position embedding in BERT embedding, the part-of-speech category sequence of the text is mapped to a fixed-size dense vector, denoted as part-of-speech embedding, to indicate the part-of-speech information of the word. The token embedding, segment embedding, position embedding, and part-of-speech embedding are merged through vector addition to obtain the final embedding representation Embeddings of the input text, as shown in the following formula.

[0062] Embeddings = E Token +E Position +E Segment +E 词性

[0063] By encoding the embeddings as the input of BERT into the BERT pre-trained model, we can obtain the context-based representation of S′, which is denoted as H = {h0, h1, ..., h n+1},in, Indicates s i The corresponding context-aware text feature representation, d is the word vector dimension output by BERT.

[0064] Input H into the bidirectional LSTM and obtain two output representations from two directions. Concatenating these two representations can obtain the output of the bidirectional LSTM, which is the text context representation containing part-of-speech information.

[0065] Furthermore, the method for obtaining the textual image representation V is as follows:

[0066] The image description, image label, and optical character information obtained from the input image are introduced, and the image features are mapped to the text space. The image semantic information is comprehensively summarized in the form of text. These three types of image information in text form are spliced together to replace the image features as the textual image representation V.

[0067] The specific steps include:

[0068] Use the object detection part of VinVL as the image label generation model. Input the input image I in step 1 to the region object label generation model to identify the objects in the image and generate k object labels o = {o1, o2, ..., o k} and multiple attribute labels corresponding to each object label a={a1,a2,…,a k}, where k≤100, sort the object labels in descending order based on the confidence scores. For each object, retain 0-3 attributes with confidence scores higher than the threshold 0.1 as the attributes corresponding to the object label, and place the attributes before the corresponding object label to obtain the image label V attribute:

[0069] V attribute ={a1,o1,a2,o2,…,a k , o k}

[0070] The VinVL model, which has been fine-tuned on image descriptions on MS-COCO, is used as the image description model. MS-COCO is a large image dataset, and VinVL is a pre-trained visual language model. The COCO dataset and its corresponding image description annotations are used as the fine-tuning dataset. The fine-tuning parameters include: learning rate 3e-6, batch size 16, and iteration rounds 64. The input image I in step 1 is input to the image description model to obtain the corresponding image description V caption .

[0071] Use Tesseract OCR as the OCR (Optical Character Recognition) model to detect possible text in the image. These texts may contain valuable semantic information and obtain the optical character V ocr If there is no recognizable text in the image, then V ocr May be empty.

[0072] Use special separator "[X]" to concatenate image tags V attribute , Image Description V caption , Optical Character V ocr , get the image information in text form, keep the word length of the spliced text image information consistent with the maximum length n of the input text, and obtain the text image representation output by BERT through word embedding processing and BERT text feature extraction operation Step 3. Design the entity localization task and input H into another bidirectional LSTM layer. The output is passed through the CRF (conditional random field) decoder to predict the location of the entity in the text, obtain the entity prediction result, and use the average pooling operation to obtain the entity representation of the input text.

[0073] Furthermore, the specific steps of step 3 include:

[0074] Step 3.1: Pass the input text feature H in step 2.4 through another bidirectional LSTM layer to obtain the text context feature representation R containing part-of-speech information.

[0075] Step 3.2: Input R into the CRF decoder for label prediction. CRF can convert R into the optimal label sequence in Z is a set consisting of "B", "I", and "O". Specifically, "B" represents the start word of an entity, "I" represents the non-start word of an entity, and "O" represents a word that does not belong to any entity.

[0076] Step 3.3: For an input sentence, the predicted entity position in the input text can be obtained according to z*. Since the entity predicted by the present invention may contain multiple words, and each word corresponds to a feature representation, the present invention uses an average pooling operation to extract the entity representation from the multiple words that constitute the entity. Figure 2 As shown, the present invention performs average pooling processing on the text representation R containing part-of-speech information output by the bidirectional LSTM network according to the entity prediction result. Specifically, according to the entity prediction result z * , extract all m entities identified by the entity localization task in the input text S Each entity Each of them consists of r consecutive words in the input text, where 1≤m≤n, 1≤i≤m, and 1≤r≤n. The word representation of these r words can be obtained by using the input text representation B extracted in step 2.5, which is recorded as Next, T i The element features in are concatenated into a matrix, where each row corresponds to the representation vector of a word. For each column of the matrix, that is, each dimension, the average value of all the eigenvalues of the column is taken out, thereby obtaining a vector of length d, that is, the word vector after the average pooling layer, to represent the feature average of all the constituent words of this entity. This feature average can be used as the representation of one of the entities in the input text. After the average pooling operation is performed on m entities, the representation of each entity can be obtained. By concatenating the representations of all entities, the entity representation of the input text can be obtained, which is recorded as

[0077] Step 4: Input the image representation, entity representation, and text representation into the entity-level multimodal graph fusion module, and use the cross-attention mechanism to interact the image representation with the entity representation and text representation respectively to update the entity representation E, text representation B, and image representation V of each layer, and finally obtain an effective multimodal fusion representation M.

[0078] Furthermore, the specific steps of step 4 include:

[0079] Step 4.1, reference Figure 3 This is the structure diagram of the entity-level multimodal graph fusion module of the present invention. Let the text input of the first layer be represented by B (l) , the entity is represented by E (l) , the image is represented by V (l)The multimodal graph fusion layer first performs homomodal feature interaction on these three feature representations, and then performs self-attention operation on each of them through the standard Transformer layer to perform homomodal edge interaction to obtain the updated text representation. Entity Representation Image Representation

[0080] Step 4.2: Implement image representation V′ (l) With the text representation B′ (l) The following method based on the cross attention mechanism is adopted: the entity representation B′ (l) As the query vector, the image representation V′ (l) As both a key vector and a value vector, B′ (l) and V′ (l) Input to the multi-head attention layer, the cross attention mechanism can be calculated as follows:

[0081]

[0082] MHAtt(B′ (l) , V′ (l) )=

[0083] W bv [Att1(B′ (l) , V′ (l) ), Att2(B′ (l) , V′ (l) ),…,Att m (B′ (l) , V′ (l) )]

[0084] Among them, Att i represents the i-th head of cross attention, MHAtt represents multi-head attention, m represents the maximum number of heads of multi-head attention, denote the weight matrices of the query vector, key vector, and value vector in the i-th cross attention head, respectively. Then it represents the weight matrix of the multi-head attention. Next, a feedforward neural network and layer normalization are stacked after the multi-head attention layer to obtain the text-aware image representation. The specific calculation formula is shown as follows:

[0085]

[0086]

[0087] Among them, LN represents layer normalization, FFN represents forward feedback network, represents the transition vector in the calculation process. Subsequently, the text representation after integrating visual information is obtained by the following formula:

[0088]

[0089]

[0090]

[0091] in, Represents the ratio of cross-modal fusion of text representation and image representation, w1 (l) 、w2 (l) represents the learnable weight matrix, b1 (l) 、b2 (l) Indicates the bias value, It is the text representation after integrating visual information.

[0092] Step 4.3: Entity representation E′ (l) With image representation V′ (l) The feature interaction adopts the same cross-attention mechanism-based method as the text and image interaction to obtain entity-aware image representation Then, by V (l) and V″′ (l) The image representation that incorporates both text representation and entity representation is obtained through the following formula

[0093]

[0094]

[0095]

[0096] in, represents the fusion ratio of text-aware image representation and entity-aware image representation, w3 (l) 、w4 (l) represents the learnable weight matrix, b3 (l) 、b4 (l) Indicates the offset value.

[0097] Step 4.4: Since the text representation has been updated to B″ through cross-modal fusion (l) Therefore, the entity representation should also be updated to the current text representation B″ using the average pooling operation in step 3.3 based on the entity position obtained in step 3.3 (l) The corresponding entity representation in

[0098] Step 4.5: The model integrates text representation, entity representation, and image representation in the form of a graph to obtain the text representation B″ of the lth layer output. (l) , entity representation E″ (l) , image representation V″″ (l) , and use it as the feature input of the next graph fusion layer. After L graph-based multimodal fusion layers encode the multimodal graph composed of text nodes, entity nodes, image nodes and their corresponding edges, the model obtains the final feature representation of text, entity and image, which are B final 、E final 、V final , B final With V final Splicing them together, we can get the final multimodal fusion representation

[0099] Step 5: Pass the final multimodal fusion representation M from step 4 through the conditional random field (CRF) decoder to predict labels, obtain the optimal label sequence, complete recognition, and calculate the loss. Combined with the loss obtained from the entity localization task, a weighted loss sum is obtained.

[0100] Furthermore, the specific steps of step 5 include:

[0101] Step 5.1: Input the final multimodal fusion representation M output by the entity-level multimodal graph fusion module into the CRF decoder. CRF can convert M into the optimal tag sequence. The probability formula for predicting a possible named entity tag sequence y′ given the input text S and image I is as follows:

[0102]

[0103]

[0104]

[0105] Among them, Y represents the set of all possible named entity label sequences corresponding to the input sentence, score(M, y′) is the feature score, which consists of two parts: the emission score and the transfer score. is the named entity tag y′ of the i-th element of the input text sequence S i The emission fraction, is the named entity label y′ i To the named entity label y′ i+1 The transfer score, is corresponding to y′ iIn order to make the model accurately predict the named entity label sequence, the present invention is committed to maximizing the probability corresponding to the correct named entity label sequence. Therefore, the present invention uses negative log-likelihood loss as the loss function to optimize the model and uses the back-propagation method to calculate the gradient. Given a training sample set Loss Function The definition of is as follows:

[0106]

[0107] Where N is the batch size, that is, the number of samples used in each iteration of the model training process. During the model training process, the present invention optimizes the proposed neural network by minimizing this loss function. In the testing phase, the present invention selects the named entity label sequence with the largest conditional probability as the final prediction result. The specific calculation formula of the prediction result is as follows:

[0108]

[0109] Step 5.2: Consistent with the multimodal named entity recognition task, the present invention uses CRF as the decoder for the text-based entity location recognition task to obtain the loss function of the task:

[0110] Step 5.3: Consider these two loss functions simultaneously through the weighting factor to obtain the final loss function The specific calculation process is as follows:

[0111]

[0112] Among them, λ EL is a hyperparameter used to control the contribution of entity localization task.

[0113] To demonstrate the effectiveness of the present invention, a comparative experiment was conducted using existing models. Tests were conducted on the Twitter-2017 public dataset for multimodal named entity recognition, which contains four entity types: person names (PER), place names (LOC), organization names (ORG), and other entities (MISC). Furthermore, the present invention used precision (P), recall (R), and F1 score as experimental evaluation metrics. The comparative experimental results with nine existing methods, including BiLSTM-CRF, CNN-BiLSTM-CRF, and HBiLSTM-CRF, are shown in Table 1.

[0114] Table 1 Comparative experimental results on the Twitter-2017 dataset

[0115]

[0116]

[0117] As shown in Table 1, the proposed method can effectively improve the named entity recognition effect, with a significant improvement in the F1 index, which is superior to other baseline models. The experimental data demonstrates the effectiveness of the proposed model for multimodal named entity recognition.

[0118] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. It will be apparent to those skilled in the art that various changes, modifications, substitutions, and variations of these embodiments, including components, without departing from the principles and spirit of the present invention are still within the scope of protection of the present invention.

Claims

1. A multimodal named entity recognition method based on entity semantics and visual textualization, characterized by: The steps include: Step 1: Obtain a social media image and text dataset, which includes input text S and input image I; Step 2: Process the input text S and the input image I to obtain the input text representation B and the textual image representation V respectively; The method to obtain the input text representation B is: The part-of-speech information of the input text is added to the word embedding of the input text S, and then the BERT pre-trained language model is used as a text encoder to extract the corresponding input text features H = {h0, h1, ..., h n+1 },in, Indicates s i The corresponding context-aware text feature representation, d is the word vector dimension output by BERT, and the input text feature H is input into the BiLSTM network to obtain the input text representation B; The method to obtain the textual image representation V is: By inputting the image description, image label, and optical character generated by the image, the image description, image label, and optical character are mapped to the text space respectively to obtain a comprehensive summary of the image semantic information in the form of text. The image semantic information in these three text forms is spliced together to obtain the textual image representation V; Step 3: Preset the entity localization task, input the input text feature H into another BiLSTM network, and pass the output of the BiLSTM network through the CRF decoder to predict the entity location in the text to obtain the entity prediction result, and use the average pooling operation to obtain the entity representation E of the input text; The specific steps of step 3 include: Step 3.1: Pass the input text feature H from step 2 through another bidirectional LSTM layer to obtain the text context feature representation R containing part-of-speech information. Step 3.2: Input R into the CRF decoder for label prediction. CRF converts R into the optimal label sequence. in Z is a set consisting of "B", "I", and "O", where "B" represents the start word of an entity, "I" represents the non-start word of an entity, and "O" represents the word that does not belong to any entity. * You can get the predicted entity positions in the input text, and extract the words at these positions from the text S to get the entity prediction results; Step 3.3: Based on the entity prediction results, extract all m entities identified by the entity localization task in the input text S, and obtain the entity representation of the input text through the average pooling operation, which is recorded as Step 4: Input the input text representation B, the textual image representation V, and the entity representation E into an entity-level multimodal graph fusion module. Use the cross-attention mechanism to interact the textual image representation V with the entity representation E and the input text representation B respectively to update the entity representation E, input text representation B, and textual image representation V at each layer, and finally obtain a multimodal fusion representation M. Step 5: Use the conditional random field (CRF) decoder to perform label prediction on the final multimodal fusion representation M in step 4 to obtain the optimal label sequence and complete the recognition.

2. The multimodal named entity recognition method based on entity semantics and visual textualization according to claim 1 is characterized in that: In step 2, Stanford CoreNLP is used as a part-of-speech tagging tool to identify the part-of-speech information of each word in the input text.

3. The multimodal named entity recognition method based on entity semantics and visual textualization according to claim 1 is characterized in that: The embedding method of the part-of-speech information is: Using the same mapping method as token embedding, segment embedding, and position embedding in BERT embedding, the part-of-speech category sequence of the input text is mapped to a fixed-size dense vector, recorded as part-of-speech embedding, to indicate the part-of-speech information of the word. The token embedding, segment embedding, position embedding, and part-of-speech embedding are merged through vector addition to obtain the final embedding representation Embeddings of the input text.

4. The multimodal named entity recognition method based on entity semantics and visual textualization according to claim 1 is characterized in that: In step 2, the object detection part of VinVL is used as the image label generation model to obtain the image label V attribute ={a1,o1,a2,o2,…,a k ,o k }; Use the VinVL model fine-tuned on the image description on MS-COCO as the image description model to obtain the corresponding image description V caption ; Use TesseractOCR as the OCR model to detect the text in the image and obtain the optical character V ocr .

5. The multimodal named entity recognition method based on entity semantics and visual textualization according to claim 4 is characterized in that: The VinVL model is fine-tuned using the COCO dataset and its corresponding image description annotations. The fine-tuning parameters include a learning rate of 3e-6, a batch size of 16, and 64 iterations.

6. The multimodal named entity recognition method based on entity semantics and visual textualization according to claim 4 is characterized in that: In step 2, the method of converting the image description, image label, and optical character into text form is as follows: use the separation mark "[X]" to splice the image label V attribute , Image Description V caption , Optical Character V ocr , get the image information in text form, keep the word length of the spliced text image information consistent with the maximum length n of the input text, and obtain the text image representation output by BERT through word embedding processing and BERT text feature extraction operation 7. The multimodal named entity recognition method based on entity semantics and visual textualization according to claim 1 is characterized in that: The specific steps of step 4 include: Step 4.1: The text representation of the input of the first graph fusion layer of the entity-level multimodal graph fusion module is B (l) , the entity is represented by E (l) , the image is represented by V (l) First, the three feature representations are subjected to homomodal feature interaction, and each of them is subjected to self-attention operation through the standard Transformer layer to interact with the homomodal edge and obtain the updated text representation. Entity Representation Image Representation Step 4.2: Image representation V′ (l) With the text representation B′ (l) The feature interaction of the text-aware image representation The text representation after integrating visual information is obtained through the following formula: in, Represents the ratio of cross-modal fusion of text representation and image representation, w1 (l) 、w2 (l) represents the learnable weight matrix, b1 (l) 、b2 (l) Indicates the bias value, That is, the text representation after integrating visual information; Step 4.3: Entity representation E′ (l) With image representation V′ (l) The feature interaction adopts the same cross-attention mechanism-based method as the text and image interaction to obtain entity-aware image representation Then, by V (l) and V″′ (l) The image representation that incorporates both text representation and entity representation is obtained through the following formula in, represents the fusion ratio of text-aware image representation and entity-aware image representation, w3 (l) 、w4 (l) represents the learnable weight matrix, b3 (l) 、b4 (l) Indicates the bias value; Step 4.4: The entity representation E is updated to the current text representation B″ using the average pooling operation in step 3.3 based on the entity position obtained in step 3.

2. (l) The corresponding entity representation in Step 4.5: Text representation B″ of the output of layer l (l) , entity representation E″ (l) , image representation V″″ (l) , and use it as the feature input of the next graph fusion layer, repeat steps 4.1 to 4.4, and after L graph fusion layers, the final feature representations of text, entity, and image are obtained, which are B final 、E final 、V final , B final With V final Splicing them together, we can get the final multimodal fusion representation 8. The multimodal named entity recognition method based on entity semantics and visual textualization according to claim 1 is characterized in that: The final loss function of the model constructed by steps 1 to 4 is trained The specific calculation process is as follows: Among them, λ EL It is a hyperparameter used to control the contribution of the entity localization task, and the loss function CRF is used as the decoder for text-based entity localization and recognition tasks. Given a training sample set Loss Function The definition of is as follows: Here, N is the batch size, that is, the number of samples used in each iteration of the model during training.

Images