Multi-modal named entity recognition method based on multi-granularity feature enhancement

Abstract

The invention discloses a multi-modal named entity recognition method based on multi-granularity feature enhancement, which comprises the following steps of: firstly, carrying out single-modal feature extraction on input data, and then carrying out multi-modal multi-granularity feature interaction on the input data based on a single-modal feature extraction result; and finally, performing label prediction of the input data based on a multi-granularity feature interaction result to complete entity identification. According to the scheme, a multi-modal named entity recognition framework capable of being used for multimedia image-text posts is defined, the single-modal feature extraction firstly reduces the influence of the result of irrelevant interference of texts and images, the performance of processing informal and short text contents is achieved, the problem of sparse and noisy data is effectively solved, and the recognition efficiency of the multi-modal named entities is improved. The multi-modal multi-granularity semantic interaction fully considers the correlation between the text and the picture and the local and overall necessity of the picture feature, more comprehensive visual guidance required by the text is obtained, the accuracy of final label prediction is improved, and the experiment has a good experiment result on a witter data set and has certain popularization and practical values.

Description

Technical Field

[0001] The present invention belongs to the field of machine learning and natural language processing, and specifically relates to a multimodal named entity recognition method based on multi-granularity feature enhancement. Background Art

[0002] Social media has become the main way for people to share their daily lives and express their opinions, and the number of posts has also increased. This combination of text and images makes user expressions more accurate, interesting, and diverse, better reflects user needs and emotions, and also provides a large amount of rich original corpus data for natural language processing (NLP). Named entity recognition (NER) is a prelude to many information extraction tasks. It aims to discover multiple categories of named entities from raw text data, such as people (PER), locations (LOC), and organizations (ORG). At present, text-based NER methods have good performance on well-formatted texts, such as news reports, using CNN, LSTM, and Transformer to learn context, and using softmax and CRF for decoding. However, social media texts often contain informal, colloquial expressions, even typos and spelling errors, and short texts contain less contextual information. Such text content leads to sparse data and affects accuracy. The proposal of multimodal named entity recognition (MNER) focuses on the visual features in related images, and uses visual information to enhance language features to obtain better results than NER. It mainly uses the entire image to encode each word, or uses text vectors to align with visual object features.

[0003] Despite this, multimodal named entity recognition still attaches importance to text content. Informal expressions and lack of context are still challenges that need to be overcome. In addition, there may be a lot of irrelevant information in the whole image, which interferes with entity extraction. Therefore, we cannot only focus on the visual features of the whole image and ignore the semantic correspondence between the objects in the image and the words in the text. Therefore, a multimodal named entity recognition method is urgently needed to solve the above problems and further improve the accuracy of multimodal named entity recognition on social media posts. Summary of the invention

[0004] In view of the above problems, the object of the present invention is to provide a multimodal named entity recognition method based on multi-granularity feature enhancement.

[0005] The specific technical solution for achieving the purpose of the present invention is:

[0006] A multimodal named entity recognition method based on multi-granularity feature enhancement includes the following steps:

[0007] Step 1: Extract single-modal features from input data;

[0008] Step 2: Perform multi-modal and multi-granular feature interaction on the input data based on the result of single-modal feature extraction;

[0009] Step 3: Based on the results of multi-granularity feature interaction, label prediction of input data is performed to complete entity recognition.

[0010] Compared with the prior art, the present invention has the following beneficial effects:

[0011] (1) The solution of the present invention uses a well-trained text feature enhancement technology to obtain the contextual representation of each word input, effectively weakening the interference existing in the informal expression of the text and enhancing the weight of the effective words in the text; the text features interact with the multi-granularity image features, and consider the correspondence between the text and the global features, the text and the local features, and the text and the image matching in multiple ways, and find the images that support the text features as a basis, reduce the influence of the image-irrelevant noise, and further improve the accuracy of named entity recognition;

[0012] (2) The present invention constructs a multi-cross-modal transformer structure in the multi-modal interaction stage, and the multi-level semantic interaction fully learns the most valuable word-aware multi-modal representation of the image, providing the required visual clues for the text more accurately and comprehensively;

[0013] (3) The multimodal named entity recognition method based on multi-granularity feature enhancement in the present invention defines a framework for multimodal named entity recognition that can be used for multimedia text and image posts, in which unimodal feature extraction first reduces the impact of irrelevant interference results of text and image themselves, has the performance to process informal and short text content, and effectively solves the problem of sparse and noisy data; the multimodal multi-granularity semantic interaction of this scheme fully considers the relevance of text and image, and the necessity of local and overall image features, obtains more comprehensive visual guidance required by the text, thereby improving the accuracy of the final tag prediction. This experiment has good experimental results on the Twitter dataset.

[0014] The present invention is further described below in conjunction with specific implementation modes. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a schematic diagram of the overall framework of the multimodal named entity recognition method based on multi-granularity feature enhancement of the present invention.

[0016] Figure 2 It is a schematic diagram of CLIP text sequence processing of the present invention.

[0017] Figure 3 Schematic diagram of the multi-cross-modal transformer structure of the present invention. DETAILED DESCRIPTION

[0018] Example

[0019] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention. The described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] As shown in this application and claims, unless the context clearly indicates an exception, the words "a", "an", "an kind" and / or "the" do not refer to the singular and may also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

[0021] Unless otherwise specifically stated, the relative arrangement, numerical expressions and numerical values ​​of the parts and steps set forth in these embodiments do not limit the scope of the present application. At the same time, it should be understood that, for ease of description, the sizes of the various parts shown in the accompanying drawings are not drawn according to the actual proportional relationship. The technology, method and equipment known to those of ordinary skill in the relevant field may not be discussed in detail, but in appropriate cases, the technology, method and equipment should be considered as a part of the authorization specification. In all examples shown and discussed here, any specific value should be interpreted as being merely exemplary, rather than as a limitation. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so that once a certain item is defined in an accompanying drawing, it does not need to be further discussed in subsequent drawings.

[0022] Combination Figure 1 , a multimodal named entity recognition method based on multi-granularity feature enhancement, comprising the following steps:

[0023] Step 1: Extract single-modal features from input data:

[0024] Step 1-1: Extract features from the text information in the input data to obtain the text feature representation T:

[0025] The feature extraction methods of the text information include BERT model, semantic enhancement and CLIP model;

[0026] When using the BERT model for text feature extraction, the text data X = {x0,...,x M} Input to the Encoder part of the BERT model for encoding, and obtain the feature vector as input for feature extraction to obtain the features of the input text data:

[0027] T i B =BERT(x i θ bert )

[0028]

[0029] When semantic enhancement is used for text feature extraction, each input word x is transformed into i Use the K most similar words {w i1 ,w i2 ,...,w iK} indicates that these similar words w ij Then through the Word2Vec model, we get υ ij , and assign a weight α to it ij , to weigh the effects of different words and obtain the characteristics of the input text data:

[0030]

[0031] Semantic enhancement enables the effective extraction of the contextual representation of each word in short texts and content with sparse context.

[0032] When using the CLIP model for text feature extraction, although the CLIP features of the entire text can be obtained by using the trained CLIP model, in order to extract the features of a single word input in accordance with the first two methods and obtain the feature representation of the same seq_length, it is necessary to process the inputs required by the three methods at the same time when processing word segmentation. Therefore, it is necessary to first segment the input text information into several tokens, and use the sequence of the same length of the original word combination string corresponding to each token as the text input of CLIP. This sequence is the same as the text information corresponding to the token sequence, so the extracted features have the same dimension and can be directly linearly combined;

[0033] After being encoded by the CLIP model, encode_text is used to extract the features of a single word input to obtain the text feature CLIP representation T C ;

[0034] This method has a good effect on feature extraction and matching of text and images.

[0035] Step 1-2: Extract features from the image information in the input data and obtain image feature representation:

[0036] The image feature extraction methods include ResNet network, MASK RCNNL model and CLIP model;

[0037] ResNet: When using the ResNet network for image feature extraction, the image is first divided into visual blocks, and the visual feature map is obtained using the encapsulated ResNet network. The feature vectors of the visual blocks of a certain scale are converted into the same dimension as the feature representation form of text information extraction to obtain the global visual representation G. This step can extract deeper and more meaningful features of the input image, and use it to extract the global features of the entire image.

[0038] MASK RCNNL: This method is widely used in tasks such as object classification and segmentation. When using the MASKRCNNL network for image feature extraction, the trained MASK RCNN is used to detect and classify objects in the image, obtain the predefined category of the object, and then obtain the feature representation of the detected object by looking up the category embedding table to obtain the local object representation L;

[0039] CLIP: This step is the same as ResNet, both of which take the entire image as input, but CLIP does not need to be divided into visual feature blocks for representation. When using the CLIP model for image feature extraction, the image is input into the CLIP model, and after encoding the image, encode_image is called to extract the image features of the corresponding model dimension, such as 1*1024, to obtain the image feature CLIP representation C, such as Figure 2 shown.

[0040] Step 2: Based on the results of unimodal feature extraction, perform multimodal and multi-granular feature interaction on the input data:

[0041] Step 2-1: Multi-granularity feature interaction mainly uses cross attention between text features and image features of different granularities. Text features do not consider their classification, so first of all, it is necessary to simply connect the text features represented by three different methods to obtain the text feature representation T to complete the subsequent interaction tasks.

[0042] In order to better learn word-aware multimodal representations from image features, a cross-modal transformer is used to complete the interaction. First, the text hidden representation t, global visual hidden representation g, local object hidden representation l, and CLIP image feature hidden representation c are obtained through the self-attention mechanism as the input of the cross-modal interaction. The cross-modal transformer layer combines the multi-head attention mechanism and the prefix network, and constructs Figure 3 The multi-cross-modal transformer structure uses the result of the previous layer as Q to input the next layer again, thereby completing multi-level semantic interaction.

[0043] Step 2-2: Perform multi-level semantic interaction based on the cross-modal transformer to obtain the word-aware multimodal representation S:

[0044] Step 2-2-1: Interact the global visual representation G with the text feature representation T: Based on the results obtained by the self-attention mechanism, take Q=t, K=V=g as input, complete the simple semantic interaction processing of text and image, and obtain the word-perceived image representation S g1 ,Right now:

[0045] Q, K, V input transform as query and key-value pairs to perform multi-head attention MA:

[0046] q i =QW i q ,k i =KW i k ,v i =VW i v

[0047]

[0048] Among them, W i q ,W i k ,W i v ∈R d×(d / m) is the trainable parameter of the i-th head, d and m are the number of attention heads and the representation dimension respectively. According to the input Q = t, K = V = g, the word-aware image representation S is obtained. g1 = z; then use Q = g, K = V = t as input, and complete the simple semantic interaction processing of text and picture again to obtain the image perception word representation S g2′ , and used as the input of the next layer, that is, Q = t, K = V = S g2′ , through the attention mechanism to obtain complex word perception image representation S g2 , and finally, based on the coefficient u, the final semantic interaction representation between the global visual representation G and the text feature representation T is obtained: S g =u×S g1 +(1-u)S g2 ,

[0049] Step 2-2-2: Interact the local object representation L with the text feature representation T: Based on the results obtained by the self-attention mechanism, take Q=t, K=V=l as input, complete the simple semantic interaction processing of text and image, and obtain the word-perceived image representation S l1, and then use Q = t, K = V = l as input, and complete the simple semantic interaction processing of text and picture again to obtain the image perception word representation S l2′ , and used as the input of the next layer, that is, Q = l, K = V = S l2′ , and obtain the complex word-aware image representation S l2 Finally, the final semantic interaction between the local object representation L and the text feature T is obtained through the gating function as S l =u×S l1 +(1-u)S l2 ,in

[0050] Step 2-2-3, interact the image feature CLIP representation C with the text feature representation T: Different from the previous one, the CLIP feature lacks a dimension, so we need to expand a dimension first to correspond to the seq_length dimension. Therefore, we first expand the dimension of the image feature CLIP representation C, and according to the results obtained by the self-attention mechanism, use Q=t, K=V=c as input to complete the simple semantic interaction processing of text and image, and obtain the word-aware image representation S c1 , and then use Q = c, K = V = t as input, and complete the simple semantic interaction processing of text and picture again to obtain the image perception word representation S c2′ , and used as the input of the next layer, that is, Q = l, K = V = S c2′ , and obtain the complex word-aware image representation S c2 Finally, the final semantic interaction between the image feature CLIP representation C and the text feature T is obtained through the gating function as S c =u×S c1 +(1-u)S c2 ,in

[0051] Step 2-2-4: Based on the results of steps 2-2-1 to 2-2-3, a linear function is used to link the word-aware multimodal representation of a single word input:

[0052] S=linear(S g ,S l ,S c )

[0053] Step 3: Predict the label of the input data based on the results of multi-granularity feature interaction to complete entity recognition:

[0054] Step 3-1: Although multimodal interaction is used to find effective input from words in the image and reduce the interference of irrelevant information in the image, text features are still an important part. Therefore, in order to fully utilize these two parts, the text feature representation T and the word-aware multimodal representation S are connected using torch.cat to obtain the final multimodal representation P = {P i |P i ∈R 2d ,i=0,...,M};

[0055] Step 3-2: Build a label prediction model based on the CRF layer and train the model using maximum likelihood estimation;

[0056] Step 3-3: The final multimodal representation P = {P i |P i ∈R 2d ,i=0,...,M} Input the trained CRF layer label prediction model to generate the probability of predicting the label sequence y:

[0057]

[0058] in, is a Potential function, and Y is all possible label sequences;

[0059] Step 3-4: Maximize the score decoding according to the formula to output the final label prediction result:

[0060]

[0061] The predicted label is the type of entity to be recognized, such as a task, a location, etc. The work of named entity recognition, also known as sequence labeling, is to find entity labels in the sequence.

[0062] The present invention defines a framework for multimodal named entity recognition that can be used for multimedia text and picture posts based on a multimodal named entity recognition method enhanced by multi-granularity features, wherein unimodal feature extraction first reduces the impact of irrelevant interference results of text and image themselves, has the performance of processing informal and short text content, and effectively solves the problem of sparse and noisy data; the multimodal multi-granularity semantic interaction of this scheme fully considers the relevance of text and image, and the necessity of local and overall image features, obtains more comprehensive visual guidance required by the text, thereby improving the accuracy of the final tag prediction. This experiment has good experimental results on the Twitter dataset.

[0063] The present invention also provides a multimodal named entity recognition system based on multi-granularity feature enhancement, comprising the following modules:

[0064] Single-modal feature extraction module: used to extract single-modal features from input data;

[0065] Multimodal multi-granularity feature interaction module: used for multimodal multi-granularity feature interaction of input data based on the results of single-modal feature extraction;

[0066] Label prediction module: used to predict the label of input data based on the results of multi-granularity feature interaction to complete entity recognition.

[0067] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the following steps are implemented:

[0068] Step 1: Extract single-modal features from input data;

[0069] Step 2: Perform multi-modal and multi-granular feature interaction on the input data based on the result of single-modal feature extraction;

[0070] Step 3: Based on the results of multi-granularity feature interaction, label prediction of input data is performed to complete entity recognition.

[0071] A computer storable medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0072] Step 1: Extract single-modal features from input data;

[0073] Step 2: Perform multi-modal and multi-granular feature interaction on the input data based on the result of single-modal feature extraction;

[0074] Step 3: Based on the results of multi-granularity feature interaction, label prediction of input data is performed to complete entity recognition.

[0075] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A multimodal named entity recognition method based on multi-granularity feature enhancement, characterized in that: The following steps are involved: Step 1: Extract single-modal features from input data; Step 2: Perform multi-modal and multi-granular feature interaction on the input data based on the result of single-modal feature extraction; Step 3: Based on the results of multi-granularity feature interaction, label prediction of input data is performed to complete entity recognition.

2. The multimodal named entity recognition method based on multi-granularity feature enhancement according to claim 1 is characterized in that: The unimodal feature extraction in step 1 is specifically as follows: Step 1-1, extracting features from text information in the input data to obtain text feature representation T; Step 1-2: extract features from the image information in the input data to obtain image feature representation.

3. According to the multimodal named entity recognition method based on multi-granularity feature enhancement in claim 2, the feature extraction method of the text information in step 1-1 comprises a BERT model, semantic enhancement and a CLIP model; in, When using the BERT model for text feature extraction, the text data X = {x0,...,x M } Input to the Encoder part of the BERT model for encoding, and obtain the feature vector as input for feature extraction to obtain the features of the input text data: T i B =BERT(x i ;θ bert ) When semantic enhancement is used for text feature extraction, each input word x is transformed into i Use the K most similar words {w i1 ,w i2 ,...,w iK } indicates that these similar words w ij Then through the Word2Vec model, we get υ ij , and assign a weight α to it ij , get the features of the input text data: When using the CLIP model for text feature extraction, the input text information is first segmented into several tokens, and the sequence of the same length of the original word combination string corresponding to each token is used as the text input of CLIP. After encoding by the CLIP model, encode_text is used to extract the features of the single word input to obtain the text feature CLIP representation T C .

4. The multimodal named entity recognition method based on multi-granularity feature enhancement according to claim 2 is characterized in that: The image feature extraction methods in step 1-2 include ResNet network, MASK RCNNL model and CLIP model; When using the ResNet network to extract image features, the image is first divided into visual blocks, and the visual feature map is obtained using the encapsulated ResNet network. The feature vectors of the visual blocks of a certain scale are converted into the same dimension as the feature representation form of text information extraction to obtain the global visual representation G. When using the MASK RCNNL network for image feature extraction, the trained MASK RCNN is used to detect and classify objects in the image to obtain the predefined categories of the objects. The feature representation of the detected objects is then obtained by looking up the category embedding table to obtain the local object representation L. When using the CLIP model to extract image features, the image is input into the CLIP model, and after encoding the image, encode_image is called to extract the image features of the corresponding model dimensions to obtain the CLIP representation C of the image features.

5. The multimodal named entity recognition method based on multi-granularity feature enhancement according to claim 4 is characterized in that: The multi-modal and multi-granular feature interaction of the input data in step 2 is specifically as follows: Step 2-1, obtain text hidden representation t, global visual hidden representation g, local object hidden representation l, CLIP image feature hidden representation c through the self-attention mechanism; Step 2-2: Perform multi-level semantic interaction based on the cross-modal transformer to obtain the word-aware multimodal representation S.

6. The multimodal named entity recognition method based on multi-granularity feature enhancement according to claim 5 is characterized in that: The multi-level semantic interaction based on the cross-modal transformer in step 2-2 is specifically as follows: Step 2-2-1: Interact the global visual representation G with the text feature representation T: Based on the results obtained by the self-attention mechanism, take Q=t, K=V=g as input, complete the simple semantic interaction processing of text and image, and obtain the word-perceived image representation S g1 ,Right now: Q, K, V input transform as query and key-value pairs to perform multi-head attention MA: q i =QW i q ,k i =KW i k ,v i =VW i v Among them, W i q ,W i k ,W i v ∈R d×(d / m) is the trainable parameter of the i-th head, d and m are the number of attention heads and the representation dimension respectively. According to the input Q = t, K = V = g, the word-aware image representation S is obtained. g1 = z; then use Q = g, K = V = t as input, and complete the simple semantic interaction processing of text and picture again to obtain the image perception word representation S g2′ , and used as the input of the next layer, that is, Q = t, K = V = S g2′ , through the attention mechanism to obtain complex word perception image representation S g2 , and finally, based on the coefficient u, the final semantic interaction representation between the global visual representation G and the text feature representation T is obtained: S g =u×S g1 +(1-u)S g2 , Step 2-2-2: Interact the local object representation L with the text feature representation T: Based on the results obtained by the self-attention mechanism, take Q=t, K=V=l as input, complete the simple semantic interaction processing of text and image, and obtain the word-perceived image representation S l1 , and then use Q = t, K = V = l as input, and complete the simple semantic interaction processing of text and picture again to obtain the image perception word representation S l2′ , and used as the input of the next layer, that is, Q = l, K = V = S l2′ , and obtain the complex word-aware image representation S l2 Finally, the final semantic interaction between the local object representation L and the text feature T is obtained through the gating function as S l =u×S l1 +(1-u)S l2 ,in Step 2-2-3: Interact the image feature CLIP representation C with the text feature representation T: First, expand the dimension of the image feature CLIP representation C, and according to the results obtained by the self-attention mechanism, use Q=t, K=V=c as input to complete the simple semantic interaction processing of text and image, and obtain the word-aware image representation S c1 , and then use Q = c, K = V = t as input, and complete the simple semantic interaction processing of text and picture again to obtain the image perception word representation S c2′ , and used as the input of the next layer, that is, Q = l, K = V = S c2′ , and obtain the complex word-aware image representation S c2 Finally, the final semantic interaction between the image feature CLIP representation C and the text feature T is obtained through the gating function as S c =u×S c1 +(1-u)S c2 ,in Step 2-2-4: Based on the results of steps 2-2-1 to 2-2-3, a linear function is used to link the word-aware multimodal representation of a single word input: S=linear(S g ,S l ,S c )。 7. The multimodal named entity recognition method based on multi-granularity feature enhancement according to claim 5 is characterized in that: The label prediction in step 3 is specifically: Step 3-1, connect the text feature representation T and the word-aware multimodal representation S to obtain the final multimodal representation P; Step 3-2: Build a label prediction model based on the CRF layer and train the model using maximum likelihood estimation; Step 3-3: The final multimodal representation P = {P i |P i ∈R 2d ,i=0,…,M} Input the trained CRF layer label prediction model to generate the probability of predicting the label sequence y: in, is a Potential function, Y is all possible label sequences; Step 3-4: Maximize the score decoding according to the formula to output the final label prediction result:

8. A multimodal named entity recognition system based on multi-granularity feature enhancement, characterized in that: Includes the following modules: Single-modal feature extraction module: used to extract single-modal features from input data; Multimodal multi-granularity feature interaction module: used for multimodal multi-granularity feature interaction of input data based on the results of single-modal feature extraction; Label prediction module: used to predict the label of input data based on the results of multi-granularity feature interaction to complete entity recognition.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer storable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images