Multi-modal named entity identification method and system

Abstract

The invention discloses a multi-modal named entity recognition method and system, and mainly relates to the technical field of computer vision and natural language processing. Comprising the following steps: extracting text features, multi-modal text features, multi-modal image features and visual object features of texts and pictures; obtaining text hidden features according to the extracted text features; obtaining all entity features in the collected text according to the obtained text hidden features; according to the multi-modal text features and the multi-modal image features, obtaining normalized multi-modal image features and text-image correlation scores; outputting visual object features after punishment and screening; outputting a multi-modal fusion feature; and outputting the entity label of each word in the text. The method has the beneficial effects that the problem that coarse-grained visual information and fine-grained visual information cannot be used at the same time after the correlation between the text and the image is judged is solved while the problem that the effect of judging the correlation between the text and the image is poor is solved.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and natural language processing, specifically to a multimodal named entity recognition method and system based on a pre-trained model to imply text-image relationships. Background Technology

[0002] With the widespread adoption and development of internet applications, tremendous convenience has been brought about, and the generation of massive amounts of data is also a significant advantage. This data broadly covers information from various aspects such as economy, technology, culture, and politics, with unstructured natural language text data showing a significant growth trend. However, as the scale of data continues to expand, a crucial problem arises: how to efficiently extract important information from massive amounts of unstructured text. To address this, Information Extraction (IE) technology has been proposed. This technology provides powerful support for the classification, extraction, and reconstruction of large amounts of natural language text, bridging the difficulty of extracting useful information from data in the context of information overload, and has attracted widespread attention in research both domestically and internationally. Information extraction is defined as the extraction of important information from unstructured text, such as entities, relationships, and events.

[0003] The multimodal named entity recognition task was first proposed by Moon et al., originating from the application of entity extraction tasks in the social media field, aiming to improve the accuracy of entity recognition. With the popularity of social platforms such as Twitter, Instagram, and Snapchat, users can create multimedia posts, often using text and images together, which provides rich data resources for multimodal learning.

[0004] In the dataset, irrelevant text-image pairs constitute a large proportion, and visual information unrelated to the text can have uncertain or even negative effects on multimodal model learning. To address this issue, existing methods are not very effective, mainly for the following reasons: (1) the methods for judging the relevance between text and images are ineffective, mistakenly filtering out images in relevant text-image pairs; (2) after filtering out images unrelated to the text, no additional relevant visual information is provided for the text, and the text lacks auxiliary information; (3) it does not achieve the simultaneous use of coarse-grained and fine-grained visual information after judging the relevance between text and images; (4) when using fine-grained visual information, it does not accurately select the visual objects most relevant to the entities in the text.

[0005] Therefore, there is an urgent need for a multimodal named entity recognition method and system based on pre-trained models to push text-image relationships to solve the above problems. Summary of the Invention

[0006] The purpose of this invention is to provide a multimodal named entity recognition method and system based on a pre-trained model to imply text-image relationships. This method addresses the problem of poor performance in judging text-image correlation, and also solves the problem of not being able to use both coarse-grained and fine-grained visual information after judging the correlation between text and image.

[0007] To achieve the above objectives, the present invention employs the following technical solution: On the one hand, a multimodal named entity recognition method is provided, including the following steps: S1: Collect text and images, and extract corresponding text features, multimodal text features, multimodal image features, and visual object features; S2: Obtain hidden text features based on the text features extracted in step S1; S3: Based on the text hidden features obtained in step S2, obtain all entity features in the collected text; S4: Based on the multimodal text features and multimodal image features in step S1, obtain the normalized multimodal image features and the text-image correlation score; S5: Based on the text-image relevance score obtained in step S4, the entity features obtained in step S3, and the visual object features obtained in step S1, output the penalized and filtered visual object features. S6: Based on the text hiding features obtained in step S2, the multimodal image features extracted in step S1, and the visual object features obtained in step S4, perform cross-modal semantic interactions between text hiding features and multimodal image features, and between text hiding features and visual object features, and then concatenate the corresponding output information to output multimodal fusion features. S7: Based on the multimodal fusion features output in step S6, output the entity label for each word in the text.

[0008] Preferably, step S1 specifically includes: The collected text is input into the text pre-trained model BERT, which outputs text features. ; The collected text and images are input into the CLIP multimodal pre-trained models, text encoder and image encoder, respectively, to obtain multimodal text features. Multimodal image features ; The collected images are input into the pre-trained model MASK RCNN to obtain visual objects, and the features of the visual objects are obtained by querying the category feature lookup table.

[0009] Preferably, step S2 specifically includes: Text features The input is fed into a 4-layer stacked self-attention transformer to capture the intramodal relations of the text, obtaining the text-hidden representation of each word. .

[0010] Preferably, step S3 specifically includes: Hiding text indicates Input a conditional random field to predict the label sequence of the text. ; Based on label sequence The tags in the text are hidden from view. Select the entity features corresponding to all entities ,in Representing the The characteristics of an entity This represents the total number of entities contained in the input sentence.

[0011] Preferably, step S4 specifically includes: Multimodal text features Multimodal image features Normalize them separately to obtain the normalized multimodal text features. Multimodal image features Using multimodal text features Multimodal image features To calculate the relevance score between the input text and the input image:

[0012] In the formula, @ represents the dot product of two matrices, for easy visualization; Set a hyperparameter As a gating mechanism, it is used to determine whether the input text and the input image are related, and selects visual information related to the text as auxiliary information, specifically: if This indicates that the input text is related to the input image, and multimodal image features are incorporated. The visual information is used to pair with the input text; otherwise, it means that the input text and the input image are unrelated, and the multimodal text features are used. As visual information paired with the input text.

[0013] Preferably, step S5 specifically includes: Penalizing visual object features based on the text-image correlation score using coarse-grained gating includes: constructing a mask matrix using the correlation score. ,pass Penalize visual object features; Visual features are filtered based on entity characteristics using fine-grained gating, including: using visual object features as... Entity features as The attention mechanism is used to calculate the relevance score between each visual region feature and each entity feature. The relevance scores of each visual region feature and all entity features are then summed to obtain the result. Each correlation score and, The number of features of the visual object; from Each correlation score and the top selected The visual region features corresponding to each value will be... The visual region features are summed and averaged to obtain the final visual region features that need to be input into the multimodal interaction module.

[0014] Preferably, step S6 specifically includes: Stack two cross-modal transformers to perform multimodal semantic interactions: Multimodal image features As Text hiding representation As and Input the first cross-modal transformer and output the word representation obtained from shallow visual perception. ; Hiding text indicates As Word representation based on shallow visual perception As and Input the second cross-modal transformer and output the word representation obtained from visual perception. ; Hiding text indicates As Multimodal image features As and Input another cross-modal transformer and output a visual representation of word perception. ; Use gate functions To obtain the final coarse-grained multimodal fusion features :

[0015]

[0016] In the formula, This represents the sigmoid activation function; , It is a weight matrix; Indicates features and To splice; By replacing the input of visual information from multimodal image features with visual object features, fine-grained multimodal fusion features are obtained. ; coarse-grained multimodal fusion features Features with fine-grained multimodal fusion Combining them yields multimodal fusion features. :

[0017] In the formula, Represents a linear function; Indicates features and Then, the parts are assembled.

[0018] Preferably, step S7 specifically comprises: Multimodal fusion features The input is fed into the Conditional Random Field decoding module, which outputs a sequence of entity labels for each word in the text. :

[0019] In the formula, Represents a text sequence. Represents all predicted label sequences of a text sequence. A sequence of labels, used to calculate each is The probability value of the label sequence. The function is used to select the label sequence corresponding to the highest probability value as the final label sequence.

[0020] On the other hand, a recognition system based on the multimodal named entity recognition method described above is provided, comprising: The feature extraction module is used to extract corresponding text features, multimodal text features, multimodal image features, and visual object features from the input text and images. The self-attention module is used to: output text hidden features based on the text features; The entity span recognition module is used to: output all entity features in the input text based on the text hiding features; The text-image relationship reasoning module is used to: output normalized multimodal image features and text-image correlation scores based on the multimodal text features and multimodal image features; The visual object filtering module is used to: output penalized and filtered visual object features based on the text-image relevance score, entity features, and visual object features; The multimodal interaction module is used to: perform cross-modal semantic interactions between the text hiding features and the multimodal image features, and between the text hiding features and the visual object features, respectively, based on the text hiding features, the multimodal image features, and the visual object features, and to concatenate the corresponding output information to output multimodal fusion features; The Conditional Random Field decoding module is used to: output the entity label of each word in the text based on the multimodal fusion features.

[0021] Compared with the prior art, the beneficial effects of the present invention are as follows: This paper proposes a multimodal named entity recognition method and system based on a pre-trained model to infer text-image relationships. For input text and images, it extracts corresponding text features, multimodal text features, multimodal image features, and visual object features. A self-attention module is designed to output hidden text features based on the text features. An entity span recognition module is designed to output all entity features in the input text based on the hidden text features. A text-image relationship inference module is designed to output normalized multimodal image features and text-image correlation based on the multimodal text features and multimodal image features. The system includes: a score; a visual object filtering module to output penalized and filtered visual object features based on the text-image relevance score, entity features, and visual object features; a multimodal interaction module to perform cross-modal semantic interactions between text-hidden features and multimodal image features, and between text-hidden features and visual object features, based on the text-hidden features, multimodal image features, and visual object features, and to concatenate the corresponding output information to output multimodal fusion features; and a conditional random field decoding module to output entity labels for each word in the text based on the multimodal fusion features. This invention improves performance by simultaneously using both coarse-grained and fine-grained visual information after determining the correlation between text and images. Specifically, a text-image relationship reasoning module is designed to address the poor performance in determining text-image correlation and the lack of additional relevant visual information provided to the text after filtering out images unrelated to it. Furthermore, to simultaneously utilize both coarse-grained and fine-grained visual information, an entity span recognition module and a visual object filtering module are designed to address the inability to simultaneously use both coarse-grained and fine-grained visual information after determining the correlation between text and images, and the failure to accurately filter the most relevant visual objects in the text when using fine-grained visual information. Attached Figure Description

[0022] Figure 1 This is a flowchart illustrating a multimodal named entity recognition method based on a pre-trained model for inferring text-image relationships, as described in an embodiment of the present invention. Figure 2 This is an architecture diagram of a multimodal named entity recognition method based on a pre-trained model for inferring text-image relationships, as described in an embodiment of the present invention. Figure 3 This is a schematic diagram of the structure of a multimodal named entity recognition system based on a pre-trained model for inferring text-image relationships, as described in an embodiment of the present invention. Detailed Implementation

[0023] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined in this application.

[0024] In this invention, terms such as "upper," "lower," "left," "right," "front," "back," "vertical," "horizontal," "side," and "bottom" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. These terms are only used to facilitate the description of the structural relationships of the various components or elements of this invention and do not specifically refer to any component or element in this invention. They should not be construed as limiting the invention.

[0025] Example: A multimodal named entity recognition method based on pre-trained model inference of text-image relationships, such as Figure 1 As shown, the method includes the following steps: S1. For the input text and images, extract the corresponding text features, multimodal text features, multimodal image features, and visual object features; S2. Output the hidden text features based on the text features; S3. Based on the text hiding features, output all entity features in the input text; S4. Based on the multimodal text features and multimodal image features, output the normalized multimodal image features and the text-image correlation score; S5. Based on the text-image correlation score, entity features, and visual object features, output the penalized and filtered visual object features. S6. Based on the text hiding features, multimodal image features, and visual object features, perform cross-modal semantic interactions between text hiding features and multimodal image features, and between text hiding features and visual object features, respectively, and concatenate the corresponding output information to output multimodal fusion features. S7. Based on the multimodal fusion features, output the entity label for each word in the text.

[0026] To improve performance by using both coarse-grained and fine-grained visual information after determining the relevance between text and images, the system first obtains entity features from the text through an entity span recognition module. Then, it obtains the relevance score between the text and the image, as well as the multimodal image features related to the text, through a text-image relationship reasoning module. Finally, it uses the relevance score to penalize the visual object features and uses the entity features to select the visual objects most relevant to the entities in the text.

[0027] Combination Figure 2 The embodiments of the present invention will be described in detail below: In S1, a feature extraction module is designed: for the input text and images, the corresponding text features, multimodal text features, multimodal image features, and visual object features are extracted; the text pre-trained model BERT, the multimodal pre-trained model, and the MASK RCNN are used to extract text features, multimodal text features, multimodal image features, and visual object features, respectively. According to an embodiment of the present invention, 1) Extracting text features: The input text is fed into the pre-trained text model BERT, and the output text features are... ; 2) Extract multimodal text features and multimodal image features: The input text and image are fed into the CLIP text encoder and image encoder, respectively, of the multimodal pre-trained models to obtain multimodal text features. Multimodal image features ; 3) Extracting visual object features: The input image is fed into the pre-trained model MASK RCNN to obtain the visual object, and the features of the visual object are obtained by querying the category feature lookup table.

[0028] In S2, a self-attention module is designed to output text hiding features based on the text features. According to an embodiment of the present invention, text features The input is fed into a 4-layer stacked self-attention transformer to capture the intramodal relations of the text, obtaining the text-hidden representation of each word. .

[0029] In S3, an entity span recognition module is designed: based on the text hiding features, it outputs all entity features in the input text.

[0030] According to embodiments of the present invention, text hiding representation Input a conditional random field (span) to predict the label sequence of the text. ; Based on label sequence The tags in the text are hidden from view. Select the entity features corresponding to all entities ,in Representing the The characteristics of an entity This represents the total number of entities contained in the input sentence.

[0031] In S4, a text-image relationship reasoning module is designed: based on the multimodal text features and multimodal image features, it outputs the normalized multimodal image features and the text-image correlation score.

[0032] According to embodiments of the present invention, multimodal text features are... Multimodal image features Normalize them separately to obtain the normalized multimodal text features. Multimodal image features Next, we will use multimodal text features. Multimodal image features To calculate the relevance score between the input text and the input image. The calculation method is as follows:

[0033] In the formula, @ represents the dot product of two matrices. For easier visualization, the correlation score is multiplied by 100.

[0034] Set a hyperparameter As a gating mechanism, it determines whether the input text and the input image are related, and selects visual information related to the text as auxiliary information. Specifically, if... This indicates that the input text is related to the input image, so the multimodal image features will be used. The visual information is used to pair with the input text; otherwise, it means that the input text and the input image are unrelated, and the multimodal text features are used. This serves as visual information paired with the input text. For ease of later explanation, the obtained... or Collectively referred to as .

[0035] In S5, a visual object filtering module is designed: based on the text-image correlation score, entity features, and visual object features, it outputs penalized and filtered visual object features.

[0036] According to an embodiment of the present invention, firstly, visual object features are penalized based on the text-image correlation score using coarse-grained gating. Specifically, a mask matrix is ​​constructed using the correlation score. ,pass Penalize visual object features; Then, visual features are filtered based on the entity features using fine-grained gating. Specifically, visual object features are used as... Entity features as The attention mechanism is used to calculate the relevance score between each visual region feature and each entity feature. The relevance scores of each visual region feature and all entity features are then summed to obtain the result. (Number of visual object features) and correlation scores.

[0037] Finally, from Each correlation score and the top selected The visual region features corresponding to each value will be... The visual region features are summed and averaged to obtain the final visual region features that need to be input into the multimodal interaction module.

[0038] In S6, a multimodal interaction module is designed: based on the text hiding features, as well as the multimodal image features and visual object features, cross-modal semantic interactions are performed between the text hiding features and multimodal image features, and between the text hiding features and visual object features, respectively, and the corresponding output information is concatenated to output multimodal fusion features.

[0039] According to an embodiment of the present invention, the multimodal interaction module designed for interaction between text hiding features and multimodal image features has the same internal structure as the multimodal interaction module for interaction between text hiding features and visual object features. The only difference between the two multimodal interaction modules is that the input visual information is multimodal image features, while the input visual information is visual object features. The following description will focus on the multimodal interaction module for interaction between text hiding features and multimodal image features.

[0040] Stacking two cross-modal transformers to perform multimodal semantic interaction: first, multimodal image features... As Text hiding representation As and Input the first cross-modal transformer and output the word representation obtained from shallow visual perception. Then the text hiding representation T is used as... The word representation P in shallow visual perception and Input the second cross-modal transformer and output the word representation obtained from visual perception. ; Hiding text indicates As Multimodal image features As and Input another cross-modal transformer and output a visual representation of word perception. ; Use gate functions To obtain the final coarse-grained multimodal fusion features The calculation process is as follows:

[0041]

[0042] In the formula, This represents the sigmoid activation function; , It is a weight matrix; Indicates features and Then, the parts are assembled.

[0043] Similar to the process described above, the input of visual information is changed from multimodal image features to visual object features to obtain fine-grained multimodal fusion features. .

[0044] Finally, the coarse-grained multimodal fusion features are... Features with fine-grained multimodal fusion Combining them yields multimodal fusion features. The combination formula is:

[0045] In the formula, Represents a linear function; Indicates features and Then, the parts are assembled.

[0046] In S7, a conditional random field decoding module is designed to output the entity label of each word in the text based on the multimodal fusion features.

[0047] According to embodiments of the present invention, multimodal fusion features are... The input is fed into the Conditional Random Field decoding module, which outputs a sequence of entity labels for each word in the text. :

[0048] In the formula, Represents a text sequence. Represents all predicted label sequences of a text sequence. A sequence of labels, used to calculate each is The probability value of the label sequence. The function is used to select the label sequence corresponding to the highest probability value as the final label sequence. .

[0049] Another embodiment of the present invention provides a multimodal named entity recognition system based on a pre-trained model to infer text-image relationships, such as... Figure 3 As shown, the system includes: The feature extraction module is configured to extract corresponding text features, multimodal text features, multimodal image features, and visual object features from the input text and images. A self-attention module is configured to output hidden text features based on the text features; An entity span recognition module is configured to output all entity features in the input text based on the text hiding features; The text-image relationship reasoning module is configured to output normalized multimodal image features and a text-image correlation score based on the multimodal text features and multimodal image features. A visual object filtering module is configured to output penalized and filtered visual object features based on the text-image relevance score, entity features, and visual object features. The multimodal interaction module is configured to perform cross-modal semantic interactions between the text hiding features and the multimodal image features, and between the text hiding features and the visual object features, respectively, based on the text hiding features, the multimodal image features, and the visual object features, and to concatenate the corresponding output information to output multimodal fusion features; The Conditional Random Field decoding module is configured to output entity labels for each word in the text based on the multimodal fusion features.

[0050] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A multimodal named entity recognition method, characterized in that, Includes the following steps: S1: Collect text and images, and extract corresponding text features, multimodal text features, multimodal image features, and visual object features; S2: Obtain hidden text features based on the text features extracted in step S1; S3: Based on the text hidden features obtained in step S2, obtain all entity features in the collected text; S4: Based on the multimodal text features and multimodal image features in step S1, obtain the normalized multimodal image features and the text-image correlation score; S5: Based on the text-image relevance score obtained in step S4, the entity features obtained in step S3, and the visual object features obtained in step S1, output the penalized and filtered visual object features. S6: Based on the text hiding features obtained in step S2, the multimodal image features extracted in step S1, and the visual object features obtained in step S4, perform cross-modal semantic interactions between text hiding features and multimodal image features, and between text hiding features and visual object features, and then concatenate the corresponding output information to output multimodal fusion features. S7: Based on the multimodal fusion features output in step S6, output the entity label for each word in the text.

2. The multimodal named entity recognition method according to claim 1, characterized in that, Step S1 is as follows: The collected text is input into the text pre-trained model BERT, which outputs text features. ; The collected text and images are input into the CLIP multimodal pre-trained models, text encoder and image encoder, respectively, to obtain multimodal text features. Multimodal image features ; The collected images are input into the pre-trained model MASK RCNN to obtain visual objects, and the features of the visual objects are obtained by querying the category feature lookup table.

3. The multimodal named entity recognition method according to claim 1, characterized in that, Step S2 is as follows: Text features The input is fed into a 4-layer stacked self-attention transformer to capture the intramodal relations of the text, obtaining the text-hidden representation of each word. .

4. The multimodal named entity recognition method according to claim 1, characterized in that, Step S3 is as follows: Hiding text indicates Input a conditional random field to predict the label sequence of the text. ; Based on label sequence The tags in the text are hidden from view. Select the entity features corresponding to all entities ,in Representing the The characteristics of an entity This represents the total number of entities contained in the input sentence.

5. The multimodal named entity recognition method according to claim 1, characterized in that, Step S4 is as follows: Multimodal text features Multimodal image features Normalize them separately to obtain the normalized multimodal text features. Multimodal image features Using multimodal text features Multimodal image features To calculate the relevance score between the input text and the input image: In the formula, @ represents the dot product of two matrices, for easy visualization; Set a hyperparameter As a gating mechanism, it is used to determine whether the input text and the input image are related, and selects visual information related to the text as auxiliary information, specifically: if This indicates that the input text is related to the input image, and multimodal image features are incorporated. As visual information paired with the input text; Otherwise, it means the input text and input image are unrelated, and the multimodal text features will be used. As visual information paired with the input text.

6. The multimodal named entity recognition method according to claim 1, characterized in that, Step S5 is as follows: Penalizing visual object features based on the text-image correlation score using coarse-grained gating includes: constructing a mask matrix using the correlation score. ,pass Penalize visual object features; Visual features are filtered based on entity characteristics using fine-grained gating, including: using visual object features as... Entity features as The attention mechanism is used to calculate the relevance score between each visual region feature and each entity feature. The relevance scores of each visual region feature and all entity features are then summed to obtain the result. Each correlation score and, The number of features of the visual object; from Each correlation score and the top selected The visual region features corresponding to each value will be... The visual region features are summed and averaged to obtain the final visual region features that need to be input into the multimodal interaction module.

7. The multimodal named entity recognition method according to claim 1, characterized in that, Step S6 is as follows: Stack two cross-modal transformers to perform multimodal semantic interactions: Multimodal image features As Text hiding representation As and Input the first cross-modal transformer and output the word representation obtained from shallow visual perception. ; Hiding text indicates As Word representation based on shallow visual perception As and Input the second cross-modal transformer and output the word representation obtained from visual perception. ; Hiding text indicates As Multimodal image features As and Input another cross-modal transformer and output a visual representation of word perception. ; Use gate functions To obtain the final coarse-grained multimodal fusion features : In the formula, This represents the sigmoid activation function; , It is a weight matrix; Indicates features and To splice; By replacing the input of visual information from multimodal image features with visual object features, fine-grained multimodal fusion features are obtained. ; coarse-grained multimodal fusion features Features with fine-grained multimodal fusion Combining them yields multimodal fusion features. : In the formula, Represents a linear function; Indicates features and Then, the parts are assembled.

8. The multimodal named entity recognition method according to claim 1, characterized in that, Step S7 specifically involves: Multimodal fusion features The input is fed into the Conditional Random Field decoding module, which outputs a sequence of entity labels for each word in the text. : In the formula, Represents a text sequence. Represents all predicted label sequences of a text sequence. A sequence of labels, used to calculate each is The probability value of the label sequence. The function is used to select the label sequence corresponding to the highest probability value as the final label sequence.

9. A recognition system based on the multimodal named entity recognition method as described in claim 1, characterized in that, include: The feature extraction module is used to extract corresponding text features, multimodal text features, multimodal image features, and visual object features from the input text and images. The self-attention module is used to: output text hidden features based on the text features; The entity span recognition module is used to: output all entity features in the input text based on the text hiding features; The text-image relationship reasoning module is used to: output normalized multimodal image features and text-image correlation scores based on the multimodal text features and multimodal image features; The visual object filtering module is used to: output penalized and filtered visual object features based on the text-image relevance score, entity features, and visual object features; The multimodal interaction module is used to: perform cross-modal semantic interactions between the text hiding features and the multimodal image features, and between the text hiding features and the visual object features, respectively, based on the text hiding features, the multimodal image features, and the visual object features, and to concatenate the corresponding output information to output multimodal fusion features; The Conditional Random Field decoding module is used to: output the entity label of each word in the text based on the multimodal fusion features.

Images