Multi-modal entity identification method and device, terminal equipment and medium

Abstract

The invention discloses a multi-mode entity recognition method and device, terminal equipment and a medium, and the method comprises the steps: obtaining input data which at least comprises text data and image data; according to the input data and a pre-trained recognition model, entity information corresponding to the input data is determined, and the pre-trained recognition model is used for extracting text features and image features of multi-modal sample data and fusing the text features and the image features to obtain fused features; according to the embodiment of the invention, a multi-modal feature fusion coding module is added, and the module uses three visual features to represent various aspects of image semantics. And finally, integrating the features through cooperative representation, and jointly decoding the text and the multi-modal representation by using a shared multi-task label decoder so as to carry out entity prediction, thereby improving the accuracy of multi-modal entity prediction.

Description

Technical Field

[0001] The present application belongs to the field of data processing technology, and in particular relates to a multimodal entity recognition method, apparatus, terminal device and medium. Background Art

[0002] As social posts increasingly exhibit multimodal characteristics, the multimodal named entity recognition (MNER) task on social media plays a vital role in various downstream tasks, such as targeted advertising and social user location. However, existing MNER methods face challenges such as the semantic gap between text and images and ambiguity in multimodal representations.

[0003] With the widespread application of multimedia technologies and the continuous development of social media platforms, multimodal data including text, images, and audio is increasingly emerging. This data is a key source of information for many applications, including social media user detection and public opinion analysis. Traditional named entity recognition (NER) relies primarily on internal sentence features (such as vocabulary and grammar) and external textual data (such as knowledge graphs and networks) to extract semantic information to support information extraction. However, as multimodal data gradually replaces real-world data, traditional NER faces significant challenges. Therefore, multimodal NER (MNER) has become a hot research topic. By adding photos as additional input, MNER improves traditional text-based NER. Therefore, multimodal NER has emerged to improve traditional text-based NER by adding images. For example, if only text exists, the entity type of the named entity "Mitty" cannot be inferred as it could be of type PER or type MISC.

[0004] Existing MNER systems achieve the fusion and alignment of textual and visual information through various cross-modal methods. Although significant progress has been made, there are still some limitations:

[0005] (1) Text-image mismatch: The images paired with the text are not necessarily perfectly aligned; they may be partially or completely mismatched. Therefore, even when there is little correlation between the text and the image, the text should still be ranked first. For example, if the photo that the text matches does not contain any information related to "ABC", it is difficult to determine whether "ABC" is of MISC type or LOC type. This is an example of text-image mismatch. In this case, the lack of effective correlation between text and image poses a challenge to multimodal named entity recognition.

[0006] (2) Limited utilization of visual context: Current methods typically use a cross-modal attention mechanism to promote semantic interaction between word representations and visual features, thereby achieving feature fusion between text and image modalities. However, this mechanism focuses primarily on the interaction of global information between modalities, while ignoring the fine-grained semantic correspondence and alignment between text and image visual objects. Although text and images are highly correlated, under the influence of the attention mechanism, high-scoring regions are not necessarily highly correlated with named entities in the text. Instead, trivial visual features that affect text comprehension are introduced. Therefore, how to improve the accuracy of multimodal entity recognition is an urgent problem that needs to be solved. Summary of the Invention

[0007] The present application aims to provide a multimodal entity recognition method, apparatus, terminal device and storage medium to address the deficiencies in the prior art. The technical problems to be solved by the present application are achieved through the following technical solutions.

[0008] In a first aspect, an embodiment of the present application provides a multimodal entity recognition method, the method comprising:

[0009] Acquiring input data, wherein the input data includes at least text data and image data;

[0010] According to the input data and a pre-trained recognition model, entity information corresponding to the input data is determined, wherein the pre-trained recognition model extracts text features and image features from multimodal sample data, and fuses the text features and image features to obtain fused features, which are then used to train a training model based on a cross-modal attention mechanism.

[0011] Optionally, the pre-trained recognition model is obtained by:

[0012] Acquiring multimodal sample data, wherein the multimodal sample data includes at least sample text data and sample image data;

[0013] Extracting text features and image features from the multimodal sample data respectively, and determining fusion features corresponding to the multimodal sample data;

[0014] Training the training model based on the cross-modal attention mechanism according to the fusion features;

[0015] Calculating a loss function during training, wherein the loss function includes at least a maximum conditional likelihood loss function and a divergence loss function;

[0016] When the function value of the loss function meets a preset condition, the training model corresponding to the function value is determined as the recognition model.

[0017] Optionally, extracting text features and image features from the multimodal sample data respectively to determine fusion features corresponding to the multimodal sample data includes:

[0018] Obtaining a word input sequence in the sample text data;

[0019] Determine an enhanced semantic feature vector using the BERT model and the word input sequence;

[0020] Processing the sample image data to obtain a local image feature vector, an image visual feature vector, and a global image feature vector, and fusing the local image feature vector, the image visual feature vector, and the global image feature vector to obtain a fused visual feature vector;

[0021] A visual space feature encoding vector is determined based on the enhanced semantic feature vector and the fused visual feature vector by utilizing spatial encoding and learning between graph nodes.

[0022] Optionally, the using the BERT model and the word input sequence to determine the enhanced semantic feature vector includes:

[0023] Using the BERT model to encode the word input sequence to obtain a text feature vector;

[0024] The text feature vector is semantically enhanced according to a pre-trained Word2Vec model to obtain an enhanced semantic feature vector.

[0025] Optionally, the training model based on the cross-modal attention mechanism is trained according to the fusion feature, including:

[0026] Using a multi-task label decoder to perform label decoding on the hidden vector in the fusion feature;

[0027] Use KL divergence loss function for label alignment;

[0028] The loss summation method is used to obtain the label prediction after training;

[0029] The training model based on the cross-modal attention mechanism is trained according to the trained label prediction and the preset label sequence.

[0030] Optionally, calculating the loss function during the training process includes:

[0031] Inputting the enhanced semantic feature vector, the fused visual feature vector, and the visual space feature encoding vector into a network model of a multi-head attention mechanism, respectively, performing fine-grained semantic decoding on word semantics and multimodal representations, and obtaining text representation data and visual representation data with enhanced text semantics;

[0032] Predicting a hidden vector based on the projection function, the text representation data enhanced with text semantics, and the visual representation data to obtain a prediction sequence;

[0033] Using a conditional random field to label the predicted sequence, a transition score, an emission score and weight parameters are determined;

[0034] Calculating a loss value for multi-task label decoding using a maximum conditional likelihood function loss function and the transition score, emission score, and weight parameters;

[0035] The Kullback-Leibler divergence loss function is used to calculate the degree of alignment between predicted labels;

[0036] The function value of the loss function during training is calculated based on the alignment between the loss value of multi-task label decoding and the predicted label.

[0037] In a second aspect, an embodiment of the present application provides a multimodal entity recognition device, the device comprising:

[0038] An acquisition module, configured to acquire input data, wherein the input data includes at least text data and image data;

[0039] The recognition module is used to determine the entity information corresponding to the input data based on the input data and a pre-trained recognition model, wherein the pre-trained recognition model extracts text features and image features from multimodal sample data and fuses the text features and image features to obtain fused features, which are used to train a training model based on a cross-modal attention mechanism.

[0040] Optionally, the device further comprises a training module, wherein the training module is configured to:

[0041] Acquiring multimodal sample data, wherein the multimodal sample data includes at least sample text data and sample image data;

[0042] Extracting text features and image features from the multimodal sample data respectively, and determining fusion features corresponding to the multimodal sample data;

[0043] Training the training model based on the cross-modal attention mechanism according to the fusion features;

[0044] Calculating a loss function during training, wherein the loss function includes at least a maximum conditional likelihood loss function and a divergence loss function;

[0045] When the function value of the loss function meets a preset condition, the training model corresponding to the function value is determined as the recognition model.

[0046] Optionally, the training module is used to:

[0047] Obtaining a word input sequence in the sample text data;

[0048] Determine an enhanced semantic feature vector using the BERT model and the word input sequence;

[0049] Processing the sample image data to obtain a local image feature vector, an image visual feature vector, and a global image feature vector, and fusing the local image feature vector, the image visual feature vector, and the global image feature vector to obtain a fused visual feature vector;

[0050] A visual space feature encoding vector is determined based on the enhanced semantic feature vector and the fused visual feature vector by utilizing spatial encoding and learning between graph nodes.

[0051] Optionally, the training module is used to:

[0052] Using the BERT model to encode the word input sequence to obtain a text feature vector;

[0053] The text feature vector is semantically enhanced according to a pre-trained Word2Vec model to obtain an enhanced semantic feature vector.

[0054] Optionally, the training module is used to:

[0055] Using a multi-task label decoder to perform label decoding on the hidden vector in the fusion feature;

[0056] Use KL divergence loss function for label alignment;

[0057] The loss summation method is used to obtain the label prediction after training;

[0058] The training model based on the cross-modal attention mechanism is trained according to the trained label prediction and the preset label sequence.

[0059] Optionally, the training module is used to:

[0060] Inputting the enhanced semantic feature vector, the fused visual feature vector, and the visual space feature encoding vector into a network model of a multi-head attention mechanism, respectively, performing fine-grained semantic decoding on word semantics and multimodal representations, and obtaining text representation data and visual representation data with enhanced text semantics;

[0061] Predicting a hidden vector based on the projection function, the text representation data enhanced with text semantics, and the visual representation data to obtain a prediction sequence;

[0062] Using a conditional random field to label the predicted sequence, a transition score, an emission score and weight parameters are determined;

[0063] Calculating a loss value for multi-task label decoding using a maximum conditional likelihood function loss function and the transition score, emission score, and weight parameters;

[0064] The Kullback-Leibler divergence loss function is used to calculate the degree of alignment between predicted labels;

[0065] The function value of the loss function during training is calculated based on the alignment between the loss value of multi-task label decoding and the predicted label.

[0066] In a third aspect, an embodiment of the present application provides a terminal device, comprising: at least one processor and a memory;

[0067] The memory stores a computer program; the at least one processor executes the computer program stored in the memory to implement the multimodal entity recognition method provided in the first aspect.

[0068] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed, it implements the multimodal entity recognition method provided in the first aspect.

[0069] The embodiments of the present application include the following advantages:

[0070] The multimodal entity recognition method, apparatus, terminal device and medium provided in the embodiment of the present application obtain input data, wherein the input data includes at least text data and image data; determine the entity information corresponding to the input data based on the input data and a pre-trained recognition model, wherein the pre-trained recognition model extracts text features and image features from multimodal sample data, and fuses the text features and image features to obtain fused features, which are obtained by training a training model based on a cross-modal attention mechanism using the fused features. The embodiment of the present application adds a multimodal feature fusion encoding module, which uses three visual features to represent various aspects of image semantics. Finally, these features are integrated through collaborative representation, and a shared multi-task label decoder is used to jointly decode text and multimodal representations for entity prediction, thereby improving the accuracy of multimodal entity prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] In order to more clearly illustrate the embodiments of the present application or the existing technical solutions, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0072] Figure 1 This is a flowchart of a multimodal entity recognition method in one embodiment of the present application;

[0073] Figure 2 This is a schematic diagram of model training in one embodiment of the present application;

[0074] Figure 3 This is a diagram of the multi-task label decoding process in one embodiment of the present application;

[0075] Figure 4 This is a structural block diagram of an embodiment of a multimodal entity recognition device of the present application;

[0076] Figure 5 This is a schematic diagram of the structure of a terminal device of the present application. DETAILED DESCRIPTION

[0077] To make the objectives, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0078] An embodiment of the present application provides a multimodal entity recognition method for identifying multimodal entities. The embodiment is performed by a multimodal entity recognition apparatus, which is provided on a terminal device, for example, the terminal device at least includes a computer terminal.

[0079] Reference Figure 1 , shows a flowchart of a multimodal entity recognition method embodiment of the present application, which may specifically include the following steps:

[0080] S101, obtaining input data, wherein the input data includes at least text data and image data;

[0081] Specifically, the terminal device obtains input data, which is multimodal data and includes at least text data and image data.

[0082] S102. Determine entity information corresponding to the input data based on the input data and a pre-trained recognition model, wherein the pre-trained recognition model extracts text features and image features from multimodal sample data, and fuses the text features and image features to obtain fused features, which are then used to train a training model based on a cross-modal attention mechanism.

[0083] Specifically, the terminal device pre-extracts text features and image features from the multimodal sample data, and fuses the text features and image features to obtain fused features. The fused features are used to train a training model based on the cross-modal attention mechanism to obtain a recognition model.

[0084] The terminal device processes the acquired multimodal data to obtain a feature vector, and inputs the feature vector into the recognition model to obtain entity information corresponding to the input data.

[0085] Another embodiment of the present application further supplements the multimodal entity recognition method provided in the above embodiment.

[0086] Optionally, a pre-trained recognition model is obtained by:

[0087] Acquiring multimodal sample data, wherein the multimodal sample data includes at least sample text data and sample image data;

[0088] Extracting text features and image features from the multimodal sample data respectively, and determining fusion features corresponding to the multimodal sample data;

[0089] According to the fusion features, the training model based on the cross-modal attention mechanism is trained;

[0090] Calculating the loss function during the training process, wherein the loss function includes at least a maximum conditional likelihood loss function and a divergence loss function;

[0091] When the function value of the loss function meets the preset conditions, the training model corresponding to the function value is determined as the recognition model.

[0092] The recognition model provided in the embodiment of the present application includes:

[0093] (1) Text feature encoding: The BERT model is used for text feature extraction and semantic enhancement is introduced to solve the problems of incomplete semantic description and data sparsity caused by the informality and conciseness of social network content.

[0094] (2) Multimodal feature encoding: A multi-angle feature representation extraction method is used to explore the semantic interactions between specific visual features and text features. Specifically, by using three different visual models to represent the complete semantic information of the image, the correspondence between fine-grained semantic elements such as objects, attributes, and relationships in text and images is strengthened, thereby enhancing the semantic richness and accuracy of multimodal representation.

[0095] (3) Image space coding: This part generates a spatial structure diagram by calculating the relative spatial distance between objects to solve the problem that objects in visual space are often scattered and irregular.

[0096] (4) Label prediction: This part first inputs the feature encoding obtained from the above three parts into the multi-head attention mechanism to obtain the text representation T and visual representation K with enhanced text semantics. Then, a shared multi-task label decoder is used to jointly decode the text and multimodal representations, capture the fine-grained entity semantics in the multimodal representation, and use this semantic information to generate predicted labels.

[0097] Optionally, extracting text features and image features from the multimodal sample data separately to determine fusion features corresponding to the multimodal sample data includes:

[0098] Get a word input sequence in sample text data;

[0099] Using the BERT model and word input sequence, we determine the enhanced semantic feature vector.

[0100] Processing the sample image data to obtain a local image feature vector, an image visual feature vector and a global image feature vector, and fusing the local image feature vector, the image visual feature vector and the global image feature vector to obtain a fused visual feature vector;

[0101] The spatial encoding and learning between graph nodes are used to determine the visual spatial feature encoding vector based on the enhanced semantic feature vector and the fused visual feature vector.

[0102] Specifically, given a pair of inputs, each of which consists of a text sentence X and an image I. Finding words in X and classifying them with appropriate entity categories is the goal of MNER, which is considered as a sequence labeling task. Let Y = {y0,y1,...,y M-1} represents the matching tag sequence, let X={x0,x1,...,x M-1} represents the input sequence of M words, let I = {o0,o1,...,o n} represents the target detected in the image. i ∈ζ, ζ represents the predefined label set in the standard BIO2 labeling scheme.

[0103] The embodiment of this application is about text feature encoding, which is divided into two parts: text feature extraction and semantic enhancement, such as Figure 2 The first part of the present application embodiment is used to encode text features. Before being input into the text Transformer encoder, each word in the two different representations is combined through a simple concatenation and linear transformation.

[0104] Optionally, a BERT model and a word input sequence are used to determine an enhanced semantic feature vector, including:

[0105] Use the BERT model to encode the word input sequence to obtain the text feature vector;

[0106] The text feature vector is semantically enhanced according to the pre-trained Word2Vec model to obtain the enhanced semantic feature vector.

[0107] Because BERT performs well in dynamically extracting features of the same word in multiple contexts, BERT is used as the encoder for text feature extraction. Given an input sequence X = {x0, x1, ..., x M-1}, input X into BERT to obtain the token-level text representation S', as shown in formula (1).

[0108] S′=BERT(x i ,θ bert ) (1)

[0109] where S′={S′0,S′1,...,S′ n-1}, S′ irepresents the i-th word feature output in the encoder, n is the feature length, d is the feature dimension of the hidden layer, and θ bert is the parameter of bert.

[0110] Among them, text semantic enhancement can significantly improve the performance of a series of NLP tasks. For the problems of incomplete semantic description and sparse data caused by the informality and conciseness of social network content, semantic enhancement is introduced to enrich the context word representation of each input word.

[0111] In particular, we consider the N most relevant words for each input word in the embedding space, which are denoted as {ω i1 ,ω i2 ,...ω iN}, to generate a semantically improved form of each word. Since not all N words are equally helpful for label prediction, a cautious module is adopted to weigh the influence of various words. For comparable words ω ij , whose weights are initially as shown in equation (2):

[0112]

[0113] where h i and h ij are respectively embedded from the pre-trained Word2Vec model x i and ω ij ; N is the N most relevant words for each input word; h in is the ω embedded from the pre-trained Word2Vec model ij .

[0114] Then, calculate x i Semantic enhancement representation S i ”, as shown in formula (3):

[0115]

[0116] Where: α ij For ω ij The weight of .

[0117] Finally, the final representation S can be obtained through equation (4):

[0118]

[0119] Where: [·,·] represents the concatenation operation, and Linear(·) is the dimension-aligned linear transformation.

[0120] Optionally, a training model based on a cross-modal attention mechanism is trained based on the fused features, including:

[0121] A multi-task label decoder is used to decode the labels of the hidden vectors in the fusion features;

[0122] Use KL divergence loss function for label alignment;

[0123] The loss summation method is used to obtain the label prediction after training;

[0124] The training model based on the cross-modal attention mechanism is trained according to the trained label predictions and the preset label sequence.

[0125] Specifically, multimodal feature encoding is an extension of text feature encoding, which is mainly divided into three parts: local visual feature extraction, visual label description and global visual feature extraction, such as Figure 2 As shown in the second part of . Considering the uniqueness of each visual feature in expressing different semantic information in an image, the embodiments of the present application adopt a multi-angle feature representation extraction method to explore the semantic interaction between specific visual features and text features. By using three different visual models to represent the complete semantic information of the image, these methods aim to strengthen the correspondence between fine-grained semantic elements such as objects, attributes, and relationships in text and images, thereby improving the semantic richness and accuracy of multimodal representation.

[0126] The embodiments of this application include extracting local image features from image data. Image-level features capture the overall visual information of the image, such as scene and category. However, due to different objectives, features extracted directly from general visual models often cannot meet the specific requirements of the MNER task.

[0127] Faster-RCNN-ECA helps the model better understand and distinguish features of various categories by adaptively adjusting the weights of different features. It introduces the efficient channel attention (ECA) mechanism based on the original Faster-RCNN to effectively suppress noise and irrelevant features and enhance the model's sensitivity to contextual information. Since ECA can effectively highlight important features and suppress irrelevant features, in order to solve the semantic gap problem in a single visual feature, Faster RCNN ECA is used to extract local object feature representations from the input image, as shown in Equation (5):

[0128] G=L(f i ,θ) (5)

[0129] Where G={G0,G1,...,G n-1}, contains the embedding of n label words, f i is the object region feature extracted by Faster-RCNN-ECA; θ is the learnable parameter of the linear transformation.

[0130] The present embodiment also includes image description generation for image data. In the Image Caption Generation module, entities in the entire tweet are identified by using text and images. Inspired by the ViECap framework, a transferable caption generation method based on CLIP is adopted. The present embodiment is trained on a plain text corpus, which improves the interpretability of text-image integration and enables the capture of each entity in the image. The image converter ViECap is used to process the input image. By extracting visual descriptions to describe the fine-grained visual semantics of the image, it is converted into a textual representation H as shown in Equation (6):

[0131] H=ViECap(I) (6)

[0132] In the embodiment of the present application, global image feature extraction is performed on image data. It is a standard practice to use convolutional neural networks (CNNs) to extract visual information from photos. Previous work mainly uses ResNet to capture overall image information. However, these technologies cannot fully utilize the fine-grained information in photos. The solution to this problem is to split the input image into N×N image patches, which will be called l1, l2, ..., l N×N , and then input into the visual encoder. In order to represent the morphological information and hidden relationship information of image features, ViT is used as the visual encoder. Next, a linear layer is added to obtain the feature dimension of the text feature, as shown in Equation (7):

[0133] L0,L1,...,L N =ViT({l1,l2,...,l N×N}) (7)

[0134] in is the Nth output of the image encoder. The size of the image feature hidden layer is represented by d, which is the feature dimension of the hidden layer.

[0135] In the embodiment of the present application, objects in the visual space are often scattered and irregular, which is in sharp contrast to the semantic inheritance of the system. Therefore, a spatial modeling structure diagram is chosen to be constructed.

[0136] First, derive the spatial relationship r ij Between any two objects i and o j Based on their area sizes and positions, calculate their relative distance d ij Then, a spatial structure graph G = (V, E) is generated, where V = {I, o1, ..., o N} represents the set of nodes, and E represents the set of connecting edges. Subsequently, the improved R-GCN with the initial spatial features is used to update the node representation, as shown in Equation (8):

[0137]

[0138] Where φ is a nonlinear function, Ω(i) represents the neighbor set of i, r ij Indicates the direction of the relationship between i and j, Displays the transformation weights of labels and edge directions. b is the bias term, and It refers to the representation of j at the kth layer.

[0139] Finally, a gating mechanism is used to modify the depiction in the k+1 layer. By using the spatial encoding and learning between graph nodes, the visual space feature encoding vector V1{v [IMG] ,v1,v2,...,v N}.

[0140] Optionally, calculate the loss function during training, including:

[0141] The enhanced semantic feature vector, fused visual feature vector, and visual space feature encoding vector are respectively input into the network model of the multi-head attention mechanism to perform fine-grained semantic decoding on word semantics and multimodal representations, thereby obtaining text representation data and visual representation data with enhanced text semantics.

[0142] Based on the projection function and the text representation data enhanced by text semantics and the visual representation data, the hidden vector is predicted to obtain a prediction sequence;

[0143] Use conditional random fields to label the predicted sequence and determine the transition score, emission score and weight parameters;

[0144] The loss value of multi-task label decoding is calculated using the maximum conditional likelihood loss function and the transition score, emission score and weight parameters;

[0145] The Kullback-Leibler divergence loss function is used to calculate the degree of alignment between predicted labels;

[0146] The function value of the loss function during training is calculated based on the alignment between the loss value of multi-task label decoding and the predicted label.

[0147] Specifically, label prediction in this embodiment is divided into three steps: label decoding, label alignment, and loss function. The first step is to use a multi-task label decoder (MLD) to decode the input latent vector M; the second step is to use the KL divergence loss function for label alignment; and finally, the final label prediction is obtained by summing the losses.

[0148] The Multi-Task Label Decoder (MLD) aggregates the prediction vectors of NER, entity boundary detection, and entity category detection, and collaboratively mines the fine-grained entity semantics in the features. This improves the quality of the input features in the CRF, thereby improving the accuracy of NER. Figure 3 The structure of the MLD model is shown.

[0149] First, the semantically enhanced text feature vector S, the fused visual feature vector V1, and the visual space encoding V2 are respectively input into the multi-head attention mechanism to perform fine-grained semantic decoding of word semantics and multimodal representations to obtain the text representation T and visual representation K with text semantic enhancement.

[0150] Then, let the hidden representation M = T, M = K, and use the projection function FC to transform the hidden vector Projected onto the subspace of the three tasks, we get M bio ,M plo ,M ner Then, the matrix and M bio , M plo Perform a cross product operation The fusion results in a more semantically accurate prediction vector Y, as shown in equation (9):

[0151]

[0152] Considering the dependencies between labels, a conditional random field is used to label the predicted sequence in , as shown in equations (10), (11), and (12):

[0153]

[0154] in refers to the label y i to y i+1 The transition score, is the label y i The emission fraction of is the label y i Unique weight parameter, α i is the multimodal feature of the input CRF layer.

[0155] Then, the maximum conditional likelihood loss function is used to calculate the loss of multi-task label decoding, as shown in Equation (13):

[0156]

[0157] Using MLD() to represent the multi-task label decoder, for the input hidden vector M, the above formula can be expressed as Equation (14):

[0158]

[0159] Although multimodal visual features provide richer semantics, they also introduce more complex visual semantics and noise, which may lead to semantic differences in multimodal representations and cause prediction errors. To address this issue, the Kullback-Leibler (KL) divergence loss function is adopted to minimize the difference between the prediction vectors of text and multimodal representations, thereby filtering out visual noise and guiding the multimodal representation to learn the accurate semantics of the best predicted label, as shown in Equation (15):

[0160]

[0161] The sum loss method is used to combine various tasks of joint learning and obtain the final prediction label of the model. The loss is expressed as shown in equation (16):

[0162]

[0163] in, They are used to evaluate the semantic accuracy of text representation and multimodal representation, L KL Used to evaluate the degree of alignment between predicted labels to address the problem of semantic bias.

[0164] The embodiment of the present application proposes a new semantically enhanced multimodal entity recognition method SEM. The method proposes to use the Transformer architecture and cross-modal attention mechanism to explore the semantic complementarity between text and visual features. Specifically, a semantic enhancement method is designed to strengthen the external support for text information. In addition, a multimodal feature fusion encoding module is introduced, which uses three visual features to represent various aspects of image semantics. Finally, these features are integrated through collaborative representation, and a shared multi-task label decoder is used to jointly decode text and multimodal representations for entity prediction. Experimental results on two public Twitter datasets demonstrate the superiority of the proposed model.

[0165] This application example proposes a new semantic enhancement method to improve the semantics of image text to address the aforementioned challenges. The model first uses a Transformer and a cross-modal attention mechanism to explore the semantic complementarity between text and visual features. To alleviate the problems of incomplete text information and data sparsity, a semantic enhancement method is provided to enhance external support for text information.

[0166] To address the issue of limited visual context utilization, a multimodal feature fusion module is introduced that leverages three types of visual features to represent different aspects of image semantics. These features are then combined through collaborative representation to describe the complete image semantics, which are subsequently fused with natural language features using a Transformer model (a deep learning model based on a self-attention mechanism). To handle the heterogeneity of character features, regional visual features, and BERT encoding features, a cross-modal attention mechanism is employed to explore semantic complementarity, guiding the fusion process to generate multimodal representations with enhanced text-image semantics. Finally, a shared multi-task label decoder is used to jointly decode the text and multimodal representations for entity prediction. Tests on two publicly accessible Twitter datasets show how effective the proposed approach is.

[0167] The embodiment of this application proposes a new semantic enhancement method to strengthen the external support for text information. It uses Transformer and cross-modal attention mechanism to explore semantic complementarity and provides a multimodal feature fusion encoding method. It uses BERT to jointly encode global and local image descriptions, subtitles and text to promote the fusion of text and image semantics. Through extensive experiments and comprehensive analysis, strong evidence is provided that SEM is competitive with existing SOTA models in the field. The model SEM is evaluated using two publicly available multimodal NER datasets and compared with multiple methods.

[0168] The multimodal sample data used in the embodiments of this application are from the Twitter-2015 and Twitter-2017 datasets based on Twitter. TWITTER-2015 contains 12,800 entities and 8,257 tweets. TWITTER-2017 contains 4,819 tweets from 8,724 entities. The dataset is divided into three parts: training set, validation set, and test set. Each text in the dataset is linked to an image. The number of text-image pairs is shown in Table 1 (Number of text images). In addition, the data volume statistical details of personal names (PER), place names (LOC), organization names (ORG), and MISC are shown in Table 1.

[0169] Table 1. Basic statistics of Twitter in 2015 and 2017

[0170]

[0171] The proposed model is compared with some standard high-quality NER methods, such as multimodal methods (text-image pairs as input) and unimodal methods (text only as input). Regarding unimodal methods, the following four classic models are used for comparison:

[0172] BiLSTM CRF: It is a highly traditional neural network model for sequence annotation that has influenced many other studies by automatically learning contextual word representations without requiring extensive manual feature engineering.

[0173] CNN-BiLSTM-CRF: It extends the work of BiLSTM CRF and applies the character-level word representations learned by CNN to Twitter data.

[0174] BERT: applies a language model to MNER using softmax, which is pre-trained on a large amount of unlabeled text input.

[0175] BERT-CRF: The decoder (developed by BERT) uses normal CRF layers instead of softmax layers.

[0176] For multimodal methods, various classic models are compared as follows:

[0177] GAVTT-BERT-CRF and AdaCAN BERT CRF: They replaced the original sentence encoder BiLSTM with BERT by combining visual attention with whole image features.

[0178] UMT: uses a cross-modal transformer to encode image-text pairs and creates an entity span recognition module to reduce bias when combining visual data.

[0179] UMGF: Both improve cross-modal interactions between text words and visual objects through GNN.

[0180] MAF: Improve the consistency between text and images by designing additional contrasting exercises, assigning visual cues to support words.

[0181] MNER-QG and ITA: Use prior knowledge of images, captions, and OCR to guide visual perception of word information.

[0182] MGCMT is a preliminary attempt at multi-level semantic alignment at different visual levels.

[0183] The experimental results were evaluated based on F1, recall, and overall precision. To ensure fairness, some baseline findings using the same dataset were cited, and the comparison results are shown in Table 2. The results show that the method achieves published state-of-the-art performance on both benchmark datasets. In addition, the following conclusions are drawn:

[0184] Table 2. Comparison of model performance for different texts and text images

[0185]

[0186] (1) BERT-based techniques outperform BiLSTM-based encoders on the dataset, indicating that the strong external knowledge support of the pre-trained model contributes to its high efficiency. The results show that CRF improves NER performance by benefiting from the link constraints between consecutive labels.

[0187] (2) By comparing all multimodal methods with unimodal methods, it is observed that unimodal methods generally perform worse than multimodal methods, which suggests that visual information is valuable to multimodal methods. When comparing UMGF and UMT, UMGF outperforms UMT, likely because fine-grained visual objects are more capable of successfully guiding more entities. The results of UMT and MAF demonstrate how important it is to evaluate the level of text-image matching and remove visual noise.

[0188] (3) The results of ITA and MNER-QG show that it is successful to support NER by converting image content into textual knowledge. However, this approach relies on other resources such as OCR models and manually crafted templates.

[0189] (4) The results of MGCMT show that multi-level semantic interactions are generally more beneficial than single-granularity information. SEM achieves a more comprehensive visual semantics by collaboratively expressing various visual features, effectively exploring the semantic interactions between text features and these visual features, thereby obtaining a multimodal representation with enhanced text-image semantics and improving the NER capability.

[0190] (5) Compared with the UMT and MAF models based on joint image-text encoding, the SEM model has significantly improved the Precision, Recall, and F1 evaluation indicators. In addition, SEM increases the F1 value by 1.16% and 0.02% on the Twitter-2015 and Twitter-2017 datasets respectively compared with the state-of-the-art models. These performance improvements are attributed to the multi-task label decoding module, which enhances the label decoding ability.

[0191] Based on the above embodiment, an ablation experiment can be performed:

[0192] To confirm the effectiveness of different SEM model components, ablation experiments were performed. Table 3 shows the comparison of the entire model and its ablated version (expressing UL as a single visual feature learning task, GV as spatial visual encoding, and MLD as a multi-task label decoder).

[0193] The research results of the examples of this application are as follows:

[0194] (1) SEM performs better when all three modules are included; when any component is removed, performance suffers.

[0195] (2) The UL module plays an important role in MNER, as removing the UL module (including visual labels, visual descriptions (KL), and local visual features) leads to a decrease in overall performance. This decrease is particularly evident in single-object scenes in both datasets, indicating that it is useful to understand the relationships between local objects in visual scenes.

[0196] (3) The removal of the GV module weakens the model’s ability to record key data about inter-object relations and scene knowledge, both of which are necessary for multimodal reasoning.

[0197] (4) The elimination of the MLD module leads to a significant drop in performance, probably because without it, the model can only guide the learning of multimodal representations of entity semantics and lacks the components required for entity boundary detection and semantic learning.

[0198] Table 3. Ablation experiment comparison table

[0199]

[0200] Based on the above examples, a generalization experiment can be performed:

[0201] To evaluate the generalization ability of our model and compare methods, we performed cross-validation, considering the different data properties of the two datasets. Table 4 shows that the Twitter-2017 → Twitter-2015 dataset was used to test the model trained on the Twitter-2016 dataset, and vice versa. The results show that our model significantly outperforms the other models, which may confirm its higher generalization and transfer capabilities.

[0202] Table 4. Generalization performance comparison of SEM and other models

[0203]

[0204] To more clearly illustrate the progress of our model, we selected four test cases from the test dataset and compared them with the predictions of BERT-CRF and UMGF. We used the four test cases from the test set to compare our model with other leading methods. The predictions are denoted by × and √, indicating incorrect and correct predictions, respectively.

[0205] The embodiment of this application proposes a new semantic enhancement method to fully integrate the semantics of image text. The model initially uses BERT to jointly encode global and local image descriptions, image titles, and text, effectively fusing text-image semantics to form a multimodal feature NER module. The cross-modal attention mechanism is used to integrate the semantic information of character features with text features, solving the problem of text semantic gaps and obtaining a semantically enhanced text representation. The multi-task label decoder is then called to decode the text representation to form a text feature NER module. Comprehensive experiments and analysis show that the model is competitive with existing SOTA models.

[0206] The embodiment of the present application proposes a cross-modal semantic enhancement fusion method, which makes full use of the Transformer architecture and cross-modal attention mechanism to explore the semantic complementarity between text and visual features to obtain new SOTA results.

[0207] The multimodal entity recognition method provided by the embodiment of the present application obtains input data, wherein the input data includes at least text data and image data; determines the entity information corresponding to the input data based on the input data and a pre-trained recognition model, wherein the pre-trained recognition model extracts text features and image features from the multimodal sample data, and fuses the text features and image features to obtain fused features, which are obtained by training a training model based on a cross-modal attention mechanism using the fused features. The embodiment of the present application adds a multimodal feature fusion encoding module, which uses three visual features to represent various aspects of image semantics. Finally, these features are integrated through collaborative representation, and a shared multi-task label decoder is used to jointly decode the text and multimodal representations to perform entity prediction, thereby improving the accuracy of multimodal entity prediction.

[0208] Another embodiment of the present application provides a multimodal entity recognition device for executing the multimodal entity recognition method provided in the above embodiment.

[0209] Reference Figure 4 , shows a structural block diagram of an embodiment of a multimodal entity recognition device of the present application. The device may specifically include the following modules: an acquisition module 601 and a recognition module 602, wherein:

[0210] The acquisition module 601 is used to acquire input data, wherein the input data at least includes text data and image data;

[0211] The recognition module 602 is used to determine the entity information corresponding to the input data based on the input data and a pre-trained recognition model, wherein the pre-trained recognition model extracts text features and image features from multimodal sample data, and fuses the text features and image features to obtain fused features, which are used to train a training model based on a cross-modal attention mechanism.

[0212] Another embodiment of the present application further supplements the multimodal entity recognition device provided in the above embodiment.

[0213] Optionally, the device further includes a training module, the training module being configured to:

[0214] Acquiring multimodal sample data, wherein the multimodal sample data includes at least sample text data and sample image data;

[0215] Extracting text features and image features from the multimodal sample data respectively, and determining fusion features corresponding to the multimodal sample data;

[0216] According to the fusion features, the training model based on the cross-modal attention mechanism is trained;

[0217] Calculating the loss function during the training process, wherein the loss function includes at least a maximum conditional likelihood loss function and a divergence loss function;

[0218] When the function value of the loss function meets the preset conditions, the training model corresponding to the function value is determined as the recognition model.

[0219] Optionally, the training module is used to:

[0220] Get a word input sequence in sample text data;

[0221] Using the BERT model and word input sequence, we determine the enhanced semantic feature vector.

[0222] Processing the sample image data to obtain a local image feature vector, an image visual feature vector and a global image feature vector, and fusing the local image feature vector, the image visual feature vector and the global image feature vector to obtain a fused visual feature vector;

[0223] The spatial encoding and learning between graph nodes are used to determine the visual spatial feature encoding vector based on the enhanced semantic feature vector and the fused visual feature vector.

[0224] Optionally, the training module is used to:

[0225] Use the BERT model to encode the word input sequence to obtain the text feature vector;

[0226] The text feature vector is semantically enhanced according to the pre-trained Word2Vec model to obtain the enhanced semantic feature vector.

[0227] Optionally, the training module is used to:

[0228] A multi-task label decoder is used to decode the labels of the hidden vectors in the fusion features;

[0229] Use KL divergence loss function for label alignment;

[0230] The loss summation method is used to obtain the label prediction after training;

[0231] The training model based on the cross-modal attention mechanism is trained according to the trained label predictions and the preset label sequence.

[0232] Optionally, the training module is used to:

[0233] The enhanced semantic feature vector, fused visual feature vector, and visual space feature encoding vector are respectively input into the network model of the multi-head attention mechanism to perform fine-grained semantic decoding on word semantics and multimodal representations, thereby obtaining text representation data and visual representation data with enhanced text semantics.

[0234] Based on the projection function and the text representation data enhanced by text semantics and the visual representation data, the hidden vector is predicted to obtain a prediction sequence;

[0235] Use conditional random fields to label the predicted sequence and determine the transition score, emission score and weight parameters;

[0236] The loss value of multi-task label decoding is calculated using the maximum conditional likelihood loss function and the transition score, emission score and weight parameters;

[0237] The Kullback-Leibler divergence loss function is used to calculate the degree of alignment between predicted labels;

[0238] The function value of the loss function during training is calculated based on the alignment between the loss value of multi-task label decoding and the predicted label.

[0239] The multimodal entity recognition device provided by the embodiment of the present application obtains input data, wherein the input data includes at least text data and image data; determines the entity information corresponding to the input data based on the input data and a pre-trained recognition model, wherein the pre-trained recognition model extracts text features and image features from the multimodal sample data, and fuses the text features and image features to obtain fused features, which are obtained by training a training model based on a cross-modal attention mechanism using the fused features. The embodiment of the present application adds a multimodal feature fusion encoding module, which uses three visual features to represent various aspects of image semantics. Finally, these features are integrated through collaborative representation, and a shared multi-task label decoder is used to jointly decode the text and multimodal representations to perform entity prediction, thereby improving the accuracy of multimodal entity prediction.

[0240] Yet another embodiment of the present application provides a terminal device for executing the multimodal entity recognition method provided in the above embodiment.

[0241] Figure 5 This is a schematic diagram of the structure of a terminal device of the present application, such as Figure 5 As shown, the terminal device includes: at least one processor 701 and a memory 702;

[0242] The memory stores a computer program; and at least one processor executes the computer program stored in the memory to implement the multimodal entity recognition method provided in the above embodiment.

[0243] Yet another embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed, the multimodal entity recognition method provided by any of the above embodiments is implemented.

[0244] It should be noted that the above detailed description is exemplary and is intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the art to which this application belongs.

[0245] It should be noted that the terms used herein are intended only to describe specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0246] It should be noted that the terms "first," "second," and the like in the specification and claims of this application and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, such that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0247] In addition, the terms "comprises" and "comprising" and any variations thereof are intended to cover a non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, product, or apparatus.

[0248] For ease of description, spatially relative terms such as "above", "above", "on the upper surface of", "above", etc. may be used herein to describe the spatial positional relationship of a device or feature to other devices or features as shown in the figures. It should be understood that spatially relative terms are intended to include different orientations of the device in use or operation in addition to the orientation described in the figures. For example, if the device in the drawings is inverted, the device described as "above other devices or structures" or "above other devices or structures" will be subsequently positioned as "below other devices or structures" or "below other devices or structures". Thus, the exemplary term "above" can include both "above" and "below". The device can also be positioned in other different ways, such as rotated 90 degrees or in other orientations, and the spatially relative descriptions used herein are interpreted accordingly.

[0249] In the above detailed description, reference is made to the accompanying drawings, which form a part hereof. In the drawings, similar symbols typically identify similar components, unless the context dictates otherwise. The illustrated embodiments described in the detailed description, drawings, and claims are not meant to be limiting. Other embodiments may be used, and other changes may be made, without departing from the spirit or scope of the subject matter presented herein.

[0250] The above description is merely a preferred embodiment of the present application and is not intended to limit the present application. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present application shall be included within the scope of protection of the present application.

Claims

1. A multimodal entity recognition method, characterized in that: The method comprises: Acquiring input data, wherein the input data includes at least text data and image data; According to the input data and a pre-trained recognition model, entity information corresponding to the input data is determined, wherein the pre-trained recognition model extracts text features and image features from multimodal sample data, and fuses the text features and image features to obtain fused features, which are then used to train a training model based on a cross-modal attention mechanism.

2. The multimodal entity recognition method according to claim 1, characterized in that The pre-trained recognition model is obtained in the following way: Acquiring multimodal sample data, wherein the multimodal sample data includes at least sample text data and sample image data; Extracting text features and image features from the multimodal sample data respectively, and determining fusion features corresponding to the multimodal sample data; Training the training model based on the cross-modal attention mechanism according to the fusion features; Calculating a loss function during training, wherein the loss function includes at least a maximum conditional likelihood loss function and a divergence loss function; When the function value of the loss function meets a preset condition, the training model corresponding to the function value is determined as the recognition model.

3. The multimodal entity recognition method according to claim 2, characterized in that: The extracting text features and image features from the multimodal sample data respectively to determine fusion features corresponding to the multimodal sample data includes: Obtaining a word input sequence in the sample text data; Determine an enhanced semantic feature vector using the BERT model and the word input sequence; Processing the sample image data to obtain a local image feature vector, an image visual feature vector, and a global image feature vector, and fusing the local image feature vector, the image visual feature vector, and the global image feature vector to obtain a fused visual feature vector; A visual space feature encoding vector is determined based on the enhanced semantic feature vector and the fused visual feature vector by utilizing spatial encoding and learning between graph nodes.

4. The multimodal entity recognition method according to claim 3, characterized in that: The method of using the BERT model and the word input sequence to determine the enhanced semantic feature vector includes: Using the BERT model to encode the word input sequence to obtain a text feature vector; The text feature vector is semantically enhanced according to a pre-trained Word2Vec model to obtain an enhanced semantic feature vector.

5. The multimodal entity recognition method according to claim 2, characterized in that: The step of training the training model based on the cross-modal attention mechanism according to the fusion features includes: Using a multi-task label decoder to perform label decoding on the hidden vector in the fusion feature; Use KL divergence loss function for label alignment; The loss summation method is used to obtain the label prediction after training; The training model based on the cross-modal attention mechanism is trained according to the trained label prediction and the preset label sequence.

6. The multimodal entity recognition method according to claim 3, characterized in that: The loss function in the calculation training process includes: Inputting the enhanced semantic feature vector, the fused visual feature vector, and the visual space feature encoding vector into a network model of a multi-head attention mechanism, respectively, performing fine-grained semantic decoding on word semantics and multimodal representations, and obtaining text representation data and visual representation data with enhanced text semantics; Predicting a hidden vector based on the projection function, the text representation data enhanced with text semantics, and the visual representation data to obtain a prediction sequence; Using a conditional random field to label the predicted sequence, a transition score, an emission score and weight parameters are determined; Calculating a loss value for multi-task label decoding using a maximum conditional likelihood function loss function and the transition score, emission score, and weight parameters; The Kullback-Leibler divergence loss function is used to calculate the degree of alignment between predicted labels; The function value of the loss function during training is calculated based on the alignment between the loss value of multi-task label decoding and the predicted label.

7. A multimodal entity recognition device, characterized in that: The device comprises: An acquisition module, configured to acquire input data, wherein the input data includes at least text data and image data; The recognition module is used to determine the entity information corresponding to the input data based on the input data and a pre-trained recognition model, wherein the pre-trained recognition model extracts text features and image features from multimodal sample data and fuses the text features and image features to obtain fused features, which are used to train a training model based on a cross-modal attention mechanism.

8. The multimodal entity recognition device according to claim 7, characterized in that: The apparatus further comprises a training module, wherein the training module is configured to: Acquiring multimodal sample data, wherein the multimodal sample data includes at least sample text data and sample image data; Extracting text features and image features from the multimodal sample data respectively, and determining fusion features corresponding to the multimodal sample data; Training the training model based on the cross-modal attention mechanism according to the fusion features; Calculating a loss function during training, wherein the loss function includes at least a maximum conditional likelihood loss function and a divergence loss function; When the function value of the loss function meets a preset condition, the training model corresponding to the function value is determined as the recognition model.

9. A terminal device, characterized in that: include: at least one processor and memory; The memory stores a computer program; The at least one processor executes the computer program stored in the memory to implement the multimodal entity recognition method according to any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed, implements the multimodal entity recognition method according to any one of claims 1 to 6.

Images