An optimal feature selection multimodal named entity recognition method based on attention mechanism

Abstract

A multimodal named entity recognition method using optimal feature selection based on an attention mechanism belongs to the field of artificial intelligence. The technical solution includes: obtaining text and text-related images; converting the text-related images into image captions using an image caption generation model; concatenating the text and image captions as cross-modal text; extracting related keyword phrases from the cross-modal text using a keyword extraction model; obtaining contextual representations of the cross-modal text and related keyword phrases using a pre-trained model; and processing the contextual representations of the cross-modal text and related keyword phrases using a backdoor causal attention network. The present invention optimizes text and image features through the causal attention mechanism, fully narrowing the semantic gap between text and image modalities and improving the accuracy and robustness of multimodal named entity recognition.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence and natural language processing, and focuses on using the attention mechanism for multimodal named entity recognition, optimizing feature selection and improving recognition accuracy. Background Art

[0002] Named entity recognition is an important task in the field of natural language processing. Its goal is to identify entities with specific meanings in text and classify them into predefined categories. These entities are usually noun phrases, such as person names, locations, organizations, etc.

[0003] However, in pure text named entity recognition, a single data source often faces the challenges of entity ambiguity and contextual understanding bias, while image information can provide the model with additional contextual information to help the model identify and classify entities more accurately. For example, the "apple" mentioned in the text may refer to the fruit or to Apple Inc., but by combining image information, the model can more easily distinguish between the two situations. Therefore, it is very necessary to combine text data and image data for research. On the other hand, the research on multimodal named entity recognition can promote the ability to learn to represent it between different modalities, which not only improves the performance of named entity recognition, but also provides new research ideas for other multimodal tasks. This cross-modal learning ability helps the model have better generalization ability when facing different fields or tasks, especially when text information is not enough to determine the entity type, information from other modalities can provide key clues.

[0004] To better implement multimodal named entity recognition, we introduced some important theories from causal inference theory, including the causal intervention theory. Causal intervention is an effective way to process data feature representations. Specifically, causal intervention captures the true causal relationship in feature variables by introducing a structural causal model, allowing us to simulate or calculate the results after directly intervening in a certain variable, which is different from simply observing the natural changes of the variable. This intervention can be adding, removing, or changing certain attributes of the variable. In this way, we can estimate the causal effect between variables, that is, how one variable affects another variable.

[0005] Mutual information theory also provides a method for evaluating the mutual information between two data feature representations, helping us identify correlations between data features. Mutual information quantifies the interdependence between two random variables, measuring the amount of information one random variable contains about the other. This can help us better align data representations across different modalities.

[0006] In multimodal named entity recognition, the primary challenge is how to properly interact with image and text information: obtaining effective visual information while suppressing interfering information. However, existing research methods have certain limitations. Although methods based on fine-grained visual cues can allow the model to focus on more visual information, this visual information is chaotic, and the bias in the data prevents the model from obtaining the optimal feature representation. Secondly, data of different modalities have different data storage formats. There is a natural modality gap between image and text data. In addition, text features and image features are obtained by different encoders, which makes it more difficult for the model to obtain effective information from image and text pairs. To address the above problems, a multimodal named entity recognition method based on optimal feature selection of the attention mechanism is proposed. The core of this method is to utilize the powerful ability of the attention mechanism to obtain the optimal image and text feature representation, thereby improving the performance and robustness of multimodal named entity recognition. Summary of the Invention

[0007] In order to meet the needs in the above background, the present invention provides a multimodal named entity recognition method based on optimal feature selection of attention mechanism. First, the text data and image data of multimodal named entity recognition are respectively encoded using a pre-trained language model and a pre-trained image model. In the text modality, the backdoor causal attention mechanism is used to block the backdoor path and obtain the optimal features in the text feature representation. In the image modality, the frontdoor causal attention mechanism is used to obtain the optimal features in the image feature representation by introducing appropriate mediating variables. Finally, through a feature alignment optimizer based on mutual information theory, the semantic distance between image information and text information is shortened, the modal gap is narrowed, and a consistent text image representation is obtained, thereby making more accurate entity predictions.

[0008] The technical solution is as follows:

[0009] A multimodal named entity recognition method with optimal feature selection based on attention mechanism, the steps are as follows:

[0010] Step 1: Convert text-associated images into image captions through an image caption generation model. Image captions can provide more semantic information for text data. Text data and image captions are connected as cross-modal text data for further research. The image caption generation process is described as follows:

[0011] C=Caption(Image),

[0012] Among them, C is the image caption and Image is the text associated image.

[0013] Step 2: Extract related keyword groups from the cross-modal text using the keyword extraction model. Considering the length of the cross-modal text, limit the number of related keyword groups to three groups, and treat the related keyword groups and the cross-modal text together as text data. The keyword extraction process can be expressed as follows:

[0014] K = Keyword_extraction(T_D)

[0015] Among them, K is the keyword data set and T_D is the original text data.

[0016] Step 3: Use the visualization toolkit to obtain the k most prominent objects in the text-related image as local visual information, and the text-related image as global visual information. The two are combined as the image data of this method. The local image extraction process is expressed as:

[0017] V object =Visual_tool(V_D)

[0018] Among them, V object is the local visual object, and V_D is the original image data.

[0019] Step 4: Use the pre-trained language model and pre-trained visual model to encode the text data and image data respectively to obtain the corresponding text feature representation and image feature representation. The process of obtaining the text feature representation and image feature representation is described as follows:

[0020] H T =PLM(F_T)

[0021] H V =VLM(F_V)

[0022] Among them, H T is the text feature representation, H V is the image feature representation, F_T is the total text data, and F_V is the total image data.

[0023] Step 5: Design a backdoor causal attention mechanism to block the backdoor paths in the text data, so that the model focuses on the optimal text features. The process of selecting the optimal text features is expressed as:

[0024] H T _B=FD_cause(H T )

[0025] Among them, H T _B is the optimal text feature, FD_cause is the front-door causal attention mechanism, H T It is a text feature representation.

[0026] Step 6: Design a front-door causal attention mechanism. By introducing front-door variables as intermediaries, an effective front-door path is constructed to help the model estimate the direct causal relationship between image data and labels, allowing the model to focus on the optimal image features. The selection process of the optimal image features is expressed as:

[0027] H V _B=BD_cause(H V )

[0028] Among them, H V _B is the optimal image feature, BD_cause() is the backdoor causal attention mechanism, H V It is the image feature representation.

[0029] Step 7: For the optimal text features and image features obtained in steps 5 and 6, a feature alignment optimizer based on mutual information theory is designed to narrow the semantic distance between image information and text information, reduce the modality gap, and improve the accuracy of entity recognition. The feature alignment optimizer is expressed as:

[0030] (H T _BN,H V _BN)=L MI (H T _B,H V _B)

[0031] Among them, H T _BN is the aligned text feature, H V _BN is the image feature after alignment, L MI () is the feature alignment optimizer, H T _B is the optimal text feature, H V _B is the optimal image feature.

[0032] Step 8: Use conditional random fields to process the multimodal feature representation and predict entity information. The entity prediction process is expressed as:

[0033] EI=CRF(M_feature)

[0034] Among them, EI is the predicted entity information and M_feature is the multimodal feature representation.

[0035] Step 9: Feed the predicted entity information back into the attention-based neural network to further optimize the backdoor causal attention network, frontdoor causal attention network, and feature alignment optimizer to improve the prediction accuracy.

[0036] Step 10: Repeat steps 5 to 9 until the predetermined number of iterations is reached or the inference result meets the pre-set performance index.

[0037] Furthermore, for step 4, the pre-trained model has been trained on a large amount of data and has better generalization ability. Using the pre-trained model to process text and image data can extract deeper and more abstract features, improving the model's ability to process data.

[0038] Furthermore, for step 5, the backdoor causal attention mechanism is designed for text features, using cross-modal text and related keyword sets to approximate the confounding factor Z t , eliminate the existing backdoor path {X←Z t →Y}, we get the decongested data, and then combine it with the graph convolution operation to get the text features of the enhanced feature representation. The intervention distribution of the backdoor intervention is expressed as:

[0039]

[0040] Among them, Z is the confounding factor in the text feature, X represents the text feature, Y represents the target result, and do() represents the intervention operation.

[0041] Furthermore, for step 6, the front-door causal attention mechanism is designed for image features. By introducing mediating variables, a front-door path {X→M→Y} is constructed. Through intra-sample and inter-sample sampling, the confounding factors in the image modality are removed to obtain the optimal image features. The intervention distribution of the front-door intervention is expressed as:

[0042]

[0043] Among them, M is the mediating variable, X represents the image feature, Y represents the target result, and do() represents the intervention operation.

[0044] Furthermore, for step 7, a feature alignment optimizer based on mutual information theory is implemented in the form of contrastive learning. It uses the measurement power of mutual information to maximize relevant information and minimize irrelevant information, promoting consistency and relevance between different modalities. This improves the model's robustness for multimodal named entity recognition tasks. The objective function of the feature alignment optimizer is expressed as:

[0045] L MI =log(q(F T ; F V + ))+log(1-q(F T ; F V - ))

[0046] Among them, (F T ; F V + ) is the correct pairing of graphic representation, (F T; F V - ) is an incorrectly paired graphic representation.

[0047] Furthermore, the multimodal named entity recognition model includes:

[0048] Text Feature Extraction Layer: This method uses the BERT model to encode cross-modal text data and associated keyword phrases, obtaining deep semantic features of these data. As a pre-trained model, BERT can capture rich linguistic features and contextual information, providing rich text features for multimodal named entity recognition tasks. The process is as follows:

[0049] H t ={h1,h2,L h n}=BERT(T),

[0050] Among them, Ht is the text feature representation vector, h1, h2,…, hn correspond to text data x1, x2,…, xn, and T = {x1, x2,…, xn} is text data.

[0051] Image feature extraction layer: This layer uses the ResNet model to process image data, combining local object images (by default, four images) and global object images (i.e., text-related images) as image data. These images are then fed into the ResNet model to extract fine-grained features of the image. As a pre-trained model, ResNet extracts features rich in semantic information, providing rich image features for multimodal named entity recognition tasks. The process is described as follows:

[0052] H v ={v1,v2,L v n}=ResNet(V),

[0053] Among them, Hv is the image feature representation vector, v1, v2,…, vn corresponds to image data V1, V2,…, Vn, and V = {V1, V2,…, Vn} is the image data.

[0054] Backdoor Causal Attention Network Layer: This layer optimizes and selects text features through a backdoor causal intervention network based on the attention mechanism. This step can utilize the effect of backdoor causal intervention to remove confounding factors, filter out irrelevant features in the text features, and obtain a decongested text feature representation without the influence of confounding factors, thereby providing the optimal text features for multimodal named entity recognition. By utilizing the feature selection capability of the backdoor causal attention network, the model can predict entity information based on the optimal text features. The process of performing the decongestion operation by the approximate do operator can be expressed as:

[0055]

[0056] Here, the effect of the do operator is approximated by applying the normalized weighted geometric mean (NWGM).

[0057] Graph convolution layer: This layer uses the learned causal relationships to build fully connected edges, removes the convolutional features and uses them as graph nodes to build a graph convolution network, captures the local structural information in the feature graph, and enhances the spatial correlation of the feature graph. The calculation process can be expressed as:

[0058] y i =GCN(F i ,G_index),i=1,2,3,4

[0059] In this formula:

[0060] y i It is the deconvoluted representation of the enhanced feature representation.

[0061] GCN(,) is a graph convolution operation.

[0062] F i is the i-th decongested feature.

[0063] G_index defines the adjacency relationship of the graph.

[0064] i is the i-th text feature.

[0065] Text Feature Fusion Layer: This layer fuses cross-modal text features with associated keyword features to produce the final text representation. This step, performed using a vector concatenation function, aims to integrate information from different text features and provide a unified text feature representation for subsequent data processing.

[0066] Causal Attention Network Layer: This layer uses the attention mechanism to implement a neural network based on front-gate causal intervention. This layer is implemented in two parts. To optimize the selection of image features using front-gate intervention, one part is implemented through within-sample sampling, and the other part is implemented through between-sample sampling.

[0067] The in-sample sampling calculation process is:

[0068]

[0069] Indicates the result obtained by sampling within the sample.

[0070] MatMul is the matrix multiplication function, which will operate on the eigenvectors.

[0071] v is the Value of the current sample.

[0072] Soft max() is a function used in classification problems, which can convert a real vector or matrix into a probability distribution.

[0073] q T It is the transpose of the Query of the current sample.

[0074] k is the key of the current sample.

[0075] The inter-sample sampling calculation process is:

[0076]

[0077] Indicates the result obtained by sampling between samples.

[0078] MatMul is the matrix multiplication function, which will operate on the eigenvectors.

[0079] V is the value of the sample between the current samples.

[0080] Soft max() is a function used in classification problems, which can convert a real vector or matrix into a probability distribution.

[0081] q T It is the transpose of the Query of the current sample.

[0082] K is the key between samples.

[0083] Image feature fusion layer: This layer fuses the features sampled within a sample with the features sampled between samples to obtain the final image feature representation. This step can be performed using a vector concatenation function. The purpose is to integrate the information of different image features and provide a unified image feature representation for subsequent data processing.

[0084] Feature Selection Optimization Layer: This layer addresses the significant semantic gap between different modalities by designing a feature selection optimizer. To achieve this, this layer uses mutual information theory as its foundation. In deep learning, this is calculated using the Kullback-Leibler divergence between two feature representations. To reduce computational costs and avoid extreme computational complexity, contrastive learning is employed: correctly paired image-text representations are used as positive examples, while incorrectly paired image-text representations are used as negative examples. The process is described as follows:

[0085] L MI =log(q(F T ; F V + ))+log(1-q(F T ; F V- ))

[0086] Among them, (F T ; F V + ) is the correct pairing of graphic representation, (F T ; F V - ) is an incorrectly paired graphic representation.

[0087] Cross-entropy loss: The feature selection optimizer uses the standard cross-entropy loss for optimization.

[0088] Multimodal feature fusion layer: This layer fuses the selected text features with the image features to obtain a multimodal feature representation that includes both visual and textual information. This step integrates visual information with textual information through the attention mechanism, integrating information from the image and text data to provide more comprehensive information for further research. The core formula of this process is expressed as follows:

[0089] C=MultiHeadAtt(Z T ,Z V ),

[0090] Among them, Z T is the image feature, Z V is the text feature, C is the multimodal feature representation, and MultiHeadAtt() is the multi-head attention mechanism.

[0091] Entity Prediction Layer: After obtaining a text representation that incorporates visual information, Conditional Random Fields (CRF) is used as the model's decoder. Given a sentence S and its corresponding correct sequence label Y, the correct probability of Y is output. The formula involved in this process can be expressed as:

[0092]

[0093] Among them, X is the input feature sequence, Y is the output feature sequence, Z(x) is the normalization factor, tk is the transfer feature function, sl is the state feature function, λ k and μ l is the corresponding weight parameter.

[0094] The beneficial effects of the present invention are:

[0095] This invention uses an image captioning model to extract captions from text-related images, combining them with the text to provide additional contextual information. A keyword extraction model is used to extract related keyword groups from cross-modal text, providing semantic structure information for the model. A visualization toolkit is used to obtain local visual objects from text-related images, which, together with the global visual objects of the text-related images, provide the model with a complete set of fine-grained visual information. The BERT model is used to encode text data, and the ResNet model is used to extract features from image data, fully utilizing both image and text information in multimodal named entity recognition. A text backdoor causal attention network is implemented to filter text information through backdoor intervention to obtain optimized text features, enhancing the model's ability to understand text entity information. An image frontdoor causal attention network is implemented to introduce mediating variables, implementing frontdoor intervention through within-sample and between-sample sampling to obtain decongested image information, enhancing the model's ability to capture complex image information. A feature optimal selector based on mutual information theory is used to enable the model to focus on relevant portions of image and text information, while discarding irrelevant portions, fully utilizing effective information. This approach helps improve the probability of entity prediction and enhances model robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0096] Figure 1 It is the overall flow chart of the present invention;

[0097] Figure 2 This is the network structure diagram of the visual encoder ResNet in the present invention;

[0098] Figure 3 This is a network structure diagram of the pre-trained language model BERT in the present invention;

[0099] Figure 4 It is the overall structural diagram of the present invention; DETAILED DESCRIPTION

[0100] The following will refer to the attached Figure 1-4 The specific operation steps of the present invention's optimal feature selection multimodal named entity recognition method based on the attention mechanism are described in more detail.

[0101] The overall implementation process of the present invention mainly includes four parts: data processing module, multimodal feature extraction module, causal attention feature selection module, and optimal feature interaction module.

[0102] The flow chart of the present invention is as follows Figure 1 As shown, each step of the flowchart will be described in detail below.

[0103] Step 1:

[0104] The datasets used in this paper are Twitter-2015 and Twitter-2017, two publicly available and authoritative multimodal named entity recognition datasets. These datasets provide text and text-associated images, with the text containing correct entity labels. Good datasets provide accurate training information for the model, ensuring that the model learns effective features and patterns, thereby improving the model's generalization ability and prediction accuracy.

[0105] Step 2: Text Data Processing

[0106] When processing text data, the primary problem is that the text data in these two data sets are short, and some data cannot even form a sentence with complete language semantics. Image subtitles can provide complete image descriptions and provide additional semantic information for text data. Therefore, the present invention uses the [SEP] tag to connect text data and image subtitles as cross-modal input text, giving the text modality more complete contextual information. Then, from the perspective of language semantic relations, we study how to select the optimal features. Keywords play an important role in text. A group of keywords that are related to each other in meaning help to construct the semantic field of the text. Therefore, a keyword extraction model is used to extract keywords from the cross-modal input text, and similar keywords are clustered into groups and integrated into associated keyword groups. The associated keyword group and the cross-modal input text are used together as new text data. It can be expressed as:

[0107] C={c1,c2,c,L,c n}

[0108] T={t1,t2,t,L,t n}

[0109] T′={T+[SEP]+C}

[0110]

[0111] Among them, C represents the image caption of the text-associated image, T represents the original text data, T′ represents the cross-modal text data composed of image caption and original text data, K i Represents the i-th associated keyword group.

[0112] Step 3: Image Data Processing

[0113] When processing image data, according to the division of computer vision, visual information is divided into global visual information and local visual information: global visual information provides global abstract concepts, while local visual information provides specific local semantic units. In the present invention, in the research of this article, both types of visual information are used to enhance the performance of the model. First, the local visual information is obtained using a visualization toolkit, and then the local visual information and the global visual information (text-associated image) are combined into a picture group, which is then uniformly adjusted to 224*224 pixels. This picture group is used as the new image data. It can be expressed as:

[0114] V={v0,v1,L,v m}

[0115] V represents image data.

[0116] Step 4: Multimodal feature extraction

[0117] Text feature extraction:

[0118] The BERT pre-trained model is used to encode the cross-modal input text and the associated keyword phrases separately. The BERT model can provide complete contextual information for the model of the present invention and provide rich text features for entity prediction. For the cross-modal input text, a "[CLS]" tag is added at the beginning of the data and a "[SEP]" tag is added at the end. Considering that the associated keyword phrases are composed of words and the sentences they constitute are relatively short, "[CLS]" and "[SEP]" are not added for them.

[0119] The characteristics of text data can be expressed as:

[0120] H t ={h0,h1,L,h n},

[0121]

[0122] in, represents the global text representation, It is H t The context word representation is, represents the context word representation of K1,

[0123] represents the context word representation of K2,

[0124] The context word representation of K3, n, is the dimension of text representation.

[0125] Image feature extraction:

[0126] Using the ResNet model to extract image features can bring powerful feature extraction capabilities to the model, including the capture of visual information such as texture, shape, and color. The multi-layer stacking structure of the ResNet model helps to capture different levels of abstract features in the image, thereby improving the integrity of the image features. In addition, the features extracted by the ResNet pre-trained model have rich semantic information, which enables the model of the present invention to better utilize image data. The features of image data can be expressed as:

[0127] H v ={h v0 ,h v1 ,L,h vm}

[0128] Among them, because the output dimension of ResNet is 2048, in order to better integrate the visual representation with the text representation, the linear transformation matrix is ​​used here Project the image feature representation into the same dimensional space as the text representation.

[0129] Step 5: Causal Attention Feature Selection

[0130] Text backdoor attention mechanism:

[0131] The text backdoor attention mechanism uses a backdoor causal intervention approach to design an attention mechanism. This step simulates the "do" operator in causal intervention. From the perspective of text language semantics, a backdoor adjustment strategy is designed to approximate the distribution of the confounding factor set. This strategy uses cross-modal text and keyword association sets to approximate the potential confounding factor set. Leveraging multiple data types, the node representation is greatly enriched, enabling it to capture more complex contextual information. The probability of the confounding factor, P(Z = z), is approximately expressed as follows:

[0132]

[0133] Where |H| is the number of samples in H, and |k| is the number of times keyword K appears.

[0134] In this theoretical framework, each keyword-association linguistic representation must be generated as a deconvoluted representation. This ensures that the impact of each keyword association set on model predictions is appropriately considered and integrated. After obtaining the deconvoluted linguistic representation, the results are integrated based on the interrelationships between these representations using an attention mechanism and a graph convolutional network. Ultimately, the optimal text features are obtained.

[0135] Image front-door attention mechanism:

[0136] The image front-gate attention mechanism uses a front-gate causal intervention approach to design an attention mechanism. This approach introduces a front-gate variable to help estimate the direct causal effect of the cause variable on the outcome variable while controlling for potential confounding factors. In practice, this approach employs both in-sample and between-sample sampling, leveraging relationships between different samples to enhance the model's ability to select optimal features. The in-sample and between-sample sampling approaches can be expressed as:

[0137]

[0138] Among them, h(x) and f(x) represent network mapping functions, It is a representation of in-sample sampling, which obtains relevant useful information from the current input sample; It is a representation of inter-sample sampling, which obtains useful information of other samples through cross-sample form to help the model make judgments and ultimately obtain the optimal image information.

[0139] Step 6: Optimal Feature Interaction

[0140] After obtaining the optimal feature representations for text and images, the optimal feature selector is used to shorten the modal semantic distance. The optimal feature optimizer uses a network architecture based on mutual information theory. This network helps identify and select feature representations that are most relevant to the target variable, improving the model's ability to model nonlinear relationships in the data. This mechanism allows the module to dynamically adjust the correlation between image and text feature representations, allowing the model to more accurately capture patterns and relationships in the data, improving prediction accuracy and model generalization.

[0141] In this way, the mutual information between the feature and the target variable is calculated to quantify the contribution of the feature to the target variable information. The larger the mutual information value, the stronger the correlation between the feature and the target variable, and therefore the more important this feature is for predicting the target variable.

[0142] Ultimately, the optimal feature selector, through an attention mechanism, fuses image and text information to produce a multimodal feature representation. This fusion process involves mapping image information to text information, and is not a simple vector addition or concatenation operation. Through the attention mechanism, the fusion not only preserves important image information but also maintains the integrity of the text, thus enhancing the robustness of the model.

[0143] Step 7: Model Prediction

[0144] Multimodal named entity recognition uses conditional random fields to predict entity information. The model's loss function uses both the negative log-likelihood loss and the optimal feature optimizer's loss function, which are combined by adding them together. The model continuously adjusts the embedding vectors to minimize the loss function, thereby improving the accuracy of multimodal named entity recognition.

[0145] Further, the data processing is described as follows:

[0146] Text data processing:

[0147] Specifically, we first use the image caption model to generate the image description, denoted as C = {c1,c2,c,L,c n The image captioning model is a multimodal artificial intelligence technology that combines computer vision and natural language processing. It analyzes the image content, understands the objects, scenes, and actions in the image, and then generates accurate and coherent natural language descriptions. The original plain text information is recorded as T = {t1, t2, t, L, t n}, connect the plain text information T and the image description C, denoted as T′={T+[SEP]+C}. Use, for example, a graph-based causal inference (GCI) framework to extract keywords from T′, and cluster similar keywords into groups, integrating them into a set of related keywords: K i represents the i-th associated keyword set, t i K i The i-th keyword in l i represents the length of the i-th keyword set. Given the length of tweets, the keyword set is limited to three groups. Keyword extraction models are a natural language processing technique designed to automatically identify and extract words or phrases from text that best represent the text's theme and content. Keyword extraction models analyze text data using various algorithms and techniques to determine which words or phrases play a key role in the text.

[0148] In order to accurately capture the contextual semantic features of the cross-modal input text T′, this paper uses BERT as a text encoder. For the text data to be input, a “[CLS]” tag is added at the beginning of the data and a “[SEP]” tag is added at the end. It is denoted as S = {s0, s1, s2, L, s m}, where s0 is "[CLS]", s m "[SEP]". Since the sentences in the keyword set are short, "[CLS]" and "[SEP]" are not added. S, K1, K2, and K3 are input into BERT to obtain text feature representation.

[0149] Image data processing:

[0150] In the field of computer vision, visual information is divided into global visual information and local visual information: global visual information provides global abstract concepts, while local visual information provides specific local semantic units. In this paper, both types of visual information are used to enhance the performance of the model. First, the local visual information is obtained using a visualization toolkit, and these visual representations are denoted as V = {v0,v1,L,v m Given ResNet's significant achievements in computer vision, this paper selects it as the encoder for visual information. The specific operation is as follows: First, all images are uniformly resized to 224*224 pixels, and then input into the ResNet model to obtain image feature representation.

[0151] Multimodal feature extraction:

[0152] Using BERT to process text data

[0153] BERT possesses powerful language understanding capabilities. Pre-trained on large amounts of text, it learns rich language patterns and contextual relationships, enabling it to capture deep semantic information about words, phrases, and even entire sentences. BERT can handle complex language structures, understand the polysemy of words, and factor in the influence of context, resulting in more accurate word embeddings. Furthermore, BERT's bidirectional encoding mechanism allows the model to consider both contextual information and the surrounding context, which is particularly important for understanding long-range dependencies in language.

[0154] To ensure that the model can understand and correctly process text data, the raw text data needs to be converted into a format that the model can process. This typically involves tokenization, vocabulary processing, adding special tags, and creating an attention mask. The BERT model uses the WordPiece tokenization method to break text into tokens in the vocabulary. It can also handle words that do not appear in the vocabulary by breaking them down into subword units. The BERT vocabulary contains between 30,000 and 50,000 common words and subword units. After tokenization, a "[CLS]" token is added to the beginning of the sequence and a "[SEP]" token is added to the end. If there are two sentences, a "[SEP]" token is added to each sentence. The output of the "[CLS]" token represents the characteristics of the entire sentence and is often used in classification tasks, while the "[SEP]" token is used to separate sentences. Next, an attention mask is created to ensure that the model only considers the actual input when calculating self-attention, ignoring the padding, thereby improving model efficiency. Finally, the preprocessed text is input into the BERT model to obtain the embedding representation of each token, including word embedding, position embedding, and paragraph embedding. Then, the forward propagation of the multi-layer Transformer encoder is performed, and the results of each layer are output to finally obtain high-quality text feature representation.

[0155] Use ResNet to process image data

[0156] ResNet features an innovative residual architecture that, by introducing skip connections, addresses the vanishing gradient problem in deep network training, enabling the construction of deeper networks without sacrificing performance. ResNet's deep structure helps extract complex image features, enhancing the model's representational capabilities. Furthermore, pre-trained ResNet models can be transferred to various visual tasks, providing powerful feature extraction capabilities, accelerating convergence, and improving accuracy.

[0157] For image data, due to its complex data form, the ResNet model first needs to normalize the image, that is, subtract the average pixel value of the entire dataset from the pixel value of the image, and adjust the image to 224*224 pixels to adapt to the input format of the ResNet model.

[0158] The preprocessed image is then fed into the ResNet model. The core of the ResNet model is the residual block, each of which consists of two convolutional layers, which may also include batch normalization and ReLU activation functions. The outputs of these layers are added to the input to form a residual connection. As image data flows through multiple residual blocks, the model is able to capture features from low-level to high-level. At the end of the network, the feature maps are aggregated and processed through one or more fully connected layers, ultimately outputting the predicted probability for each class. This results in a high-quality image feature representation.

[0159] Causal Attention Networks:

[0160] Text Backdoor Attention Network:

[0161] The intervention distribution of backdoor intervention is P(Y|do(X)):

[0162]

[0163] In this formula:

[0164] P m Indicates a backdoor intervention operation.

[0165] Z represents confounding factors.

[0166] Then, the normalized weighted geometric mean (NWGM) is applied to approximate the effect of the do operator:

[0167]

[0168] After obtaining the deconvoluted language representation F = {F1, F2, F3, F4}, the results are integrated according to the mutual correlation between these representations through the attention mechanism and graph convolution network. First, a causal graph is constructed using the deconvoluted language representation F i As graph nodes, fully connected edges are constructed based on the causal relationships learned by tools such as GCI. Through the graph convolutional network, local structural information in the feature graph is captured and the spatial correlation of the feature graph is enhanced. This process can be expressed as:

[0169] y i =GCN(F i ,G_index),i=1,2,3,4

[0170] In this formula:

[0171] GCN() is a graph convolution operation.

[0172] G_index defines the adjacency relationship of the graph.

[0173] The four feature maps enhanced by GCN are spliced ​​together to form a feature tensor ft. In order to further highlight the key information in the feature map and suppress those unimportant features, the feature tensor ft is input into four fully connected layers. The role of these four fully connected layers is to filter and enhance the features and output the optimized feature map: attn i :

[0174] ft=concat(y1,y2,y3,y4)

[0175] attn i =σ(FC2(RELU(FC1(ft)))),i=1,2,3,4

[0176] In this formula:

[0177] FC1 and FC2 are linear layers for dimensionality reduction and dimensionality increase, respectively.

[0178] σ is the Sigmoid activation function, which is used to compress the output into the range [0, 1].

[0179] Multiply each graph processed by the attention mechanism with the corresponding original feature map to obtain the enhanced feature map. These enhanced feature maps are spliced ​​together to obtain the fused feature F T :

[0180] y′ i =y i e ReLU(attn i ),i=1,2,3,4

[0181] FT =Concat(y1,y2,y3,y4)

[0182] In this formula:

[0183] ⊙ represents element-wise multiplication.

[0184] ReLU is the activation function.

[0185] Image front-door attention network:

[0186] The access formula for front-door intervention is:

[0187]

[0188] In this formula:

[0189] P m Indicates the front door intervention operation,

[0190] M represents the mediating factor.

[0191] To implement front-door causal intervention, it is necessary not only to sample the X variable but also to obtain the M variable and pass the result to the computational network. To reduce the cost of the transfer process, the normalized weighted geometric mean (NWGM) is applied to approximate the effect of the do operator, resulting in the true causal intervention formula, which is expressed as:

[0192]

[0193] In order to implement front-door causal intervention in the deep learning framework, P(Y|do(X)) is parameterized as a network g(·) and a softmax layer is added after the network g(·). It can be expressed as:

[0194]

[0195] In this formula:

[0196] h(x) and f(x) represent network mapping functions. Essentially, it is obtained by sampling within the sample, which means extracting relevant information from the current input sample X. Essentially, it is obtained through cross-sample sampling, involving the interaction between intra-sample data and inter-sample data. Both processes can be implemented through the attention mechanism.

[0197] 1. In-sample sampling module

[0198] To simplify the description, auxiliary operations such as linear transformation are omitted. The core part of this module can be expressed as:

[0199]

[0200] Where q, k, and v all come from the current sample, and MatMul represents matrix multiplication.

[0201] 2. Inter-sample sampling module

[0202] Similar to in-sample sampling, the query vector q comes from the current sample. The difference is that the key vector K and value vector V come from inter-modal samples of the same size as q. Because the model cannot focus on all samples in the training set, this module uses random sampling to select inter-modal samples. The core of this process can be expressed as follows:

[0203]

[0204] Finally, the two vectors are combined to get the visual representation: F V .

[0205] Optimal feature selector:

[0206] For the optimal feature selector, the goal of this module is to reduce the difference between image representation and text representation. Mutual information is a powerful tool to measure the strength of association between variables. The core of this module is to use the measurement ability of mutual information to maximize relevant information and minimize irrelevant information, thereby promoting consistency and correlation between different modalities and reducing modality gaps. Mutual information can be calculated by the Kullback-Leibler divergence between two distributions. Visual information F V and text message F T The mutual information can be expressed as:

[0207]

[0208] In this formula:

[0209] I(·;·) represents the MI between two random variables.

[0210] p(·) and p(·,·) represent the boundary probability and joint probability of two samples, respectively.

[0211] p(·) and p(·|·) represent the prior and posterior distributions of the two variables.

[0212] However, directly calculating mutual information often requires integrating over the entire latent space, which is a very difficult task. To address this problem, we use the variational lower bound method to approximate the true posterior distribution by introducing an easy-to-calculate approximate posterior distribution q(,). The implementation equation is as follows:

[0213]

[0214] In order to implement it in the model and avoid extreme situations, the present invention uses contrastive learning to measure MI, and the objective function of this module can be obtained:

[0215] L MI =log(q(F T ; F V + ))+log(1-q(F T ; F V - ))

[0216] In this formula:

[0217] (F T ; F V + ) means the correctly matched graphic and text representations.

[0218] (F T ; F V - ) is an incorrectly paired graphic and text representation.

[0219] This module can be optimized using the standard cross-entropy loss.

[0220] Loss Function

[0221] In our invention, the loss function of our model is:

[0222] L=L ner +L MI

[0223] In this formula:

[0224] L ner is the negative log-likelihood function.

[0225] L MI is the loss function of the optimal feature selector, which is implemented by the standard cross entropy function.

[0226] It should be noted that the above description is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. Those skilled in the art will recognize that equivalent substitutions or obvious variations can be made without departing from the scope of the present invention, and that such variations, while maintaining the same performance or application, should be considered to fall within the scope of protection of the present invention.

Claims

1. A multimodal named entity recognition method based on optimal feature selection of attention mechanism, characterized by: Here are the steps: Step 1: Convert text-associated images into image captions through an image caption generation model. Image captions can provide more semantic information for text data. Text data and image captions are connected as cross-modal text data for further research. The image caption generation process is described as follows: C=Caption(Image), Among them, C is the image caption, Image is the text associated image, Step 2: Extract related keyword groups from the cross-modal text using the keyword extraction model. Considering the length of the cross-modal text, the number of related keyword groups is limited to three. The related keyword groups and the cross-modal text are taken together as text data. The keyword extraction process can be expressed as follows: K = Keyword_extraction(T_D) Among them, K is the keyword data set, T_D is the original text data, Step 3: Use the visualization toolkit to obtain the k most prominent objects in the text-associated image as local visual information, and the text-associated image as global visual information. The two are combined as the image data of this method. The local image extraction process is expressed as: V object =Visual_tool(V_D), Among them, V object is the local visual object, V_D is the original image data, Step 4: Use the pre-trained language model and pre-trained visual model to encode the text data and image data respectively, and obtain the corresponding text feature representation and image feature representation. The process of obtaining the text feature representation and image feature representation is expressed as follows: Among them, H T is the text feature representation, H V is the image feature representation, F_T is all text data, F_V is all image data, Step 5: Design a backdoor causal attention mechanism to block the backdoor paths in the text data, so that the model focuses on the optimal text features. The selection process of the optimal text features is expressed as: H T _B=FD_cause(H T ), Among them, H T _B is the optimal text feature, FD_cause is the front-door causal attention mechanism, H T is the text feature representation, Step 6: Design a front-door causal attention mechanism. By introducing front-door variables as intermediaries, we construct an effective front-door path to help the model estimate the direct causal relationship between image data and labels, allowing the model to focus on the optimal image features. The selection process of the optimal image features is expressed as: H V _B=BD_cause(H V ), Among them, H V _B is the optimal image feature, BD_cause() is the backdoor causal attention mechanism, H V is the image feature representation, Step 7: For the optimal text features and image features obtained in steps 5 and 6, a feature alignment optimizer based on mutual information theory is designed to narrow the semantic distance between image and text information, reduce the modality gap, and improve the accuracy of entity recognition. The feature alignment optimizer is expressed as: (H T _BN,H V _BN)=L MI (H T _B,H V _B), Among them, H T _BN is the aligned text feature, H V _BN is the image feature after alignment, L MI () is the feature alignment optimizer, H T _B is the optimal text feature, H V _B is the optimal image feature, Step 8: Use conditional random fields to process the multimodal feature representation and predict entity information. The entity prediction process is expressed as: EI=CRF(M_feature), Among them, EI is the predicted entity information, M_feature is the multimodal feature representation, Step 9: Feed the predicted entity information back into the attention-based neural network to further optimize the backdoor causal attention network, frontdoor causal attention network, and feature alignment optimizer to improve the accuracy of the prediction. Step 10: Repeat steps 5 to 9 until the predetermined number of iterations is reached or the inference result meets the pre-set performance index.

2. The optimal feature selection multimodal named entity recognition method based on the attention mechanism according to claim 1 is characterized in that: For step 5, the backdoor causal attention mechanism is designed for text features, using cross-modal text and related keyword sets to approximate the confounding factor Z t , eliminate the existing backdoor path {X←Z t →Y}, we get the decongested data and combine it with the graph convolution operation to get the text features of the enhanced feature representation. The intervention distribution of the backdoor intervention is expressed as: Among them, Z is the confounding factor in the text feature, X represents the text feature, Y represents the target result, and do() represents the intervention operation.

3. The optimal feature selection multimodal named entity recognition method based on the attention mechanism according to claim 1 is characterized in that: For step 6, the front-door causal attention mechanism is designed for image features. By introducing mediating variables, a front-door path {X→M→Y} is constructed. Through intra-sample and inter-sample sampling, the confounding factors in the image modality are removed to obtain the optimal image features. The intervention distribution of the front-door intervention is expressed as: Among them, M is the mediating variable, X represents the image feature, Y represents the target result, and do() represents the intervention operation.

4. The optimal feature selection multimodal named entity recognition method based on the attention mechanism according to claim 1 is characterized in that: For step 7, the feature alignment optimizer based on mutual information theory is implemented in the form of contrastive learning. It uses the measurement ability of mutual information to maximize relevant information and minimize irrelevant information, promote consistency and correlation between different modalities, and improve the robustness of the model for multimodal named entity recognition tasks. The objective function of the feature alignment optimizer is expressed as: L MI =log(q(F T ;F V + ))+log(1-q(F T ;F V - )), Among them, (F T ; F V + ) is the correct graphic representation, (F T ; F V - ) is an incorrectly paired graphic representation.

Images