A multi-modal joint entity relation extraction method for improving extraction accuracy

Abstract

The application discloses a multimodal joint entity relation extraction method for improving extraction accuracy, comprising the following steps: S1, obtaining paired multimodal data samples containing text sequences and corresponding images; S2, respectively encoding the text and the image into vectors, and calculating a text-image similarity score; S3, inputting and transforming the encoded features and performing block processing; S4, constructing a double-channel feature fusion module, including a time sequence enhancement module and an improved similarity gate cross attention module; S5, constructing an adaptive threshold gate mechanism, and performing gate fusion on the cross-modal cross attention output to obtain multimodal fusion features; S6, inputting the multimodal fusion features into a joint prediction module, and outputting entity relation five tuples. Through the learnable adaptive threshold gate mechanism and the similarity modulation adaptive layer normalization mechanism, the application can suppress low correlation visual noise in a weak image-text correlation scene, and improve the accuracy and robustness of the multimodal joint entity relation extraction.

Description

Technical Field

[0001] This invention relates to the field of natural language processing and multimodal information extraction technology, specifically to a multimodal joint entity relation extraction method for improving extraction accuracy. Background Technology

[0002] The Joint Multimodal Entity-Relation Extraction (JMERE) task requires extracting entity-relation quintuples from text-image pairs, placing higher demands on the model's multimodal feature fusion and selection capabilities. With the development of large pre-trained language models, existing research mainly focuses on improving feature alignment between text and visual modalities. However, there are still significant shortcomings in capturing temporal information within text sequences. Furthermore, these methods have certain limitations in integrating image and text features, struggling to distinguish irrelevant images and being susceptible to interference from image information unrelated to the text.

[0003] Existing methods mostly employ static similarity exploitation and standard layer normalization techniques. Although these methods have improved the performance of multimodal entity relation extraction to some extent, they still have certain limitations. First, current similarity exploitation methods mostly use static linear multiplication strategies, which make it difficult to dynamically adjust the amount of visual information introduced based on the degree of text-image correlation, especially when the text and image are weakly correlated, easily introducing noise. Second, multimodal fusion only uses standard layer normalization techniques, which cannot dynamically adjust the feature space distribution based on similarity scores, limiting the model's ability to model the strength of multimodal associations. Summary of the Invention

[0004] The purpose of this invention is to provide a multimodal joint entity relation extraction method to improve extraction accuracy. It combines multimodal information such as text and images, and improves the entity relation extraction accuracy through adaptive threshold gating and similarity modulation layer normalization mechanism. This solves the noise interference problem caused by static similarity utilization in existing multimodal entity relation extraction methods, as well as the problem that traditional layer normalization cannot dynamically adapt to the multimodal association strength.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a multimodal joint entity relation extraction method for improving extraction accuracy, comprising the following steps:

[0006] S1: Obtain paired multimodal data samples containing text sequences and image content;

[0007] S2: Feature encoding is performed on text data and image data respectively to obtain text context semantic feature representation and image region visual feature representation, and the image-text similarity score SC between the text sequence and the corresponding image is calculated. This includes using a pre-trained language model to perform context semantic modeling on the text sequence to generate text context semantic feature representation; using the visual encoding module in the vision-language pre-trained model to extract features from the corresponding image to generate image visual feature representation; and calculating the image-text similarity score based on the text context semantic feature representation and the image visual feature representation. The image-text similarity score is used for subsequent cross-modal attention fusion, adaptive threshold gating, and similarity modulation adaptive layer normalization processing.

[0008] S3: Perform linear mapping and feature segmentation on the text context semantic feature representation and image visual feature representation to obtain text feature blocks and image feature blocks for the named entity recognition channel and relation extraction channel; specifically, this includes: mapping the text context semantic feature representation to text fusion features through a trainable linear transformation with a non-linear activation function, and dividing the text fusion features into multiple text feature blocks; mapping the image visual feature representation to image fusion features through a trainable linear transformation with a non-linear activation function, and dividing the image fusion features into multiple image feature blocks; inputting the text feature blocks and image feature blocks into the named entity recognition feature fusion channel and the relation extraction feature fusion channel, respectively;

[0009] S4: Construct a dual-channel feature fusion module, including: naming an entity recognition feature fusion channel for generating entity prediction features and a relation extraction feature fusion channel for generating relation prediction features; in the dual-channel feature fusion module, the text sequence modeling network is initialized using global image features, and the information interaction between text features and image features is realized through a cross-modal cross-attention mechanism;

[0010] The initialization of the text sequence modeling network using global image features includes:

[0011] Image feature blocks are pooled to obtain global image features; these global image features are used as the initial hidden state and initial cell state of the text sequence modeling network; the text feature blocks are then subjected to temporal enhancement modeling through the text sequence modeling network to obtain text temporal enhancement features containing word order information and long-distance dependency information.

[0012] The cross-modal attention mechanism includes: using text temporal enhancement features as query vectors and text feature blocks or image feature blocks as key-value vectors to construct a multi-head cross-attention structure; obtaining cross-modal attention output through the multi-head cross-attention structure; and obtaining preliminary fusion features through residual connections and normalization.

[0013] S5: An adaptive threshold gating mechanism is constructed based on the image-text similarity score (SC) to perform gating fusion on the cross-modal cross-attention output. An adaptive layer normalization mechanism based on similarity modulation is used to adjust the feature distribution of the gated fused features to obtain multimodal fused features. The similarity threshold gating includes: generating a gated similarity score based on the image-text similarity score (SC) using learnable gating parameters, and weighting the cross-modal attention output using the gated similarity score to reduce or suppress the influence of image features on text features when the image-text similarity is lower than the dynamic threshold determined by the learnable gating parameters. The similarity modulation adaptive layer normalization includes: performing layer normalization on the gated fused features to obtain standard normalized features, mapping the image-text similarity score (SC) to scaling and offset parameters, and dynamically modulating the standard normalized features using the scaling and offset parameters.

[0014] S6: Input the multimodal fusion features into the joint prediction module to predict entity boundaries, entity types, and inter-entity relationship types, and output entity relationship quintuples based on the prediction results; wherein, the joint prediction module includes an entity prediction module and a relationship prediction module; the entity prediction module predicts entity boundaries and entity types based on the multimodal fusion features, and the relationship prediction module predicts the relationship types between entities based on the multimodal fusion features, and assembles and outputs entity relationship quintuples based on the entity prediction results and relationship prediction results.

[0015] Furthermore, S1 also includes standardization and word segmentation of the acquired text sequence, as well as size and pixel normalization of the corresponding image, denoted as... ,in , This represents the input text sequence. This represents the corresponding image data. Indicates the length of the text sequence. Represents the first in the text sequence One word.

[0016] Furthermore, the specific calculation method for the image-text similarity score SC in S2 is as follows:

[0017] S201: Encoding the text sequence using a pre-trained BERT model: , This represents the text sequence feature matrix output by the pre-trained language model. This represents the input text sequence. Indicates the length of the text sequence. This represents the dimension of the initial hidden layer of features output by the pre-trained model;

[0018] S202: The image is encoded using the MED module of a pre-trained BLIP model. , This represents the image feature matrix output by the visual model. This represents the corresponding image data. Indicates the length of the image feature sequence. This represents the dimension of the initial hidden layer of features output by the pre-trained model;

[0019] S203: Use BLIP to calculate text-image cosine similarity as a basis for subsequent gating: , This represents the calculated image-text similarity score. This represents the function for calculating cosine similarity. This represents the image and text feature matrix output by the visual model. This represents the multimodal encoder / decoder module in a pre-trained visual model; similarity value. It will serve as the control signal for the gated cross-attention module, used to control the interference of irrelevant image information on entity and relation extraction.

[0020] Furthermore, the linear mapping and feature block processing methods in S3 are as follows:

[0021] S301: Text Features Mapping and dividing the data using a trainable linear transformation with Tanh activation: , ;

[0022] S302: Image features Mapping and dividing the data using a trainable linear transformation with Tanh activation:

[0023] , ,

[0024] in, This represents the comprehensive features of the mapped text. This represents the mapped image text features. , , and, These are trainable parameters. This represents the hyperbolic tangent activation function. Indicates the hidden feature dimension. This indicates a block operation. Indicates the dimension of the partitioning operation. and This indicates a special certificate block.

[0025] Furthermore, the initialization of the text sequence modeling network in S4 using global image features includes:

[0026] S401: LSTM is introduced in the first layer of the NER and RE channels to perform pooling processing on the image feature blocks and obtain the global features of the image;

[0027] S402: Using global image features as the initial hidden states and initial cell states of the text sequence modeling network:

[0028] , ,

[0029] in, and This represents the initial cell state and the initial hidden state. It is the output time feature sequence of the LSTM layer. This represents the max pooling operation;

[0030] S403: Temporal enhancement modeling of text feature blocks is performed through a text sequence modeling network to obtain text temporal enhancement features containing word order information and long-distance dependency information.

[0031] Furthermore, the cross-modal cross-attention mechanism in S4 is achieved by fusing the first layer residual + LayerNorm structure through multi-head cross-attention and a feedforward network, including:

[0032] S404: Using text temporal enhancement features as query vectors and text feature blocks or image feature blocks as key vectors and value vectors, a multi-head cross-attention structure is constructed.

[0033] S405: Cross-modal attention output is obtained through a multi-head cross-attention structure, and preliminary fusion features are obtained through residual connections and normalization.

[0034] Furthermore, the adaptive threshold gating mechanism in S5 includes: S501: Setting learnable gating parameters and And calculate the gated similarity score based on the image-text similarity score SC. ,in: , S502: Modify the linear unit activation function; S502: Gated similarity scores The cross-modal attention output is multiplied element-wise to control the degree to which visual information or cross-modal interaction information enters the text feature channel, and gated fusion features are obtained through residual connections and layer normalization:

[0035] ,

[0036] in, This represents the intermediate features after adaptive gating attention fusion. This represents the normalization function of the first standard layer. As key-value feature input, This represents element-wise multiplication. This represents the multi-head cross-attention calculation function. This represents the query vector, key vector, and value vector in a multi-head attention mechanism.

[0037] S503: When When the gated similarity score is less than or equal to zero, The value is zero or approximately zero, thereby reducing or blocking low-correlation image information from entering the text feature channel;

[0038] S504: When When the similarity score is greater than zero, the gated similarity score is... The similarity score SC increases with the increase of the image-text similarity score, thereby enhancing the supplementary role of highly relevant image information on text features.

[0039] Furthermore, the similarity modulation adaptive layer normalization mechanism in S5 is as follows:

[0040] S505: The gated and fused features are input into the feedforward neural network, and after residual connections and layer normalization, standard normalized features are obtained. :

[0041] ,

[0042] in, This represents the standard normalized features without similarity modulation. This represents the normalization function of the second standard layer. This represents a feedforward neural network;

[0043] S506: Input the image-text similarity score into the linear mapping layer to generate scaling parameters for feature modulation. Translation parameters :

[0044]

[0045] in, , These represent the scaling and translation parameters generated by dynamic projection of similarity scores, used for feature distribution modulation. , This represents the trainable weights and biases in the similarity linear mapping layer;

[0046] S507: Based on scaling parameters Translation parameters Standard normalization features Dynamic modulation is performed to obtain multimodal fusion features. The modulation method is as follows: ;

[0047] in, Indicates standard normalized features, This represents the modulated multimodal fusion features. This indicates element-wise multiplication.

[0048] Furthermore, the joint prediction module of S6 includes an entity prediction module and a relationship prediction module, and the specific prediction methods are as follows;

[0049] S601 MNER unit: for each entity type Construct a table, calculate the span representation, and predict the probability:

[0050] ,

[0051] ,

[0052]

[0053] in, , and For trainable parameters, This represents the multimodal features output by S5 specifically for the entity recognition channel. Represents the activation function of the exponential linear unit. This indicates a merge operation. Represents the sigmoid activation function. Represents a collection of entity types Size, Indicates prediction label and As the first The probability of the start and end boundaries of an entity class;

[0054] S602 MRE Unit: Constructs a relation table, using only the initial... :

[0055]

[0056]

[0057]

[0058] in, Multimodal features output by S5 specifically for relation extraction channels. This represents the local features corresponding to the starting labels of candidate head and tail entities. Represents global features. The trainable weight matrix and biases of the relation representation layer are represented. Represents entities ( , Relationship characteristics are represented by ) This represents the weight matrix and bias of the relation classification output layer. Indicates the prediction of candidate head and tail entities and There is a specific relationship between them. The probability of.

[0059] Furthermore, the training and inference methods for the joint prediction module are as follows:

[0060] S603: Calculate the training loss as the sum of the binary cross-entropy of the MNER and MRE units:

[0061]

[0062]

[0063]

[0064] in, and Represents the actual labels for entity tables and relation tables. and These are predicted values; ultimately, our training objective is to minimize the loss. . and Indicates the size of the entity table and the relation table;

[0065] S604: Threshold during inference Filter valid quintuples and assemble entity-relation quintuples by combining MNER and MRE results.

[0066] Compared with the prior art, the beneficial effects of the present invention are:

[0067] 1. The multimodal joint entity relation extraction method of the present invention for improving extraction accuracy utilizes learnable adaptive threshold gating. With the Sim-AdaLN mechanism, dynamic and intelligent adjustment of similarity scores is achieved, effectively suppressing interference from low-correlation images while enhancing the adaptive capability of feature distribution.

[0068] 2. The multimodal joint entity-relation extraction method of the present invention, which improves the extraction accuracy, can achieve better experimental results than the baseline model in JMERE, MNER, and MRE tasks, thereby improving the accuracy of joint entity-relation extraction.

[0069] 3. The multimodal joint entity relation extraction method of the present invention for improving extraction accuracy can enhance the robustness of the model under different batches and sequence lengths, and is suitable for real multimodal social media scenarios. Attached Figure Description

[0070] Figure 1 This is a schematic diagram of the overall process of an embodiment of the present invention;

[0071] Figure 2 This is a schematic diagram of the model structure and key module construction in the experimental case of this invention;

[0072] Figure 3 This is a schematic diagram of the dual-channel feature fusion module structure in the experimental case of this invention. Detailed Implementation

[0073] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0074] Please see Figure 1 This invention provides a multimodal joint entity relationship extraction method based on similarity threshold gating and similarity modulation adaptive layer normalization, which is used to improve extraction accuracy. The overall process includes: multimodal sample acquisition, text and image feature encoding, image-text similarity calculation, linear mapping and feature segmentation, dual-channel feature fusion, adaptive threshold gating fusion, similarity modulation adaptive layer normalization, and joint entity relationship prediction.

[0075] See Figure 1-2 The dual-channel feature fusion module in this invention includes a named entity recognition feature fusion channel and a relation extraction feature fusion channel. During the fusion process, a temporal enhancement module, an adaptive threshold gating mechanism, and a similarity modulation adaptive layer normalization mechanism are introduced. The specific implementation steps are as follows:

[0076] Step 1: Acquire and preprocess multimodal sample data. Let the input sample be... ,in , This represents the input text sequence. This represents the corresponding image data. Indicates the length of the text sequence. Represents the first in the text sequence Each word is segmented and standardized in the text, and the image is normalized in size and converted in channel format to provide a unified input format for subsequent model encoding and modeling.

[0077] Step 2: Perform feature encoding of text and image modalities and calculate image-text similarity. The specific steps are as follows:

[0078] S201: Uses the pre-trained language model BERT as the text encoder to process the input sentence. Mapped to contextual semantic feature representation:

[0079] ,

[0080] in, Represents the semantic feature matrix of the text context. This represents the initial feature dimension of the pre-trained model output.

[0081] S202: The multimodal encoder-decoder module MED of the pre-trained visual language model BLIP is used to extract features from image I, resulting in a visual feature representation of the image:

[0082]

[0083] .

[0084] S203: Calculate the image-text matching similarity score. To assess the semantic relevance of image and text information, calculate their cosine similarity SC:

[0085] ,

[0086] in, This represents the image-text similarity score. This represents the cosine similarity calculation function. This similarity value will be output to the global similarity control bus, serving as a key adjustment variable for subsequent adaptive gating modules.

[0087] Step 3: Perform linear mapping and feature segmentation on the encoded features. The specific steps are as follows:

[0088] S301: Incorporating semantic features of text context Through with The trainable linear transformation of the activation function is mapped to text fusion features and then processed in blocks:

[0089] ,

[0090] ,

[0091] S302: Image visual features Through with The trainable linear transformation of the activation function is mapped to image fusion features and then processed in blocks:

[0092] ,

[0093] ,

[0094] in, This represents the fused text features after mapping. This represents the fused features of the mapped image. , , and, , , Indicates the dimension of the partitioning operation.

[0095] Step 4: Construct a dual-channel feature fusion module, referring to... Figure 2 The dual-channel feature fusion module includes a Named Entity Recognition (NER) feature fusion channel and a Relation Extraction (RE) feature fusion channel, which are used to generate entity prediction features and relation prediction features, respectively. The specific methods are as follows:

[0096] S401: A text sequence modeling network is introduced in the first layer of the NER and RE channels, and the initial cell states and initial hidden states of the text sequence modeling network are initialized using global image features:

[0097] ,

[0098] ,

[0099] in and This represents the initial cell state and the initial hidden state. It is the output time feature sequence of the LSTM layer. This indicates a max pooling operation.

[0100] S402: A cross-modal attention mechanism is constructed based on text temporal enhancement features. The text temporal enhancement features are used as the query vector, and text feature blocks or image feature blocks are used as the key and value vectors to obtain cross-modal attention output. Preliminary feature fusion is achieved through residual connections and layer normalization. Specifically, the NER channel is used to generate entity recognition-related features, and the RE channel is used to generate relation extraction-related features. The RE channel can replace the input feature block with... and To obtain a feature representation that is more suitable for relationship prediction.

[0101] Step S5: Perform adaptive threshold gating fusion and similarity modulation adaptive layer normalization. The specific operation steps are as follows:

[0102] S501: Construct an adaptive threshold gating mechanism. Introduce learnable parameters. and Construct a threshold-based gating function based on the image-text similarity score SC:

[0103] ,

[0104] in, The similarity score after gating is represented by ReLU, which represents the modified linear unit activation function. When the image-text similarity is lower than a dynamic threshold determined by learnable parameters, the similarity score after gating is reduced or truncated, thereby reducing the interference of low-relevance image information on the text feature channels.

[0105] S502: The gated similarity score is multiplied element-wise with the cross-modal cross-attention output, and the gated fusion feature is obtained through residual connections and layer normalization.

[0106] ,

[0107] in, This represents the intermediate features after adaptive gating attention fusion. This represents the normalization function of the first standard layer. As key-value feature input, This represents element-wise multiplication. This represents the multi-head cross-attention calculation function. This represents the query vector, key vector, and value vector in a multi-head attention mechanism.

[0108] S503: The gated fused features are input into the feedforward neural network, and residual connections and a second-layer normalization process are performed to obtain the standard normalized features.

[0109] ,

[0110] in, This represents the standard normalized features without similarity modulation. This represents the normalization function of the second standard layer. This refers to a feed-forward network.

[0111] S504: Construct a similarity modulation adaptive layer normalization mechanism. The image-text similarity score SC is input into a linear mapping layer to generate scaling and translation parameters for feature modulation. :

[0112] ,

[0113] in, , These represent the scaling and translation parameters generated by dynamic projection of similarity scores, used for feature distribution modulation. , This represents the trainable weights and biases in the similarity linear mapping layer.

[0114] S505: Dynamically modulate the standard normalized features according to the scaling and translation parameters to obtain the final multimodal fusion features:

[0115] ,

[0116] in, This represents the output features after normalization by the similarity modulation adaptive layer. Through this mechanism, the image-text similarity score is not only used to control the fusion strength of visual information, but also to dynamically adjust the spatial distribution of the fused features.

[0117] After the above processing, the features used for entity prediction are obtained respectively. and features used for relationship prediction .

[0118] Step 6: Perform joint entity and relationship prediction based on the table filling mechanism, and then perform training and inference. The specific operation steps are as follows:

[0119] S601: Entity prediction. Based on features of NER channel output. For each entity type, an entity prediction table is constructed, and the probability of any two tokens in the text sequence serving as the entity start boundary and end boundary is calculated, thereby obtaining the entity boundary and entity type prediction results.

[0120] S602: Relationship prediction. Based on features of the RE channel output. A relationship prediction table is constructed to predict the relationship types between candidate entities, thereby obtaining the relationship prediction results.

[0121] S6-3: During the training phase, the sum of entity prediction loss and relation prediction loss is used as the joint training objective.

[0122] ,

[0123] in, Indicates the entity's predicted loss. Both the relationship prediction loss and the loss function can be calculated using the binary cross-entropy loss function. The training objective is to minimize the joint loss function. .

[0124] S604: During the inference phase, the entity prediction results and relationship prediction results are filtered according to a preset threshold, retaining valid entities and valid relationships, and combining entity boundaries, entity types and relationship types between entities to finally output entity-relationship quintuples.

[0125] To further illustrate the implementation process of the above method, the following specific experimental examples are provided:

[0126] I. Dataset and Experiment Setup:

[0127] In this embodiment, two publicly available multimodal information extraction datasets, JMERE and Twitter-2015, are used to validate the proposed multimodal joint entity relation extraction model. The JMERE dataset is used to evaluate the model's ability to simultaneously perform Named Entity Recognition (NER) and Relation Extraction (RE) in image-text scenarios; the Twitter-2015 dataset is used to evaluate the model's multimodal named entity recognition ability in short text-text image-text scenarios on social media.

[0128] The aforementioned datasets all consist of text and their corresponding images, exhibiting characteristics such as short texts, incomplete semantic expression, inconsistent correlation between images and text, and significant visual noise. These datasets effectively simulate real-world social media or open-domain image-text understanding tasks. This invention primarily targets multimodal information extraction scenarios where entity locations are unknown and automatic extraction of entity boundaries, entity types, and inter-entity relationships from image-text input is required.

[0129] This embodiment adopts (Accuracy, #P) Recall (#R) and F1-score are the main evaluation metrics. Precision represents the proportion of correct predictions, recall represents the proportion of true results correctly identified by the model, and the F1 score is used to comprehensively measure both precision and recall. The calculation method is as follows:

[0130] ,

[0131] ,

[0132] ,

[0133] in, This indicates the number of entities or relationships that were correctly identified. This indicates the number of incorrect predictions for entities or relations. This indicates the number of entities or relations that were missed. For the JMERE dataset, this example reports the metrics for the relation extraction task JMERE (RE) and the named entity recognition task #MNER (NER), respectively; for the Twitter-2015 dataset, it reports the metrics for the multimodal named entity recognition task.

[0134] II. Model Structure and Key Modules:

[0135] The overall structure of the TESGA-V2 model proposed in this invention is shown in Figure 2. The input sample includes a text segment and its corresponding image. The model first obtains the text context representation through a text encoder, and simultaneously obtains image features and image-text similarity scores through an image-text encoding and matching module. Then, the similarity scores are input into a gated similarity module to generate an adaptive control signal used to adjust the cross-modal fusion strength. This control signal acts on two aspects: firstly, in the entity prediction branch, assisting the model in identifying entity boundaries and entity categories; and secondly, in the relation prediction branch, assisting the model in determining the semantic relationship between the head and tail entities.

[0136] Furthermore, such as Figure 3 As shown, this invention introduces two improved modules based on the original TESGA model: an adaptive threshold gating module and a similarity modulation adaptive layer normalization module (Sim-AdaLN). These modules use the image-text similarity score as a control signal, enabling the model to dynamically adjust the fusion intensity of visual information according to the semantic correlation between text and image, and to adaptively adjust the spatial distribution of the fused features.

[0137] III. Experimental Results of Multimodal Joint Entity Relationship Extraction:

[0138] To verify the effectiveness of this invention in the task of joint entity relation extraction, this embodiment compares TESGA-V2 with the text JERE method, the multimodal pipeline method, and the multimodal joint extraction method on the JMERE dataset. The results are shown in Table 1:

[0139] Table 1

[0140]

[0141] Table 1 above presents the performance comparison results of the method provided in this embodiment of the invention with existing baseline models on the JMERE dataset for the Joint Multimodal Entity Relation Extraction (JMERE) and Named Entity Recognition (#MNER) tasks. As shown in Table 1, the method of this invention achieves 60.36% accuracy, 63.85% recall, and 62.05% F1 score on the JMERE (RE) task; and 82.19% accuracy, 85.16% recall, and 83.65% F1 score on the #MNER (NER) task. Compared with the original TESGA method, this invention improves the F1 score on RE from 60.44% to 62.05%, an improvement of 1.61 percentage points; and on NER from 82.66% to 83.65%, an improvement of 0.99 percentage points. The results show that the present invention can effectively improve the entity relationship judgment ability while maintaining the entity recognition ability, indicating that the proposed similarity gating and similarity modulation adaptive layer normalization mechanism can enhance the effective fusion of image and text information.

[0142] IV. Experimental Results of Multimodal Named Entity Recognition:

[0143] To further verify the generalization ability of this invention on multimodal named entity recognition tasks, this embodiment is compared with various text models and image-text fusion models on the Twitter-2015 dataset. The results are shown in Table 2:

[0144] Table 2

[0145]

[0146] Table 2 above presents the performance comparison results of the method provided in this embodiment of the invention with existing baseline models on the multimodal named entity recognition task on the Twitter-2015 dataset. As shown in Table 2, the method of this invention achieves an accuracy of 81.21%, a recall of 82.55%, and an F1 score of 81.88% on the Twitter-2015 dataset. Compared with the text model BERT-CRF, the F1 score is improved by 10.07 percentage points; compared with the multimodal model HVPNeT, the F1 score is improved by 6.56 percentage points; and compared with the original TESGA method, the F1 score increases from 76.20% to 81.88%, an improvement of 5.68 percentage points. These results demonstrate that in short text and weakly correlated image scenarios on social media, this invention can effectively suppress irrelevant visual noise and enhance entity boundary recognition and entity type classification by utilizing relevant visual cues.

[0147] V. Ablation Experiment:

[0148] To verify the contribution of each core improvement module to model performance, this embodiment conducts ablation experiments on the JMERE dataset, examining the impact of the adaptive threshold gating module, the Sim-AdaLN module, and the complete model structure on performance. The experimental results are shown in Table 3.

[0149] Table 3

[0150]

[0151] Table 3 above shows the ablation experiment results of the model provided in this embodiment of the invention on the JMERE and #MNER tasks, used to verify the beneficial technical effects of each core innovative module, such as Adaptive Threshold Gate and Sim-AdaLN (Adaptive Layer Normalization for Similarity Modulation). As shown in Table 3, compared to TESGA(ALL), after adding the Adaptive Threshold Gate module, the RE F1 score increased from 60.44% to 61.07%, indicating that this module can suppress irrelevant image features through a learnable threshold, thereby improving the discriminative ability of relation extraction. Further adding Sim-AdaLN increased the NER F1 score from 82.39% to 83.41%, indicating that the similarity modulation adaptive layer normalization helps the model adjust entity feature representations according to image-text relevance. The complete model Ours achieved 62.05% and 83.65% on RE F1 and NER F1 respectively, both being the best or near-optimal results in the table, indicating a synergistic effect among the three improved modules.

[0152] It should be noted that the metrics of different subtasks may fluctuate slightly during the training of the deep neural network, but the complete model outperforms the original TESGA model in both relation extraction and named entity recognition tasks, and can stably demonstrate the overall technical effect of this invention in multimodal information extraction scenarios.

[0153] VI. Summary of Implementation Results:

[0154] As can be seen from Tables 1 to 3 and Figures 1 and 2, this invention, by introducing an adaptive threshold gating mechanism driven by image-text similarity and an adaptive layer normalization mechanism for similarity modulation, enables the model to dynamically select the fusion strength of visual information based on the degree of image-text correlation of different samples, reducing the interference of weakly correlated or noisy images on text understanding. This method not only improves the joint entity relation extraction performance in the JMERE scenario but also achieves significant improvements in the Twitter-2015 multimodal named entity recognition task, verifying the effectiveness and practicality of the proposed solution.

[0155] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A multimodal joint entity relation extraction method for improving extraction accuracy, characterized in that, Includes the following steps: S1: Obtain paired multimodal data samples containing text sequences and image content; S2: Perform feature encoding on the text data and image data respectively to obtain the semantic feature representation of the text context and the visual feature representation of the image region, and calculate the image-text similarity score SC between the text sequence and the corresponding image; S3: Perform linear mapping and feature segmentation on the text context semantic feature representation and image visual feature representation to obtain text feature blocks and image feature blocks for the named entity recognition channel and relation extraction channel; S4: Construct a dual-channel feature fusion module, including: naming the entity recognition feature fusion channel NER for generating entity prediction features and the relation extraction feature fusion channel RE for generating relation prediction features; in the dual-channel feature fusion module, the text sequence modeling network is initialized using global image features, and the information interaction between text features and image features is realized through a cross-modal cross attention mechanism; S5: An adaptive threshold gating mechanism is constructed based on the image-text similarity score (SC) to perform gating fusion on the cross-modal cross-attention output. An adaptive layer normalization mechanism based on similarity modulation is used to adjust the feature distribution of the gated fused features to obtain multimodal fused features. The similarity threshold gating includes: generating a gated similarity score based on the image-text similarity score (SC) using learnable gating parameters, and weighting the cross-modal attention output using the gated similarity score to reduce or suppress the influence of image features on text features when the image-text similarity is lower than the dynamic threshold determined by the learnable gating parameters. The similarity modulation adaptive layer normalization includes: performing layer normalization on the gated fused features to obtain standard normalized features, mapping the image-text similarity score (SC) to scaling and offset parameters, and dynamically modulating the standard normalized features using the scaling and offset parameters. S6: Input the multimodal fusion features into the joint prediction module to predict entity boundaries, entity types, and inter-entity relationship types respectively, and output the entity relationship quintuple based on the prediction results.

2. The multimodal joint entity relation extraction method for improving extraction accuracy as described in claim 1, characterized in that: S1 also includes standardization and word segmentation of the acquired text sequence, as well as size and pixel normalization of the corresponding image, denoted as... ,in , This represents the input text sequence. This represents the corresponding image data. Indicates the length of the text sequence. Represents the first in the text sequence One word.

3. The multimodal joint entity relation extraction method for improving extraction accuracy as described in claim 1, characterized in that: The specific calculation method for the text-image similarity score SC in S2 is as follows: S201: Encoding the text sequence using a pre-trained BERT model: , This represents the text sequence feature matrix output by the pre-trained language model. This represents the input text sequence. Indicates the length of the text sequence. This represents the dimension of the initial hidden layer of features output by the pre-trained model; S202: The image is encoded using the MED module of a pre-trained BLIP model. , This represents the image feature matrix output by the visual model. This represents the corresponding image data. Indicates the length of the image feature sequence. This represents the dimension of the initial hidden layer of features output by the pre-trained model; S203: Use BLIP to calculate text-image cosine similarity as a basis for subsequent gating: , This represents the calculated image-text similarity score. This represents the function for calculating cosine similarity. This represents the image and text feature matrix output by the visual model. This represents the multimodal encoder / decoder module in a pre-trained visual model; similarity value. It will serve as the control signal for the gated cross-attention module, used to control the interference of irrelevant image information on entity and relation extraction.

4. The multimodal joint entity relation extraction method for improving extraction accuracy as described in claim 1, characterized in that: The linear mapping and feature block processing methods in S3 are as follows: S301: Text Features Mapping and dividing the data using a trainable linear transformation with Tanh activation: , ; S302: Image features Mapping and dividing the data using a trainable linear transformation with Tanh activation: ， ， in, This represents the comprehensive features of the mapped text. This represents the mapped image text features. , , and, These are trainable parameters. This represents the hyperbolic tangent activation function. Indicates the hidden feature dimension. This indicates a block operation. Indicates the dimension of the partitioning operation. and This indicates a special certificate block.

5. The multimodal joint entity relation extraction method for improving extraction accuracy as described in claim 1, characterized in that: The initialization of the text sequence modeling network using global image features in S4 includes: S401: LSTM is introduced in the first layer of the NER and RE channels to perform pooling processing on the image feature blocks and obtain the global features of the image; S402: Using global image features as the initial hidden states and initial cell states of the text sequence modeling network: ， ， in, and This represents the initial cell state and the initial hidden state. It is the output time feature sequence of the LSTM layer. This represents the max pooling operation; S403: Temporal enhancement modeling of text feature blocks is performed through a text sequence modeling network to obtain text temporal enhancement features containing word order information and long-distance dependency information.

6. The multimodal joint entity relation extraction method for improving extraction accuracy as described in claim 5, characterized in that: The cross-modal cross-attention mechanism in S4 is achieved by fusing the first layer residual + LayerNorm structure through multi-head cross-attention and a feedforward network, including: S404: Using text temporal enhancement features as query vectors and text feature blocks or image feature blocks as key vectors and value vectors, a multi-head cross-attention structure is constructed. S405: Cross-modal attention output is obtained through a multi-head cross-attention structure, and preliminary fusion features are obtained through residual connections and normalization.

7. The multimodal joint entity relation extraction method for improving extraction accuracy as described in claim 1, characterized in that: The adaptive threshold gating mechanism in S5 includes: S501: Set learnable gating parameters and And calculate the gated similarity score based on the image-text similarity score SC. ,in: , This represents the modified linear unit activation function; S502: Gating similarity scores The cross-modal attention output is multiplied element-wise to control the degree to which visual information or cross-modal interaction information enters the text feature channel, and gated fusion features are obtained through residual connections and layer normalization: ， in, This represents the intermediate features after adaptive gating attention fusion. This represents the normalization function of the first standard layer. As key-value feature input, This represents element-wise multiplication. This represents the multi-head cross-attention calculation function. This represents the query vector, key vector, and value vector in a multi-head attention mechanism. S503: When When the gated similarity score is less than or equal to zero, The value is zero or approximately zero, thereby reducing or blocking low-correlation image information from entering the text feature channel; S504: When When the similarity score is greater than zero, the gated similarity score is... The similarity score SC increases with the increase of the image-text similarity score, thereby enhancing the supplementary role of highly relevant image information on text features.

8. The multimodal joint entity relation extraction method for improving extraction accuracy as described in claim 7, characterized in that: The similarity modulation adaptive layer normalization mechanism in S5 is as follows: S505: The gated and fused features are input into the feedforward neural network, and after residual connections and layer normalization, standard normalized features are obtained. : ， in, This represents the standard normalized features that have not undergone similarity modulation. This represents the normalization function of the second standard layer. This represents a feedforward neural network; S506: Input the image-text similarity score into the linear mapping layer to generate scaling parameters for feature modulation. Translation parameters : in, , These represent the scaling and translation parameters generated by dynamic projection of similarity scores, used for feature distribution modulation. , This represents the trainable weights and biases in the similarity linear mapping layer; S507: Based on scaling parameters Translation parameters Standard normalization features Dynamic modulation is performed to obtain multimodal fusion features. The modulation method is as follows: ; in, Indicates standard normalized features, This represents the modulated multimodal fusion features. This indicates element-wise multiplication.

9. The multimodal joint entity relation extraction method for improving extraction accuracy as described in claim 1, characterized in that: The joint prediction module of S6 includes an entity prediction module and a relationship prediction module. The specific prediction methods are as follows; S601 MNER unit: for each entity type Construct a table, calculate the span representation, and predict the probability: ， ， in, , and For trainable parameters, This represents the multimodal features output by S5 specifically for the entity recognition channel. Represents the activation function of the exponential linear unit. This indicates a merge operation. Represents the sigmoid activation function. Represents a collection of entity types Size, Indicates prediction label and As the first The probability of the start and end boundaries of an entity class; S602 MRE Unit: Constructs a relation table, using only the initial... : in, Multimodal features output by S5 specifically for relation extraction channels. The local features corresponding to the starting labels of candidate head and tail entities are represented. Represents global features. The trainable weight matrix and biases of the relation representation layer are represented. Represents entities ( , Relationship characteristics are represented by ) This represents the weight matrix and bias of the relation classification output layer. Indicates the prediction of candidate head and tail entities and There is a specific relationship between them. The probability of.

10. The multimodal joint entity relation extraction method for improving extraction accuracy as described in claim 9, characterized in that: The training and inference methods for the joint prediction module are as follows: S603: Calculate the training loss as the sum of the binary cross-entropy of the MNER and MRE units: in, and Represents the actual labels for entity tables and relation tables. and These are predicted values; ultimately, our training objective is to minimize the loss. . and Indicates the size of the entity table and the relation table; S604: Threshold during inference Filter valid quintuples and assemble entity-relation quintuples by combining MNER and MRE results.

Images