Multimodal entity and relationship extraction method and system based on cross-modal alignment and fusion

Abstract

The present invention discloses a multimodal entity and relationship extraction method and system based on cross-modal alignment and fusion, comprising: processing and encoding input text and images to obtain a variety of image and text features; using the semantic representation of the image as an anchor point, aligning fine-grained and coarse-grained text features with pixel-level image representations, respectively, and mapping the image and text features to the same semantic space; performing multi-granularity feature fusion through text-guided dynamic gating aggregation, visual prefix cross-modal fusion, and cross-modal image-text matching, thereby increasing feature complementarity and modeling the association between noun phrases in the text and image objects to obtain a multi-granularity multimodal feature representation; fusing multi-granularity multimodal features through entity-guided attention gating, aggregating visual information related to text entities, and obtaining a final multimodal fusion representation; and performing task predictions for multimodal named entity recognition and multimodal relationship extraction based on the multimodal fusion representation.

Description

Technical Field

[0001] The present invention belongs to the field of natural language processing in computer intelligent information processing, and specifically relates to a multimodal entity and relationship extraction method and system based on cross-modal alignment and fusion. Background Art

[0002] As an important task in the field of natural language processing, named entity recognition can identify entity objects with specific meanings from text and classify them into predefined entity types. Relation extraction aims to identify and extract the relationship between entities from text. However, with the rise of multimodal data, researchers have begun to realize that relying solely on text data for information extraction cannot meet the needs of comprehensive understanding of information. Therefore, they began to integrate information from different modalities into the original task to obtain more accurate and comprehensive information. In recent years, multimodal named entity recognition (MNER) and multimodal relation extraction (MRE) have gradually become a trend. Its purpose is to realize the communication and conversion of semantic information between different modalities by introducing visual content as additional input, and then accurately extract the information contained in the target modality.

[0003] The main challenge facing MNER and MRE is how to incorporate image information into the text representation learning process to improve NER and RE tasks. So far, researchers have explored a variety of visual representation methods.

[0004] A common approach is to utilize scene graphs, which provide structured semantic information about images. As the first work on MRE, Zheng et al. proposed an efficient graph alignment model that structurally and semantically aligns the textual scene graph generated by a syntactic dependency tree with the visual scene graph. This model better leverages the correlations between visual objects and textual entities, thereby improving the performance of multimodal relation extraction tasks. Research has shown that the introduction of multimodal information supplements the missing semantics in social media text and improves relation extraction performance. However, this approach ignores entity-entity and object-object relationships in the scene graph. Therefore, Yuan et al. proposed an edge-enhanced graph alignment network that fully utilizes edge information in the scene graph to improve node matching accuracy. Separately, Wu et al. first constructed a visual scene graph and a textual scene graph containing object nodes, attribute nodes, and relationship nodes. They then fused the visual and textual scene graphs to obtain a cross-modal graph. Finally, guided by the graph information bottleneck principle, they performed structural refinement, filtering out features with less information and highlighting those relevant to the task. However, heavy reliance on scene graph models may cause loss of image information. At the same time, Li et al. demonstrated through visual shuffle experiments that coarse-grained alignment of scene graphs cannot effectively utilize visual information.

[0005] Other researchers have incorporated object or region-level features detected from the input image into the text-based Transformer architecture to mitigate the noise of irrelevant visual elements while reducing modal heterogeneity. For example, Wang et al. used the semantic representation of the image that directly translated the image into the text space as the visual context, and concatenated it with the original text as the input of the pre-trained text embedding model. Chen et al. used a multi-level pixel-level representation that combined the visual information of the complete image and local objects and used it as the prefix of each self-attention layer of the pre-trained text embedding model. Recently, Liu et al. introduced a hierarchical tracking graph to explore the intrinsic mechanism of integrating pixel-level image features into text semantic representation.

[0006] However, different types of image features provide different perspectives and information on the same activity or phenomenon, and existing work fails to comprehensively utilize all these different image representations to facilitate model decision-making.

[0007] Integrating all these different visual representations is undoubtedly challenging due to the inevitable introduction of noisy data. In terms of modality matching, existing models directly incorporate all image and object information. However, unhelpful visual content and incorrect alignment can easily provide misleading information when predicting relation or entity representations, misleading the model into making incorrect judgments.

[0008] Furthermore, in terms of modality alignment, due to the semantic and pixel-level representations of images, and the fact that original images and text originate from different semantic spaces, language and vision, respectively, there exists a significant modality gap in feature distribution and semantic representation. This inconsistency significantly increases the difficulty for models to capture cross-modal semantic relevance. Existing methods attempt to directly incorporate the image-text contrast loss from large pre-trained models. However, this contrastive learning approach, which directly interacts between image and text features, lacks explicit alignment information and lacks explicitly labeled alignment, resulting in suboptimal image and text features. Summary of the Invention

[0009] To solve the problems existing in the prior art, the present invention provides a multimodal entity and relationship extraction method and system based on cross-modal alignment and fusion, and simultaneously uses the pixel-level representation and semantic representation of the image to promote multimodal named entity recognition (MNER) and multimodal relationship extraction (MRE) tasks to provide rich contextual information.

[0010] To achieve the above object, the present invention provides the following solutions:

[0011] A multimodal entity and relationship extraction method based on cross-modal alignment and fusion, the method comprising:

[0012] Step 1: Process and encode the input text and image to obtain multiple image and text features; the multiple image and text features include: fine-grained text features, coarse-grained text features, fine-grained pixel-level image representation, coarse-grained pixel-level image representation, fine-grained semantic-level image representation, and coarse-grained semantic-level image representation;

[0013] Step 2: Use the semantic representation of the image as an anchor point for aligning image and text features. Perform feature alignment between fine-grained text features and fine-grained pixel-level image representation, and between coarse-grained text features and coarse-grained pixel-level image representation, mapping image and text features into the same semantic space.

[0014] Step 3: Multi-granularity feature fusion is performed through text-guided dynamic gating aggregation, visual prefix cross-modal fusion, and cross-modal image-text matching. This increases feature complementarity while modeling the association between noun phrases in the text and image objects, obtaining fine-grained and coarse-grained multimodal feature representations.

[0015] Step 4: Input the obtained multi-granular multimodal feature representation into the entity-guided attention gating module to aggregate the visual information related to the text entities and obtain the final multimodal fusion representation;

[0016] Step 5: Input the obtained multimodal fusion representation into the task-specific decoder to perform task predictions for multimodal named entity recognition and multimodal relationship extraction respectively.

[0017] Preferably, in step 1, the input text and paired image are processed and encoded to obtain a variety of image and text features, including:

[0018] Acquire sample data, wherein the sample data includes original text and original image data paired with the original text;

[0019] Based on the sample data, five different encoders are used to obtain diverse image representations and corresponding multi-granularity text features. The five different encoders include a complete text encoder, a noun phrase encoder, a complete image encoder, a local object encoder, and an image description and object label encoder.

[0020] Preferably, based on the sample data, five different encoders are used to obtain diverse image representations and corresponding multi-granularity text features, including:

[0021] Use the pre-trained language model Bert to initialize the full text encoder and encode the original text to obtain fine-grained text features. Use the Stanford CoreNLP tool to extract noun phrases from the original sentence text and input them into the noun phrase encoder initialized by the pre-trained language model Bert to obtain coarse-grained text features.

[0022] The full image encoder is initialized using the pre-trained Vision Transformer model and encoded into the original image to obtain fine-grained pixel-level visual features containing 12 layers of visual features. The visual grounding toolkit is used to detect objects using noun phrases extracted from the original text using the Stanford CoreNLP tool. The top N most relevant local objects are extracted from each original image and input into the local object encoder initialized by the pre-trained Vision Transformer model to obtain coarse-grained pixel-level visual features containing 12 layers of visual features.

[0023] The original image is input into the ViT-GPT2 model to obtain an image description representing the global information of the image. The image description is then regarded as plain text and encoded using the image description and object label encoder initialized by the pre-trained language model Bert to obtain a fine-grained semantic-level image representation of the image. The object detector Faster-RCNN is used to detect objects in the image, and then the object labels are encoded using the image description and object label encoder to obtain a coarse-grained semantic-level image representation of the image.

[0024] Preferably, in step 2, the semantic representation of the image is used as an anchor point for aligning image and text features, and feature alignment of fine-grained text features and fine-grained pixel-level image representation, as well as coarse-grained text features and coarse-grained pixel-level image representation, is performed respectively. Mapping the image and text features to the same semantic space includes:

[0025] The obtained coarse-grained and fine-grained text representations, image pixel representations, and image semantic representations are processed separately, including:

[0026] Perform feature alignment of text and image semantic representations. Where B represents the batch size, p,q represent the sample index labels, and assume that the positive example is a pair of B representations from the same input pair {T p ,S q} p＝q , other representations from different input pairs are considered negative examples;

[0027] Use two multi-layer perceptron layers to p and S q The global representation embedding [CLS] performs nonlinear mapping to obtain the low-dimensional embedding representation after mapping Where G represents the global feature after nonlinear mapping, d represents the dimension of the embedding space, and the global representation embedding [CLS] comes from the last layer output of the image encoder and the text encoder;

[0028] Using cosine similarity To calculate the similarity between text and image semantic representation, the similarity p between text representation and image semantic representation in a batch t2s Defined as

[0029] Where τ is a learnable temperature parameter; similarly, the similarity p between the semantic representation of the image and the text representation within a batch s2t Defined as

[0030] Let y t2s (I) and y s2t (I) represents the true one-hot similarity vector, maximizing the similarity of positive examples and minimizing the similarity of negative examples. The feature alignment contrast loss of text and image semantic representation is defined as the binary cross entropy H:L between p and y. ts =H(p t2s ,y t2s (I))+H(p s2t ,y s2t (I));

[0031] Perform feature alignment between image pixel representation and image semantic representation. Where B represents the batch size, p,q represent the sample index labels, and it is assumed that the positive example is a pair of B representations from the same input pair {I p ,S q} p＝q , other representations from different input pairs are considered negative examples;

[0032] Using two multi-layer perceptron layers to p and S q The global representation embedding [CLS] performs nonlinear mapping to obtain the low-dimensional embedding representation after mapping Where G represents the global feature after nonlinear mapping, d represents the dimension of the embedding space, and the global representation embedding [CLS] comes from the last layer output of the image encoder and the text encoder;

[0033] Using cosine similarity To calculate the similarity between the image pixel representation and the image semantic representation, the similarity p between the image pixel representation and the image semantic representation in a batch i2s Defined as

[0034] Where τ is a learnable temperature parameter. Similarly, the similarity p between the semantic representation of an image and the pixel representation of an image within a batch is s2i Defined as

[0035] Let y i2s (I) and y s2i (I) represents the true one-hot similarity vector, maximizing the similarity of positive examples and minimizing the similarity of negative examples. The feature alignment contrast loss between image pixel representation and image semantic representation is defined as the binary cross entropy H:L between p and y. is =H(p i2s ,y i2s (I))+H(p s2i ,y s2i (I));

[0036] The final semantic anchor contrast loss is defined as:

[0037]

[0038] Preferably, in step 3, performing multi-granularity feature fusion through text-guided dynamic gated aggregation, visual prefix cross-modal fusion, and cross-modal image-text matching to obtain fine-grained and coarse-grained multimodal feature representations includes:

[0039] The coarse-grained and fine-grained text representations, image pixel representations, and image semantic representations after feature alignment are processed separately; including:

[0040] The fine-grained text representation T f And image representation I f and S f Perform fine-grained feature fusion based on visual prefix transformer blocks;

[0041] Perform text-guided dynamic gating aggregation to obtain complementary image features of text at a specific level; the dynamic gating signal represents the probability of the k-th layer visual features obtained from the image encoder to the l-th fine-grained visual prefix transformer block:

[0042]

[0043] Where k and l represent the layer indices of the image encoder and visual prefix transformer blocks, respectively, and K represents the maximum number of layers of the image encoder. represents the fusion representation of the k-th layer visual features and fine-grained text features, W I is a learnable weight parameter, (α f ) l is the dynamic gating signal of the lth layer, Represents the representation of the global text features output by the previous layer of visual prefix transformer, represents the k-th layer of fine-grained pixel-level visual features, [,] represents the cascade operation;

[0044] Using the gate signal (α f ) l Get the final aggregated fine-grained hierarchical visual features

[0045] Project the aggregated fine-grained hierarchical visual features onto the visual key vector and visual value vector middle in is the visual attention mapping parameter, which encodes the fine-grained text features of the l-1 layer Projection to cross-attention text query vector key vector and value vector middle, Perform visual prefix cross-modal fusion, where They are the attention mapping parameters; the implicit features of the lth layer are calculated by visual prefix attention Iteratively update layer by layer, and the hidden features of the last layer are the cross-modal fine-grained text feature encoding representation;

[0046] Cross-modal image-text matching is performed based on the similarity matrix obtained by semantic anchor comparison learning;

[0047] Set the diagonal elements of the matrix to 0 to avoid comparison with positive samples; then, sample negative text and negative images from the same batch according to the similarity distribution, and assign sampling probabilities based on their similarity with the target image or text;

[0048] Use the last layer output of the visual prefix transformer block to embed the [CLS] token as the fusion representation of the image-text pair and predict the probability p of the image and text matching. itm , get the image-text matching loss L itm =H(y itm ,p itm (T,I)), where y itm is a two-dimensional one-hot vector representing the true value label;

[0049] Text representation T f The paired image representation I f and S f The matching probability prediction of the module is used to obtain the fine-grained multimodal feature M f ∈R n×d : Where L represents the maximum number of layers of the fine-grained visual prefix transformer block, and Represents the probability of image and text matching, which is used to determine the fine-grained multimodal features after the fusion of the retained fine-grained text features and the fine-grained pixel-level visual features. Fine-grained multimodal features after fusion of fine-grained text features and fine-grained semantic representations proportion;

[0050] Perform fine-grained text representation T c And image representation I c and S c The coarse-grained feature fusion of C .

[0051] Preferably, in step 4, inputting the obtained fine-grained and coarse-grained multimodal feature representations into the entity-guided attention gating module to obtain the final multimodal fusion representation includes:

[0052] The multimodal feature M C and M f As the input of attention gate, obtain fine-grained entity representation E f As query Q, coarse-grained noun phrase-object features as keys K and values ​​V, E f is used to focus on relevant image object features and generate entity-aware object representations: E c =Cross-att(Q=E f ,K=M c ,V=M c );

[0053] Use gate signal g e =sigmoid(W e [M c ,E f ]) Control entity-aware object representation E C The contribution of , to obtain the final multimodal fusion representation: E = g e *E c +(1-g e )*E f .

[0054] Preferably, in step 5, inputting the obtained multimodal fusion representation into a task-specific decoder to perform task predictions of multimodal named entity recognition and multimodal relationship extraction respectively includes:

[0055] Based on the obtained multimodal fusion representation E, named entity recognition and relation classification are performed respectively; including: sending the multimodal fusion representation to the CRF layer, and for the obtained label sequence y, using the cross entropy loss function to calculate the multimodal named entity recognition task loss: Where n is the length of the sequence, i represents the index number of the sequence;

[0056] The multimodal representation is input into the multilayer perceptron, and the relationship classification is performed through the softmax classifier; the cross entropy loss function is used to calculate the multimodal relationship extraction task loss: Among them, r represents the true relationship category;

[0057] Based on the multimodal named entity recognition task loss and the multimodal relationship extraction task loss, the final loss function L=L ner / re +L SACL +L itm , where L ner / re Represents the loss of the NER or RE task, and the final loss function is used to iteratively optimize the model.

[0058] The present invention also provides a multimodal entity and relationship extraction system based on cross-modal alignment and fusion, the system being used to implement any one of the methods described above, the system comprising: a diversified multimodal feature acquisition module, a semantic anchor contrast learning module, a multi-granularity visual-text collaborative fusion module, an entity-guided attention gating module, and a task-specific decoder module;

[0059] The diversified multimodal feature acquisition module is used to process and encode input text and images to obtain multiple image and text features; wherein the multiple image and text features include: fine-grained text features, coarse-grained text features, fine-grained pixel-level image representation, coarse-grained pixel-level image representation, fine-grained semantic-level image representation, and coarse-grained semantic-level image representation;

[0060] The semantic anchor contrast learning module is used to use the semantic representation of the image as the anchor point for aligning image and text features, and to align fine-grained text features with fine-grained pixel-level image representations, as well as coarse-grained text features with coarse-grained pixel-level image representations, to map image and text features into the same semantic space;

[0061] The multi-granularity visual text collaborative fusion module is used to perform multi-granularity feature fusion through text-guided dynamic gating aggregation, visual prefix cross-modal fusion, and cross-modal image-text matching, thereby increasing feature complementarity while modeling the association between noun phrases in the text and image objects, and obtaining fine-grained and coarse-grained multimodal feature representations;

[0062] The entity-guided attention gating module is used to input the obtained multi-granularity multimodal feature representation into the entity-guided attention gating module, aggregate visual information related to the text entity, and obtain the final multimodal fusion representation;

[0063] The task-specific decoder module is used to input the obtained multimodal fusion representation into the task-specific decoder to perform task predictions of multimodal named entity recognition and multimodal relationship extraction respectively.

[0064] Compared with the prior art, the present invention has the following beneficial effects:

[0065] The present invention uses five different encoders to obtain diverse image representations and corresponding multi-granularity text features. Different types of image features provide different perspectives and information on the same activity or phenomenon. The present invention comprehensively utilizes all different image features to provide rich contextual information to promote multimodal named entity recognition and multimodal relationship extraction.

[0066] The present invention introduces the semantic representation of images as anchors to ease image-text contrast learning. Compared with direct interaction between images and text, the introduction of these anchor models makes it easier to capture the representation of modality-invariant factors. In this way, image and text representations from different encoders are aligned before modal fusion, and a consistent representation is constructed to bridge the modality gap and promote cross-modal alignment.

[0067] The present invention adopts a multi-granularity vision-text collaborative fusion strategy for cross-modal fusion, adjusts the visual information weight through text-guided dynamic gating, takes noun phrases in the text as candidate entities, models the association between text entities and image objects, and improves cross-modal interaction; finally, entity-guided attention gating is introduced to retain visual features that are semantically consistent with text entities, effectively alleviating the interference of modal noise on the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0069] Figure 1 A model architecture diagram provided for an embodiment of the present invention;

[0070] Figure 2 A schematic diagram of a semantic anchor point contrast learning module provided in an embodiment of the present invention;

[0071] Figure 3Schematic diagram of text-guided dynamic gating of a multi-granularity visual text collaborative fusion module provided by an embodiment of the present invention;

[0072] Figure 4 Schematic diagram of visual prefix cross-modal fusion of the multi-granularity visual text collaborative fusion module provided by an embodiment of the present invention;

[0073] Figure 5 Two example study diagrams provided for embodiments of the present invention;

[0074] Figure 6 The embodiment of the present invention provides a method for removing the influence of different image features on the method proposed by the present invention. Figure 6 (a) is the performance of the method on the Twitter2017 dataset. Figure 6 (b) is the performance of the method on the MNRE dataset;

[0075] Figure 7 The visualization results of the modality gap between text and image provided by the embodiment of the present invention compared with the existing methods are as follows: Figure 7 (a) is the feature visualization result generated by the method HVPNet, Figure 7 (b) is the feature visualization result generated by the AMNet method. Figure 7 (c) Feature visualization results generated by the SAGE method proposed in the present invention;

[0076] Figure 8 The low-resource setting performance of the proposed method on MRE and MNER tasks provided by the embodiment of the present invention, Figure 8 (a) is the performance of the method on the Twitter2017 dataset. Figure 8 (b) shows the performance of the method on the MNRE dataset. DETAILED DESCRIPTION

[0077] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0078] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0079] Example 1

[0080] Existing multimodal entity and relationship extraction methods still face many problems in modal representation, modal alignment, and modal matching, such as underutilization of diverse image representations, alignment difficulties caused by the modality gap, and information interference caused by modal noise. This paper proposes a multimodal entity and relationship extraction method based on cross-modal alignment and fusion. It simultaneously uses pixel-level and semantic representations of images to facilitate multimodal named entity recognition (MNER) and multimodal relation extraction (MRE) tasks, providing rich contextual information.

[0081] The present invention observes that objects in images can be accurately detected and are often mentioned in paired text. Therefore, to address the problem of modal alignment, the present invention proposes a semantic anchor contrastive learning method for cross-modal alignment. This method introduces the semantic representation of the image as an anchor for contrastive learning of image and text features, maps image and text features to the same space, and constructs a consistent representation before modal fusion to bridge the modality gap. Compared to existing methods that directly interact between image and text, introducing these anchor models makes it easier to capture representations of modality-invariant factors.

[0082] Existing methods directly introduce all visual information, ignoring the correlation between text entities and objects in the image, although they have the potential to improve cross-modal interaction. Therefore, in order to solve the modal matching problem, the present invention proposes a new cross-modal fusion method, which adopts a multi-granularity visual-text collaborative mechanism to capture fine-grained and coarse-grained cross-modal associations at the same time to increase the complementarity between features. Specifically, the present invention designs text-guided dynamic gating, which dynamically adjusts the weight of visual information according to the text content, obtains complementary image features of text at a specific level, and uses a visual prefix-based attention method to fuse visual and text features. Through layer-by-layer iterative updates, it promotes the deep fusion of multimodal features. Finally, a cross-modal image-text matching loss is proposed to control the degree of integration of image information and reduce the influence of irrelevant visual features.

[0083] At the same time, the present invention uses noun phrases in the text as candidate entities and models their relevance to the target object in the image during the interaction process. Finally, the candidate entities are used as a bridge for interaction between text and image. Through entity-guided attention gating, all objects in the image are screened, and the key visual content is focused on information related to the text entity. By modeling the relevance between text candidate entities and target objects, cross-modal interaction is improved and modal noise is alleviated. The specific implementation process is as follows:

[0084] like Figure 1 As shown, the embodiment of the present invention provides a multimodal entity and relationship extraction method based on cross-modal alignment and fusion, which includes the following operations:

[0085] Step 1: Using a diversified multimodal feature acquisition module, the input text and paired images are processed and encoded to obtain a variety of image and text features, including fine-grained text features, coarse-grained text features, fine-grained pixel-level image representations, coarse-grained pixel-level image representations, fine-grained semantic-level image representations, and coarse-grained semantic-level image representations. Due to their different forms of existence, existing methods can only apply one of the pixel-level representation or semantic representation of the image. However, different types of image features provide different perspectives and information on the same activity or phenomenon. The present invention comprehensively utilizes all different image representations to provide rich contextual information to promote multimodal named entity recognition and multimodal relationship extraction.

[0086] Step 2: The semantic representation and pixel-level representation of the image, as well as the original image and text coming from two different semantic spaces, language and vision, respectively, have a significant modality gap in feature distribution and semantic representation. This inconsistency significantly increases the difficulty of the model in capturing cross-modal semantic correlations. However, most existing methods ignore the modality gap. The present invention proposes a semantic anchor contrast learning module for cross-modal alignment, using the semantic representation of the image as the anchor for aligning image and text features, and performing feature alignment of fine-grained text features and fine-grained pixel-level image representations, as well as coarse-grained text features and coarse-grained pixel-level image representations, to map image and text features to the same semantic space. Compared to directly interacting between images and text, introducing these anchor models makes it easier to capture the representation of modality-invariant factors.

[0087] Step 3: Existing methods directly incorporate all visual information, ignoring the correlation between text entities and objects in the image, despite their potential to improve cross-modal interaction. Therefore, this paper proposes a new cross-modal fusion method that employs a multi-granularity visual-text collaborative fusion module to perform multi-granular feature fusion through text-guided dynamic gating aggregation, visual prefix cross-modal fusion, and cross-modal image-text matching, obtaining both fine-grained and coarse-grained multimodal feature representations. By modeling the correlation between text candidate entities and target objects, this paper effectively improves cross-modal interaction and mitigates modal noise.

[0088] Step 4: Input the fine-grained and coarse-grained multimodal feature representations obtained in step 3 into the entity-guided attention gating module to aggregate the visual information related to the text entities and obtain the final multimodal fusion representation.

[0089] Step 5: Input the multimodal fusion representation obtained in step 4 into the task-specific decoder to perform task predictions for multimodal named entity recognition and multimodal relationship extraction respectively.

[0090] In this embodiment, the diversified multimodal feature acquisition in step 1 includes the following specific operations:

[0091] Obtaining sample data, where the sample data includes original text and original image data paired with the original text;

[0092] Use five different encoders to obtain diverse image representations and corresponding multi-granularity text features, including a full text encoder, a noun phrase encoder, a full image encoder, a local object encoder, and an image description and object label encoder;

[0093] Use the pre-trained language model Bert to initialize the complete text encoder, encode the original text, and obtain fine-grained text features where d T The dimension of each token; use the Stanford CoreNLP tool to extract noun phrases from the original sentence text, and then input it into the noun phrase encoder initialized by the pre-trained language model Bert to obtain coarse-grained text features Where f and c are used to represent fine-grained and coarse-grained, respectively;

[0094] Use the pre-trained vision model Vision Transformer (Vit) to initialize the full image encoder and encode the original image to obtain fine-grained pixel-level visual features containing 12 layers of visual features: in represents the output embedding representation of the kth layer Vit, d v is the image feature dimension;

[0095] The Stanford CoreNLP tool is used to extract noun phrases from the original text, and the visual grounding toolkit is used to detect objects. The top N most relevant local objects are extracted from each original image and input into the local object encoder initialized by Vit to obtain coarse-grained pixel-level visual features containing 12 layers of visual features.

[0096] The original image is input into the ViT-GPT2 model to obtain an image description representing the global information of the image. The image description is then treated as plain text and encoded using the image description and object label encoder initialized by the pre-trained language model Bert to obtain a fine-grained semantic image representation of the image. M c Indicates the length of the text of the obtained image description; use the object detector Faster-RCNN to detect the objects in the image, and then use the image description and object label encoder to encode the object label to obtain a coarse-grained semantic level image representation of the image M t Indicates the length of the obtained object label text after connection.

[0097] In this embodiment, the image and text feature comparison learning in step 2 includes the following specific operations:

[0098] like Figure 2 As shown, the obtained coarse-grained and fine-grained text representations, image pixel representations, and image semantic representations are processed separately, including:

[0099] Perform feature alignment of text and image semantic representations. Where B represents the batch size, p,q represent the sample index labels. Assume that the positive example is a pair of B representations from the same input pair {T p ,S q} p＝q , other representations from different input pairs are considered negative examples;

[0100] Use two multi-layer perceptron layers to p and S q The global representation embedding [CLS] performs nonlinear mapping to obtain the low-dimensional embedding representation after mapping Where G represents the global feature after nonlinear mapping, d represents the dimension of the embedding space, and the global representation embedding [CLS] comes from the last layer output of the image encoder and the text encoder;

[0101] Using cosine similarity To calculate the similarity between text and image semantic representation, the similarity p between text representation and image semantic representation in a batch t2s Defined as

[0102] Where τ is a learnable temperature parameter. Similarly, the similarity p between the semantic representation of the image and the text representation within a batch s2t Defined as

[0103] Let y t2s (I) and y s2t (I) represents the true one-hot similarity vector, maximizing the similarity of positive examples and minimizing the similarity of negative examples. The feature alignment contrast loss of text and image semantic representation is defined as the binary cross entropy H:L between p and y. ts =H(p t2s ,y t2s (I))+H(p s2t ,y s2t (I));

[0104] The same method is used to align the features of image pixel representation and image semantic representation. Where B represents the batch size, p,q represent the sample index labels. Assume that the positive example is a pair of B representations from the same input pair {I p ,S q} p＝q , other representations from different input pairs are considered negative examples;

[0105] Using two multi-layer perceptron layers to p and S q The global representation embedding [CLS] performs nonlinear mapping to obtain the low-dimensional embedding representation after mapping Where G represents the global feature after nonlinear mapping, d represents the dimension of the embedding space, and the global representation embedding [CLS] comes from the last layer output of the image encoder and the text encoder;

[0106] Using cosine similarity To calculate the similarity between the image pixel representation and the image semantic representation, the similarity p between the image pixel representation and the image semantic representation in a batch i2s Defined as

[0107] Where τ is a learnable temperature parameter. Similarly, the similarity p between the semantic representation of an image and the pixel representation of an image within a batch is s2i Defined as

[0108] Let y i2s (I) and y s2i (I) represents the true one-hot similarity vector, maximizing the similarity of positive examples and minimizing the similarity of negative examples. The feature alignment contrast loss between image pixel representation and image semantic representation is defined as the binary cross entropy H:L between p and y. is =H(p i2s ,y i2s (I))+H(p s2i ,y s2i (I));

[0109] The final semantic anchor contrast loss is defined as the average of these two parts of loss:

[0110]

[0111] like Figure 3 As shown, in this embodiment, the multi-granularity visual text collaborative fusion in step 3 includes the following specific operations:

[0112] The coarse-grained and fine-grained text representations, image pixel representations, and image semantic representations after feature alignment are processed separately; including:

[0113] The fine-grained text representation Tf And image representation I f and S f Perform fine-grained feature fusion based on the visual prefix transformer block (composed of several blocks);

[0114] Perform text-guided dynamic gating aggregation to obtain complementary image features of text at a specific level; the dynamic gating signal represents the probability of the k-th layer visual features obtained from the image encoder to the l-th fine-grained visual prefix transformer block:

[0115]

[0116] where k and l represent the layer indices of the image encoder and visual prefix transformer blocks, respectively, and K represents the maximum number of layers of the image encoder. represents the fusion representation of the k-th layer visual features and fine-grained text features, W I is a learnable weight parameter, (α f ) l is the dynamic gating signal of the lth layer, Represents the representation of the global text feature output by the previous layer of visual prefix transformer, [,] represents the cascade operation, represents the k-th layer of fine-grained pixel-level visual features, and [,] represents the cascade operation.

[0117] Using the gate signal (α f ) l Get the final aggregated fine-grained hierarchical visual features in is the gate weight corresponding to the k-th layer visual feature.

[0118] like Figure 4 As shown, the aggregated fine-grained hierarchical visual features are projected onto the visual key vector and visual value vector middle in is the visual attention mapping parameter. The fine-grained text feature encoding of the l-1 layer is represented Projection to cross-attention text query vector key vector and value vector middle, Perform visual prefix cross-modal fusion, where They are the attention mapping parameters; the implicit features of the lth layer are calculated by visual prefix attention Iteratively update layer by layer, and the hidden features of the last layer are the cross-modal fine-grained text feature encoding representation;

[0119] Similarity p obtained by semantic anchor point comparison learning t2s and p s2t ,construct a similarity matrix for cross-modal image-text matching, where the diagonal elements represent matching text and semantic representation pairs, while the other elements represent unmatched text and semantic representation pairs;

[0120] Set the diagonal elements of the matrix to 0 to avoid comparison with positive samples; then, sample negative text and negative images from the same batch according to the similarity distribution, where samples with higher similarity to the target image or text are given a higher sampling probability;

[0121] Use the last layer output of the visual prefix transformer block to embed the [CLS] token as the fusion representation of the image-text pair and predict the probability p of the image and text matching. itm , get the image-text matching loss L itm =H(y itm ,p itm (T,I)), where y itm is a two-dimensional one-hot vector representing the true value label;

[0122] Text representation T f The image paired with it represents I f and S f The matching probability prediction of the module is used to obtain the fine-grained multimodal feature M f ∈R n×d : Where L represents the maximum number of layers of the fine-grained visual prefix transformer block, and Represents the probability of image and text matching, which is used to determine the fine-grained text features that should be retained and the fine-grained multimodal features after the fusion of fine-grained pixel-level visual features. Fine-grained multimodal features after fusion of fine-grained text features and fine-grained semantic representations proportion;

[0123] Use the same method to perform coarse-grained text representation T c And image representation I c and S c The coarse-grained feature fusion of C .

[0124] Perform text-guided dynamic gating aggregation to obtain complementary image features of text at a specific level; the dynamic gating signal represents the probability of the k-th layer visual features obtained from the image encoder to the l-th coarse-grained visual prefix transformer block:

[0125]

[0126] where k and l represent the layer indices of the image encoder and visual prefix transformer blocks, respectively, and K represents the maximum number of layers of the image encoder. represents the fusion representation of the k-th layer visual features and coarse-grained text features, W I is a learnable weight parameter, (α c ) l is the dynamic gating signal of the lth layer, Represents the representation of the global text feature output by the previous layer of visual prefix transformer, [,] represents the cascade operation, represents the k-th layer of coarse-grained pixel-level visual features, and [,] represents the cascade operation.

[0127] Using the gate signal (α c ) l Get the final aggregated coarse-grained hierarchical visual features in is the gate weight corresponding to the k-th layer visual feature.

[0128] like Figure 4 As shown, the aggregated fine-grained hierarchical visual features are projected onto the visual key vector and visual value vector middle in is the visual attention mapping parameter. The coarse-grained text feature encoding of the l-1 layer is represented Projection to cross-attention text query vector key vector and value vector middle, Perform visual prefix cross-modal fusion, where They are the attention mapping parameters; the implicit features of the lth layer are calculated by visual prefix attention Iteratively update layer by layer, and the hidden features of the last layer are the cross-modal coarse-grained text feature encoding representation;

[0129] Similarity p obtained by semantic anchor point comparison learning t2s and p s2t,construct a similarity matrix for cross-modal image-text matching, where the diagonal elements represent matching text and semantic representation pairs, while the other elements represent unmatched text and semantic representation pairs;

[0130] Set the diagonal elements of the matrix to 0 to avoid comparison with positive samples; then, sample negative text and negative images from the same batch according to the similarity distribution, where samples with higher similarity to the target image or text are given a higher sampling probability;

[0131] Use the last layer output of the visual prefix transformer block to embed the [CLS] token as the fusion representation of the image-text pair and predict the probability p of the image and text matching. itm , get the image-text matching loss L itm =H(y itm ,p itm (T,I)), where y itm is a two-dimensional one-hot vector representing the true value label;

[0132] Text representation T c The image paired with it represents I c and S c The matching probability prediction of the module is used to obtain the fine-grained multimodal feature M c ∈r n×d : Where L represents the maximum number of layers of the fine-grained visual prefix transformer block, and Represents the probability of image and text matching, which is used to determine the coarse-grained text features that should be retained and the coarse-grained multimodal features after the fusion of coarse-grained pixel-level visual features. And the coarse-grained multimodal features after the fusion of coarse-grained text features and coarse-grained semantic representations proportion;

[0133] In this embodiment, step 4 to obtain the final multimodal fusion representation includes the following specific operations: C and M f As the input of attention gate, obtain fine-grained entity representation E f As query Q, coarse-grained noun phrase-object features as key K and value V. E f is used to focus on relevant image object features and generate entity-aware object representations: E c =Cross-att(Q=E f ,K=M c ,V=M c )

[0134] Use gate signal g e =sigmoid(W e [M c ,E f ]) Control entity-aware object representation E C The contribution of W e Represents the trainable parameters, and obtains the final multimodal fusion representation E:

[0135] E=g e *E c +(1-g e )*E f

[0136] In this embodiment, the task-specific decoding in step 5 includes the following specific operations:

[0137] Based on the obtained multimodal fusion representation E, named entity recognition and relationship classification are performed respectively; including:

[0138] The multimodal fusion representation is fed into the CRF layer, and for the obtained label sequence y, the cross entropy loss function is used to calculate the multimodal named entity recognition task loss: Where n is the length of the sequence, i represents the index number of the sequence;

[0139] The multimodal representation is input into the multilayer perceptron, and the relationship classification is performed through the softmax classifier; the cross entropy loss function is used to calculate the multimodal relationship extraction task loss: Where r represents the true relationship category;

[0140] The final loss function L=L obtained based on the above loss function ner / re +L SACL +L itm , where L ner / re Represents the loss of the NER or RE task (selected according to the specific task), and the final loss function is used to iteratively optimize the model.

[0141] Finally, the present invention Figure 5 Two examples of multimodal relation extraction are used to demonstrate the effectiveness and practical application value of the proposed method SAGE; Figure 5 The middle nodes represent the text entity candidates and the target objects in the image, where the subject and object are underlined. The relevance is defined based on the attention score of the last layer of coarse-grained feature fusion Transformer block in the multi-granularity visual-text collaborative fusion module.

[0142] In the example on the left, AMNet was misled by the three people in the image and predicted a relationship between them. Our method captures that while the object is related to the person, the subject is weakly related, successfully predicting the "none" type. In the example on the right, the text-based BERT model fails to capture the clues contained in the image. Our method achieves the correct prediction by establishing an effective link between the candidate entity and the target object in the image.

[0143] To demonstrate the effectiveness of our proposed multimodal entity and relation extraction method, SAGE, we used the following public datasets for validation: Twitter2015 and Twitter2017 for multimodal named entity recognition (MNER) and the MNRE dataset for multimodal relation extraction (MRE). Precision, recall, and F1-score were used as the primary evaluation metrics.

[0144] A comprehensive comparison was conducted with state-of-the-art and representative models on standard datasets. The proposed method, SAGE, was evaluated against three groups of models. Text-based models: The following representative models were considered, including Bert-CRF and MTB. LLM-based models: Two representative LLM baselines were selected, including ChatGPT and GPT4. Multimodal models: Multimodal models previously used for MNER and MRE tasks were considered, including EEGA, IFAformer, HVPNet, MMIB, HVFormer, VisualPT-MoE, AMNet, DebiasCL, MGCMT, and GBIT.

[0145] Table 1 MRE experimental results

[0146]

[0147] Table 2 MNER experimental results

[0148]

[0149] From Table 1 and Table 2, we can observe that:

[0150] First, all multimodal models performed better than text-based models. This demonstrates that enhancing text semantics with visual features can indeed improve model performance on both NER and MRE tasks, with this being particularly evident in MRE. However, for MNER, the performance improvement from incorporating visual information remains limited, necessitating further exploration of multimodal approaches.

[0151] Secondly, compared to other multimodal methods, methods that bridge the modality gap and mitigate modal noise, such as AMNet and MMIB, offer greater performance gains. This demonstrates the importance of consistent image-text representations and modeling cross-modal associations. Despite this, the final performance on both tasks fell short of expectations, indicating the need for more effective strategies to address these two issues.

[0152] Finally, the method proposed in this paper has significant improvements in various performance indicators compared with previous models. Compared with the previous best-performing model, the method proposed in this paper achieved F1-score performance gains of 3.02%, 0.30%, and 0.49% on three datasets, respectively, demonstrating the effectiveness of the model in multimodal entity and relationship extraction tasks.

[0153] To verify the effectiveness of each component in the present invention, the model variants were experimentally verified using the same experimental parameters. The experimental results are shown in Tables 3 and 4.

[0154] Table 3 Ablation experiment results of MRE task

[0155]

[0156] Table 4 MNER task ablation experiment results

[0157]

[0158]

[0159] Without HVF: In this ablation experiment, we only utilize the highest-level features of the pixel-level image representation. As shown in our results, this leads to performance degradation, indicating that directly using the high-level image embeddings of the visual encoder as a prior is suboptimal for multimodal entity and relation extraction. Multi-layered pixel-level image representations provide rich contextual information for the task.

[0160] w / o SACL: In this ablation experiment, the present invention replaces semantic anchor contrastive learning with the image-text contrastive loss (ITC) commonly used in large pre-trained models. Since this contrastive learning method of direct interaction between image and text features lacks explicit alignment information and no explicitly marked alignment, the model performance is degraded.

[0161] This paper further explores the role of image captions and target labels in the alignment of image and text features. "w / o SACL caption" and "w / o SACL tag" respectively indicate the use of only coarse-grained semantic-level image representation and fine-grained semantic-level image representation as semantic anchors for image and text feature alignment. The results show that both features are indispensable. They describe images at different granularities and play an important role in the alignment of text and image features.

[0162] Without EG: This paper removes entity-guided attention gating and directly concatenates the multi-granularity image-text fusion features for model prediction. We observed F1-score drops of 1.25%, 0.19%, and 0.30 on the three datasets, respectively. This suggests that effectively linking entities in text with target objects in images has the potential to improve cross-modal interaction.

[0163] To further verify the effectiveness and plug-and-play nature of the text-guided dynamic gating module in the proposed method, the module was integrated into an existing model. The RSRNet model uses a ResNeSt network to obtain pyramid visual features. Similar to HVPnet, it also uses a learnable gating controlled by a single image modality to determine the validity of the image, specifically:

[0164] AMNet believes that the gate signals obtained by this learnable neural network introduce noise, and experiments by sequentially combining the first six layers of global image features and the last six layers of object image features as visual prefixes. However, this approach forgoes the full utilization of multi-layer visual features.

[0165] The text-guided dynamic gating aggregation module proposed in this paper was combined with the aforementioned models, using the multimodal relation extraction (MRE) task as an example. The experimental results are shown in Table 5. Compared to the original model, the model integrating the text-guided dynamic gating module improved the F1 score by 1.33%, 0.63%, and 0.51%, respectively.

[0166] Table 5 Module verification experiment results

[0167]

[0168] In order to explore the effectiveness of image representation on different tasks, the present invention controls the image representation used in multi-granularity image-text feature fusion. Figure 6 The importance of different image features is shown for the tasks of Multimodal Named Entity Recognition (MNER) and Multimodal Relation Extraction (MRE), taking the Twitter2017 and MNRE datasets as examples, respectively.

[0169] First, it can be observed from the results that removing any image feature will lead to a decline in performance. Diverse image features are complementary in multimodal tasks, providing the model with diverse sources of information and observation perspectives. Second, for multimodal named entity recognition (MNER) and multimodal relation extraction (MRE) tasks, pixel-level representation plays a more significant role than semantic representation. Multi-level pixel-level representation can capture richer visual information, thus providing more comprehensive support for the task. In addition, directly translating images into text may result in the loss of some information, thereby limiting the model's full understanding of the image content. Finally, different tasks have task-specific requirements for image information. Specifically, the MNER task focuses more on global image representation, while the MRE task relies more on local object representation.

[0170] To verify the effectiveness of semantic anchor contrastive learning in alleviating the modality gap, we performed a visual analysis of text and image features and compared them with other models. First, we randomly selected 50 test samples from the MNRE dataset, obtained the text-image pair features output by the trained SAGE method, and mapped them into a two-dimensional space using the t-SNE tool.

[0171] Figure 7 The visualization comparison results of the proposed SAGE method with the HVPNet and AMNet models are shown. As can be seen from the figure, HVPNet treats visual representations as insertable visual prefixes and inserts them into the text encoder, reducing interference with the text representation architecture. However, the visualization results indicate that a modality gap still exists. AMNet introduces the image-text contrast loss commonly used in large-scale visual-language pre-training models. However, this contrastive learning method, which directly interacts between image and text features, lacks explicit label alignment and cannot achieve sufficient training for downstream tasks. Therefore, while it alleviates the modality gap to some extent, the results are not ideal, and some differences in feature distribution still exist. In contrast, SAGE performs well in aligning text and image features, resulting in a more concentrated feature distribution. The introduction of semantic anchor contrastive learning effectively bridges the modality gap.

[0172] The present invention conducts experiments in resource-scarce scenarios, randomly extracting 5% to 50% from the original training set. The experimental results are as follows: Figure 8As shown. From the figure, we can observe that: 1) In low-resource scenarios, the performance of SAGE and HVPNet is always better than the baseline; this result shows that the use of multi-level image pixel-level representation (including complete images and target objects) has a significant promoting effect on NER and RE tasks. 2) In addition, the SAGE method proposed in the present invention shows significantly better performance than HVPNet. This performance improvement may be attributed to the fact that the model of the present invention uses more diverse image representations and corresponding multi-granularity text features, which provides additional semantic support for cross-modal tasks and effectively alleviates the problem of information missing in resource-constrained scenarios. The experimental results further verify the effectiveness of the method proposed in the present invention and its advantages in data efficiency.

[0173] Example 2

[0174] The present invention also provides a multimodal entity and relationship extraction system based on cross-modal alignment and fusion, the system being used to implement any one of the methods described above, the system comprising: a diversified multimodal feature acquisition module, a semantic anchor contrast learning module, a multi-granularity visual-text collaborative fusion module, an entity-guided attention gating module, and a task-specific decoder module;

[0175] The diversified multimodal feature acquisition module is used to process and encode the input text and paired images to obtain multiple image and text features. The multiple image and text features include fine-grained text features, coarse-grained text features, fine-grained pixel-level image representation, coarse-grained pixel-level image representation, fine-grained semantic-level image representation, and coarse-grained semantic-level image representation.

[0176] The semantic anchor contrast learning module is used to use the semantic representation of the image as the anchor point for aligning image and text features. It aligns fine-grained text features with fine-grained pixel-level image representations, and coarse-grained text features with coarse-grained pixel-level image representations, mapping image and text features into the same semantic space.

[0177] The multi-granularity visual-text collaborative fusion module is used to perform multi-granularity feature fusion through text-guided dynamic gating aggregation, visual prefix cross-modal fusion, and cross-modal image-text matching. This increases feature complementarity while modeling the association between noun phrases in the text and image objects, thereby obtaining fine-grained and coarse-grained multimodal feature representations.

[0178] The entity-guided attention gating module is used to input the obtained fine-grained and coarse-grained multimodal feature representations into the entity-guided attention gating module, aggregate visual information related to text entities, and obtain the final multimodal fusion representation;

[0179] The task-specific decoder module is used to input the obtained multimodal fusion representation into the task-specific decoder to perform task predictions of multimodal named entity recognition and multimodal relation extraction respectively.

[0180] The embodiments described above are merely descriptions of preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by persons skilled in the art should fall within the scope of protection defined by the claims of the present invention.

Claims

1. A multimodal entity and relationship extraction method based on cross-modal alignment and fusion, characterized by: The method comprises: Step 1: Process and encode the input text and image to obtain multiple image and text features; the multiple image and text features include: fine-grained text features, coarse-grained text features, fine-grained pixel-level image representation, coarse-grained pixel-level image representation, fine-grained semantic-level image representation, and coarse-grained semantic-level image representation; Step 2: Use the semantic representation of the image as an anchor point for aligning image and text features. Perform feature alignment between fine-grained text features and fine-grained pixel-level image representation, and between coarse-grained text features and coarse-grained pixel-level image representation, mapping image and text features into the same semantic space. Step 3: Multi-granularity feature fusion is performed through text-guided dynamic gating aggregation, visual prefix cross-modal fusion, and cross-modal image-text matching. This increases feature complementarity while modeling the association between noun phrases in the text and image objects, obtaining fine-grained and coarse-grained multimodal feature representations. Step 4: Input the obtained multi-granular multimodal feature representation into the entity-guided attention gating module to aggregate the visual information related to the text entities and obtain the final multimodal fusion representation; Step 5: Input the obtained multimodal fusion representation into the task-specific decoder to perform task predictions for multimodal named entity recognition and multimodal relation extraction respectively; In step 3, multi-granularity feature fusion is performed through text-guided dynamic gating aggregation, visual prefix cross-modal fusion, and cross-modal image-text matching to obtain fine-grained and coarse-grained multi-modal feature representations, including: The coarse-grained and fine-grained text representations, image pixel representations, and image semantic representations after feature alignment are processed separately; including: Representing fine-grained text And image representation and Perform fine-grained feature fusion based on visual prefix transformer blocks; Perform text-guided dynamic gating aggregation to obtain complementary image features of text at a specific level; the dynamic gating signal represents the probability of the k-th layer visual features obtained from the image encoder to the l-th fine-grained visual prefix transformer block: ； ； Where k and l represent the layer indices of the image encoder and visual prefix transformer blocks, respectively, and K represents the maximum number of layers of the image encoder. represents the fusion representation of the k-th layer visual features and fine-grained text features, is a learnable weight parameter, is the dynamic gating signal of the lth layer, Represents the representation of the global text features output by the previous layer of visual prefix transformer, represents the k-th layer of fine-grained pixel-level visual features, Indicates a cascade operation; Using the gate signal of each layer Get the final aggregated fine-grained hierarchical visual features ; Project the aggregated fine-grained hierarchical visual features onto the visual key vector and visual value vector middle ,in is the visual attention mapping parameter, which encodes the fine-grained text features of the l-1 layer Projection to cross-attention text query vector , key vector and value vector middle, ; Perform visual prefix cross-modal fusion, where They are the attention mapping parameters; the implicit features of the lth layer are calculated by visual prefix attention , iteratively updated layer by layer, the hidden features of the last layer are the cross-modal fine-grained text feature encoding representation; Cross-modal image-text matching is performed based on the similarity matrix obtained by semantic anchor comparison learning; Set the diagonal elements of the matrix to 0 to avoid comparison with positive samples; then, sample negative text and negative images from the same batch according to the similarity distribution, and assign sampling probabilities based on their similarity with the target image or text; Use the last layer output of the visual prefix transformer block to embed the [CLS] token as the fusion representation of the image-text pair and predict the probability of image and text matching. , get the image-text matching loss ,in is a two-dimensional one-hot vector representing the true value label; Text representation Paired image representation and The matching probability prediction of this module is used to obtain fine-grained multimodal features. : ; where L represents the maximum number of layers of the fine-grained visual prefix transformer block, and Represents the probability of image and text matching, which is used to determine the fine-grained multimodal features after the fusion of the retained fine-grained text features and the fine-grained pixel-level visual features. Fine-grained multimodal features after fusion of fine-grained text features and fine-grained semantic representations proportion; Perform fine-grained text representation And image representation and The coarse-grained feature fusion of .

2. The method according to claim 1, characterized in that In step 1, the input text and paired image are processed and encoded to obtain a variety of image and text features including: Acquire sample data, wherein the sample data includes original text and original image data paired with the original text; Based on the sample data, five different encoders are used to obtain diverse image representations and corresponding multi-granularity text features. The five different encoders include a complete text encoder, a noun phrase encoder, a complete image encoder, a local object encoder, and an image description and object label encoder.

3. The method according to claim 2, characterized in that Based on the sample data, five different encoders are used to obtain diverse image representations and corresponding multi-granularity text features, including: Use the pre-trained language model Bert to initialize the full text encoder and encode the original text to obtain fine-grained text features. Use the Stanford CoreNLP tool to extract noun phrases from the original sentence text and input them into the noun phrase encoder initialized by the pre-trained language model Bert to obtain coarse-grained text features. The full image encoder is initialized using the pre-trained Vision Transformer model and encoded into the original image to obtain fine-grained pixel-level visual features containing 12 layers of visual features. The visual grounding toolkit is used to detect objects using noun phrases extracted from the original text using the Stanford CoreNLP tool. The top N most relevant local objects are extracted from each original image and input into the local object encoder initialized by the pre-trained Vision Transformer model to obtain coarse-grained pixel-level visual features containing 12 layers of visual features. The original image is input into the ViT-GPT2 model to obtain an image description representing the global information of the image. The image description is then regarded as plain text and encoded using the image description and object label encoder initialized by the pre-trained language model Bert to obtain a fine-grained semantic-level image representation of the image. The object detector Faster-RCNN is used to detect objects in the image, and then the object labels are encoded using the image description and object label encoder to obtain a coarse-grained semantic-level image representation of the image.

4. The method according to claim 3, characterized in that In the second step, the semantic representation of the image is used as an anchor point for aligning the image and text features, and feature alignment of fine-grained text features and fine-grained pixel-level image representation, as well as coarse-grained text features and coarse-grained pixel-level image representation, is performed. Mapping the image and text features to the same semantic space includes: The obtained coarse-grained and fine-grained text representations, image pixel representations, and image semantic representations are processed separately, including: Perform feature alignment of text and image semantic representations. , where B represents the batch size, p,q represents the sample index label, and it is assumed that the positive example is the representation of B pairs from the same input pair , other representations from different input pairs are considered negative examples; Using two multi-layer perceptron layers and The global representation embedding [CLS] performs nonlinear mapping to obtain the low-dimensional embedding representation after mapping , where G represents the global feature after nonlinear mapping, d represents the dimension of the embedding space, and the global representation embedding [CLS] comes from the last layer output of the image encoder and the text encoder; Using cosine similarity To calculate the similarity between text and image semantic representation, the similarity between text representation and image semantic representation within a batch Defined as ; in is a learnable temperature parameter; similarly, the similarity between the semantic representation of the image and the text representation within a batch Defined as ; make and Representing the true one-hot similarity vector, maximizing the similarity of positive examples and minimizing the similarity of negative examples, the feature alignment contrast loss of text and image semantic representation is defined as the binary cross entropy H between p and y: ; Perform feature alignment between image pixel representation and image semantic representation. , where B represents the batch size, p,q represents the sample index label, and it is assumed that the positive example is the representation of B pairs from the same input pair , other representations from different input pairs are considered negative examples; Using two multi-layer perceptron layers and The global representation embedding [CLS] performs nonlinear mapping to obtain the low-dimensional embedding representation after mapping , where G represents the global feature after nonlinear mapping, d represents the dimension of the embedding space, and the global representation embedding [CLS] comes from the last layer output of the image encoder and the text encoder; Using cosine similarity To calculate the similarity between the image pixel representation and the image semantic representation, the similarity between the image pixel representation and the image semantic representation within a batch Defined as ; in is a learnable temperature parameter. Similarly, the similarity between the semantic representation of an image and the pixel representation of an image within a batch is Defined as ; make and Representing the true one-hot similarity vector, maximizing the similarity of positive examples and minimizing the similarity of negative examples, the feature alignment contrast loss of image pixel representation and image semantic representation is defined as the binary cross entropy H between p and y: ; The final semantic anchor contrast loss is defined as: 。 5. The method according to claim 1, wherein In step 4, the obtained fine-grained and coarse-grained multimodal feature representations are input into the entity-guided attention gating module to obtain the final multimodal fusion representation, which includes: Multimodal features and As input to attention gating, obtain fine-grained entity representation As query Q, coarse-grained noun phrase-object features as keys K and values ​​V, It is used to focus on relevant image object features and generate entity-aware object representations: ; Using Gate Signal Controlling entity-aware object representation The contribution of , to obtain the final multimodal fusion representation: .

6. The method according to claim 5, characterized in that In step 5, the obtained multimodal fusion representation is input into a task-specific decoder to perform task predictions for multimodal named entity recognition and multimodal relationship extraction, respectively, including: Based on the obtained multimodal fusion representation E, named entity recognition and relationship classification are performed respectively; including: The multimodal fusion representation is fed into the CRF layer, and for the obtained label sequence y, the cross entropy loss function is used to calculate the multimodal named entity recognition task loss: ; Where n is the length of the sequence, and i represents the index number of the sequence; The multimodal representation is input into the multilayer perceptron, and the relationship classification is performed through the softmax classifier; the cross entropy loss function is used to calculate the multimodal relationship extraction task loss: ; Among them, r represents the real relationship category; Based on the multimodal named entity recognition task loss and the multimodal relationship extraction task loss, the final loss function is obtained ,in, Represents the loss of the NER or RE task, and the final loss function is used to iteratively optimize the model.

7. A multimodal entity and relationship extraction system based on cross-modal alignment and fusion, the system being used to implement the method according to any one of claims 1 to 6, characterized in that: The system includes: a diversified multimodal feature acquisition module, a semantic anchor contrast learning module, a multi-granularity visual text collaborative fusion module, an entity-guided attention gating module, and a task-specific decoder module; The diversified multimodal feature acquisition module is used to process and encode input text and images to obtain multiple image and text features; wherein the multiple image and text features include: fine-grained text features, coarse-grained text features, fine-grained pixel-level image representation, coarse-grained pixel-level image representation, fine-grained semantic-level image representation, and coarse-grained semantic-level image representation; The semantic anchor contrast learning module is used to use the semantic representation of the image as the anchor point for aligning image and text features, and to align fine-grained text features with fine-grained pixel-level image representations, as well as coarse-grained text features with coarse-grained pixel-level image representations, to map image and text features into the same semantic space; The multi-granularity visual text collaborative fusion module is used to perform multi-granularity feature fusion through text-guided dynamic gating aggregation, visual prefix cross-modal fusion, and cross-modal image-text matching, thereby increasing feature complementarity while modeling the association between noun phrases in the text and image objects, and obtaining fine-grained and coarse-grained multimodal feature representations; The entity-guided attention gating module is used to input the obtained multi-granularity multimodal feature representation into the entity-guided attention gating module, aggregate visual information related to the text entity, and obtain the final multimodal fusion representation; The task-specific decoder module is used to input the obtained multimodal fusion representation into the task-specific decoder to perform task predictions of multimodal named entity recognition and multimodal relationship extraction respectively.

Images