Multi-modal entity and relation extraction method and system based on cross-modal alignment and fusion

Abstract

The invention discloses a multi-modal entity and relation extraction method and system based on cross-modal alignment and fusion, and the method comprises the steps: carrying out the processing and coding of an input text and an image, and obtaining a plurality of types of image and text features; performing feature alignment on the fine-grained and coarse-grained text features and the pixel-level image representation by taking the semantic representation of the image as an anchor point, and mapping the image and the text features to the same semantic space; performing multi-granularity feature fusion through text-guided dynamic gating aggregation, visual prefix cross-modal fusion and cross-modal image-text matching, modeling association between noun phrases and image objects in a text while increasing feature complementarity, and obtaining multi-granularity multi-modal feature representation; fusing multi-granularity multi-modal features through entity guidance attention gating, and gathering visual information related to a text entity to obtain final multi-modal fusion representation; according to the multi-modal fusion representation, task prediction of multi-modal named entity recognition and multi-modal relation extraction is carried out.

Description

Technical Field

[0001] The present invention belongs to the field of natural language processing in computer intelligent information processing, and particularly relates to a multi-modal entity and relationship extraction method and system based on cross-modal alignment and fusion. Background Art

[0002] As an important task in the field of natural language processing, named entity recognition can identify entity objects with specific meanings from text and classify them into predefined entity types. Relationship extraction aims to identify and extract relationships between entities from text. However, with the rise of multi-modal data, researchers have begun to realize that relying solely on text data for information extraction cannot meet the need for a comprehensive understanding of information. Therefore, they have started to integrate information from different modalities into the original tasks to obtain more accurate and comprehensive information. In recent years, multi-modal named entity recognition (MNER) and multi-modal relationship extraction (MRE) have gradually become a trend. The aim is to introduce visual content as an additional input to achieve the exchange and conversion of semantic information between different modalities, and then accurately extract the information contained in the target modality.

[0003] The main challenge faced by MNER and MRE is how to incorporate image information into the text representation learning process to improve the NER and RE tasks. So far, researchers have explored various visual representation methods.

[0004] A commonly used method is to utilize scene graphs, which provide structured semantic information about images. As the first work on MRE, Zheng et al. proposed an efficient graph alignment model that aligns the text scene graph generated from the syntactic dependency tree and the visual scene graph both structurally and semantically, making better use of the correlation between visual objects and text entities, thereby enhancing the performance of the multi-modal relationship extraction task. Research shows that the introduction of multi-modal information complements the missing semantics in social media text and improves the performance of relationship extraction. However, this method ignores the entity-entity relationships and object-object relationships in the scene graph. Therefore, Yuan et al. proposed an edge-enhanced graph alignment network to make full use of the edge information in the scene graph to improve the matching accuracy of nodes. In addition, Wu et al. first constructed visual and text scene graphs containing object nodes, attribute nodes, and relationship nodes, then fused the visual and text scene graphs to obtain a cross-modal graph, and finally refined the structure under the guidance of the graph information bottleneck principle to filter out features with less information and highlight features related to the task. However, relying too heavily on the scene graph model may result in the loss of image information. At the same time, Li et al. proved through visual shuffle experiments that the coarse-grained alignment of scene graphs cannot effectively utilize visual information.

[0005] Some other researchers incorporate object or region-level features detected from input images into text-based Transformer architectures to mitigate the noise of irrelevant visual elements while reducing modality heterogeneity. For example, Wang et al. use the semantic representation of an image that directly translates the image into the text space as visual context, which is concatenated with the original text as the input to the pre-trained text embedding model. Chen et al. adopt a multi-level pixel-level representation, combining visual information of the complete image and local objects, and use it as the prefix for each self-attention layer of the pre-trained text embedding model. Recently, Liu et al. introduce a hierarchical tracking graph to explore the intrinsic mechanism of integrating pixel-level image features into text semantic representations.

[0006] However, different types of image features provide different perspectives and information for the same activity or phenomenon, and existing work fails to comprehensively utilize all these different image representations to facilitate model decision-making.

[0007] Integrating all these different visual representations is undoubtedly challenging because it inevitably introduces noisy data. In terms of modality matching, existing models directly introduce all image and object information. However, unhelpful visual content and incorrect alignment methods are likely to provide misleading information when predicting relationships or entity representations, misleading the model to make wrong judgments.

[0008] At the same time, in terms of modality alignment, due to the semantic representation of images and pixel-level representations, as well as the original images and texts coming from two different semantic spaces of language and vision respectively, there is a significant modality gap in their feature distributions and semantic representations. This inconsistency significantly increases the difficulty for the model to capture cross-modal semantic correlations. Existing methods attempt to directly introduce the image-text contrast loss in large pre-trained models. However, this contrast learning method of direct interaction between image and text features lacks explicit alignment information and no explicitly marked alignment, resulting in sub-optimal image and text features obtained. Summary of the Invention

[0009] To solve the problems existing in the prior art, the present invention provides a multi-modal entity and relationship extraction method and system based on cross-modal alignment and fusion, which simultaneously uses the pixel-level representation and semantic representation of images to facilitate multi-modal named entity recognition (MNER) and multi-modal relationship extraction (MRE) tasks to provide rich context information.

[0010] To achieve the above object, the present invention provides the following solutions:

[0011] A multi-modal entity and relationship extraction method based on cross-modal alignment and fusion, the method comprising:

[0012] Step 1: Process and encode the input text and images to obtain various image and text features; among them, the various image and text features include: fine-grained text features, coarse-grained text features, fine-grained pixel-level image representations, coarse-grained pixel-level image representations, fine-grained semantic-level image representations, and coarse-grained semantic-level image representations;

[0013] Step 2: Use the semantic representation of the image as the anchor point for aligning image and text features, respectively perform feature alignment between fine-grained text features and fine-grained pixel-level image representations, and between coarse-grained text features and coarse-grained pixel-level image representations, and map the image and text features to the same semantic space;

[0014] Step 3: Through text-guided dynamic gating aggregation, visual prefix cross-modal fusion, and cross-modal image-text matching, perform multi-granularity feature fusion, increase feature complementarity while modeling the association between noun phrases in the text and image objects, and obtain fine-grained and coarse-grained multi-modal feature representations;

[0015] Step 4: Input the obtained multi-granularity multi-modal feature representations into the entity-guided attention gating module to aggregate visual information related to text entities and obtain the final multi-modal fusion representation;

[0016] Step 5: Input the obtained multi-modal fusion representation into the task-specific decoder to perform task predictions for multi-modal named entity recognition and multi-modal relation extraction respectively.

[0017] Preferably, in the above Step 1, processing and encoding the input text and paired images to obtain various image and text features includes:

[0018] Obtain sample data, where the sample data contains the original text and the original image data paired with the original text;

[0019] According to the sample data, use five different encoders to obtain diverse image representations and corresponding multi-granularity text features, where the five different encoders include a complete text encoder, a noun phrase encoder, a complete image encoder, a local object encoder, and an image description and object label encoder.

[0020] Preferably, according to the sample data, using five different encoders to obtain diverse image representations and corresponding multi-granularity text features includes:

[0021] Initialize the complete text encoder with the pre-trained language model Bert, encode the original text to obtain fine-grained text features; use the Stanford CoreNLP tool to extract noun phrases from the original sentence text and input them into the noun phrase encoder initialized with the pre-trained language model Bert to obtain coarse-grained text features;

[0022] Initialize the full-image encoder using the pre-trained vision model Vision Transformer, and encode the original image to obtain fine-grained pixel-level visual features containing 12 layers of visual features. Use the noun phrases extracted from the original text by the Stanford CoreNLP tool, apply the visual grounding toolkit to detect objects, extract the top N most relevant local objects from each original image, and input them into the local object encoder initialized by the pre-trained vision model Vision Transformer to obtain coarse-grained pixel-level visual features containing 12 layers of visual features.

[0023] Input the original image into the ViT-GPT2 model to obtain an image description representing the global information of the image. Then, regard the image description as pure text and encode it using the image description and object label encoder initialized by the pre-trained language model Bert to obtain a fine-grained semantic-level image representation of the image. Use the object detector Faster-RCNN to detect objects in the image, and then use the image description and object label encoder to encode the object labels to obtain a coarse-grained semantic-level image representation of the image.

[0024] Preferably, in the second step, use the semantic representation of the image as the anchor point for aligning the image and text features, and perform feature alignment for the fine-grained text features and the fine-grained pixel-level image representation, as well as the coarse-grained text features and the coarse-grained pixel-level image representation. Mapping the image and text features to the same semantic space includes:

[0025] Process the obtained coarse-grained and fine-grained text representations, image pixel representations, and image semantic representations respectively, including:

[0026] Perform feature alignment of the text and image semantic representations. For a batch of input text representations and image semantic representation pairs where B represents the batch size, and p, q represent the sample index labels. Assume that the positive examples are B pairs of representations {T p , S q} p＝q from the same input pair, and the other representations from different input pairs are regarded as negative examples;

[0027] Use two multi-layer perceptron layers to perform non-linear mapping on the global representation embeddings [CLS] of T p and S q to obtain the mapped low-dimensional embedding representations where G represents the global features after non-linear mapping, d represents the dimension of the embedding space, and the global representation embedding [CLS] comes from the output of the last layer of the image encoder and the text encoder;

[0028] Using cosine similarity to calculate the similarity between the text and the image semantic representation, the similarity p from the text representation to the image semantic representation within a batch t2s is defined as

[0029] where τ is a learnable temperature parameter; similarly, the similarity p from the image semantic representation to the text representation within a batch s2t is defined as

[0030] Let y t2s (I) and y s2t (I) represent the true one - hot similarity vectors, maximizing the similarity of positive examples and minimizing the similarity of negative examples. The feature alignment contrast loss between the text and the image semantic representation is defined as the binary cross - entropy H between p and y: L ts = H(p t2s , y t2s (I)) + H(p s2t , y s2t (I));

[0031] Perform feature alignment between the image pixel representation and the image semantic representation. For a batch of input image pixel representation and image semantic representation pairs where B represents the batch size, p and q represent the sample index labels. Assume that the positive examples are B pairs of representations {I p , S q} p＝q from the same input pair, and other representations from different input pairs are regarded as negative examples;

[0032] Use two multi - layer perceptron layers to perform non - linear mapping on the global representation embeddings [CLS] of I p and S q to obtain the mapped low - dimensional embedding representations where G represents the global features after non - linear mapping, d represents the dimension of the embedding space, and the global representation embedding [CLS] comes from the output of the last layer of the image encoder and the text encoder;

[0033] Using cosine similarity to calculate the similarity between the image pixel representation and the image semantic representation, the similarity p from the image pixel representation to the image semantic representation within a batch i2s is defined as

[0034] where τ is a learnable temperature parameter. Similarly, the similarity p from the image semantic representation to the image pixel representation within a batch s2i is defined as

[0035] Let y i2s (I) and y s2i (I) represents a true - value one - hot similarity vector, maximizing the similarity of positive examples and minimizing the similarity of negative examples. The feature alignment contrast loss between the image pixel representation and the image semantic representation is defined as the binary cross - entropy H between p and y: L is = H(p i2s , y i2s (I)) + H(p s2i , y s2i (I));

[0036] The final semantic anchor contrast loss is defined as:

[0037]

[0038] Preferably, in the third step, through text - guided dynamic gating aggregation, visual prefix cross - modal fusion, and cross - modal image - text matching, multi - granularity feature fusion is performed to obtain fine - grained and coarse - grained multi - modal feature representations, including:

[0039] The coarse - grained and fine - grained text representations, image pixel representations, and image semantic representations after feature alignment are processed respectively; including:

[0040] The fine - grained text representation T f and the image representation I f and S f are subjected to fine - grained feature fusion based on the visual prefix transformer block;

[0041] Text - guided dynamic gating aggregation is performed to obtain complementary image features of the text at a specific level; the dynamic gating signal represents the probability from the k - th layer visual feature obtained from the image encoder to the l - th fine - grained visual prefix transformer block:

[0042]

[0043] where k and l represent the layer indices of the image encoder and the visual prefix transformer block respectively, K represents the maximum number of layers of the image encoder, represents the fusion representation of the k - th layer visual feature and the fine - grained text feature, W I is a learnable weight parameter, (α f ) l is the dynamic gating signal of the l - th layer, represents the representation of the global text feature output by the previous layer of the visual prefix transformer, represents the k - th layer fine - grained pixel - level visual feature, [,] represents the concatenation operation;

[0044] Using the door signal of each layer (α f ) l Obtain the finally aggregated fine-grained hierarchical visual features

[0045] Project the aggregated fine-grained hierarchical visual features into the visual key vector and the visual value vector in where is the visual attention mapping parameter, which encodes and represents the fine-grained text features of the l-1 layer Project into the cross-attention text query vector key vector and value vector in, Perform visual prefix cross-modal fusion, where are the attention mapping parameters respectively; calculate the hidden features of the l-th layer through visual prefix attention Iteratively update layer by layer, and the hidden features of the last layer are the cross-modal fine-grained text feature encoding representations;

[0046] Based on the similarity matrix obtained by semantic anchor contrast learning, perform cross-modal image-text matching;

[0047] Set the diagonal elements of the matrix to 0 to avoid comparison with positive samples; then, sample negative texts and negative images from the same batch according to the similarity distribution, and assign sampling probabilities based on their similarities to the target image or text;

[0048] Use the output embedding [CLS] token of the last layer of the visual prefix transformer block as the fusion representation of the image-text pair, and predict the probability p of image and text matching itm , obtain the image-text matching loss L itm =H(y itm ,p itm (T, I)), where y itm is a two-dimensional one-hot vector representing the ground truth label;

[0049] Respectively perform the matching probability prediction of the text representation T f and the paired image representation I f and S f , and use the matching probability output by this module to obtain the fine-grained multi-modal feature M f ∈R n×d : Among them, L represents the maximum number of layers of the fine-grained visual prefix transformer block, and represent the probabilities of image and text matching, which are used to determine the proportions of the fine-grained multimodal features after fusing the retained fine-grained text features and fine-grained pixel-level visual features and the fine-grained multimodal features after fusing the fine-grained text features and fine-grained semantic representations respectively;

[0050] Perform fine-grained text representation T c and image representation I c and S c to perform coarse-grained feature fusion to obtain coarse-grained multimodal feature M C .

[0051] Preferably, in step four, inputting the obtained fine-grained and coarse-grained multimodal feature representations into the entity-guided attention gating module, the steps to obtain the final multimodal fusion representation include:

[0052] Taking multimodal features M C and M f as the input of the attention gating, obtaining the fine-grained entity representation E f as the query Q, the coarse-grained noun phrase-object feature as the key K and value V, and E f is used to focus on relevant image object features to generate an entity-aware object representation: E c = Cross-att(Q = E f , K = M c , V = M c );

[0053] Using the gate signal g e = sigmoid(W e [M c , E f ) to control the contribution of the entity-aware object representation E C , and obtaining the final multimodal fusion representation: E = g e *E c + (1 - g e )*E f .

[0054] Preferably, in step five, inputting the obtained multimodal fusion representation into the task-specific decoder, the task predictions for multimodal named entity recognition and multimodal relation extraction respectively include:

[0055] Based on the obtained multi-modal fusion representation E, named entity recognition and relation classification are respectively performed, including: sending the multi-modal fusion representation into the CRF layer, and for the obtained tag sequence y, using the cross-entropy loss function to calculate the loss of the multi-modal named entity recognition task: where n is the sequence length, and i represents the index label of the sequence;

[0056] The multi-modal representation is input into a multi-layer perceptron, and through a softmax classifier, relation classification is performed; the cross-entropy loss function is used to calculate the loss of the multi-modal relation extraction task: where r represents the true relation category;

[0057] Based on the loss of the multi-modal named entity recognition task and the loss of the multi-modal relation extraction task, the obtained final loss function L = L ner / re +L SACL +L itm where L ner / re represents the loss of the NER or RE task, and the model is iteratively optimized using the final loss function.

[0058] The present invention also provides a multi-modal entity and relation extraction system based on cross-modal alignment and fusion. The system is used to implement any one of the above methods. The system includes: a diverse multi-modal feature acquisition module, a semantic anchor contrast learning module, a multi-granularity visual-text collaborative fusion module, an entity-guided attention gating module, and a task-specific decoder module;

[0059] The diverse multi-modal feature acquisition module is used to process and encode the input text and image to obtain various image and text features; among them, the various image and text features include: fine-grained text features, coarse-grained text features, fine-grained pixel-level image representations, coarse-grained pixel-level image representations, fine-grained semantic-level image representations, and coarse-grained semantic-level image representations;

[0060] The semantic anchor contrast learning module is used to use the semantic representation of the image as an anchor for aligning the image and text features, respectively perform feature alignment of the fine-grained text features and the fine-grained pixel-level image representations, and the coarse-grained text features and the coarse-grained pixel-level image representations, and map the image and text features to the same semantic space;

[0061] The multi-granularity visual-text collaborative fusion module is used to perform multi-granularity feature fusion through text-guided dynamic gating aggregation, visual prefix cross-modal fusion, and cross-modal image-text matching, model the association between noun phrases in the text and image objects while increasing feature complementarity, and obtain fine-grained and coarse-grained multi-modal feature representations;

[0062] The entity-guided attention gating module is used to input the obtained multi-granularity multi-modal feature representations into the entity-guided attention gating module, aggregate the visual information related to text entities, and obtain the final multi-modal fusion representation;

[0063] The task-specific decoder module is used to input the obtained multi-modal fusion representation into the task-specific decoder to perform task predictions for multi-modal named entity recognition and multi-modal relation extraction respectively.

[0064] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0065] The present invention uses five different encoders to obtain diverse image representations and corresponding multi-granularity text features. Different types of image features provide different perspectives and information for the same activity or phenomenon. The present invention comprehensively utilizes all different image features to provide rich context information to facilitate multi-modal named entity recognition and multi-modal relation extraction.

[0066] The present invention introduces the semantic representation of images as anchors to alleviate cross-modal contrastive learning. Compared with directly interacting between images and texts, introducing these anchor models can easily capture the representations of modality-invariant factors; in this way, before modality fusion, the image and text representations from different encoders are aligned to construct consistent representations to bridge the modality gap and promote cross-modal alignment.

[0067] The present invention adopts a multi-granularity vision-text collaborative fusion strategy for cross-modal fusion. It adjusts the visual information weight through text-guided dynamic gating, and takes the noun phrases in the text as candidate entities to model the association between text entities and image objects to improve cross-modal interaction; finally, entity-guided attention gating is introduced to retain the visual features consistent with the semantics of text entities, effectively alleviating the interference of modality noise on the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the technical solutions of the present invention, the accompanying drawings required for use in the embodiments are briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0069] Figure 1 It is the model architecture diagram provided by the embodiment of the present invention;

[0070] Figure 2 It is the schematic diagram of the semantic anchor contrastive learning module provided by the embodiment of the present invention;

[0071] Figure 3Schematic diagram of text-guided dynamic gating for the multi-granularity visual-text collaborative fusion module provided by an embodiment of the present invention;

[0072] Figure 4 Schematic diagram of visual prefix cross-modal fusion for the multi-granularity visual-text collaborative fusion module provided by an embodiment of the present invention;

[0073] Figure 5 Two example study diagrams provided by an embodiment of the present invention;

[0074] Figure 6 To remove the influence of different image features on the method proposed in the present invention, Figure 6 (a) Performance of the method on the Twitter2017 dataset, Figure 6 (b) Performance of the method on the MNRE dataset;

[0075] Figure 7 Visualization results of the modality gap between text and image compared with existing methods provided by an embodiment of the present invention, Figure 7 (a) Feature visualization results generated by the method HVPNet, Figure 7 (b) Feature visualization results generated by the method AMNet, Figure 7 (c) Feature visualization results generated by the method SAGE proposed in the present invention;

[0076] Figure 8 Low-resource setting performance of the proposed method on the MRE and MNER tasks provided by an embodiment of the present invention, Figure 8 (a) Performance of the method on the Twitter2017 dataset, Figure 8 (b) Performance of the method on the MNRE dataset. Detailed implementation manners

[0077] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0078] To make the above objects, features, and advantages of the present invention more obvious and understandable, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific implementation manners.

[0079] Embodiment 1

[0080] Existing multimodal entity and relationship extraction methods still face many problems in modal representation, modal alignment and modal matching, such as underutilization of diverse image representations, alignment difficulties caused by the modal gap, and information interference caused by modal noise. This paper proposes a multimodal entity and relationship extraction method based on cross-modal alignment and fusion, which uses the pixel-level representation and semantic representation of images to promote multimodal named entity recognition (MNER) and multimodal relation extraction (MRE) tasks to provide rich context information.

[0081] The present invention observes that objects in images can be accurately detected and are often mentioned in paired texts. Therefore, in response to the problem of modal alignment, the present invention proposes a semantic anchor contrast learning method for cross-modal alignment, which introduces the semantic representation of the image as an anchor for contrast learning of image and text features, maps the image and text features to the same space, and builds a consistent representation before modal fusion to bridge the modal gap. Compared with existing methods that directly interact between images and text, the introduction of these anchor models makes it easier to capture the representation of modality invariant factors.

[0082] Existing methods directly introduce all visual information and ignore the relevance of text entities to objects in the image, although they have the potential to improve cross-modal interactions. Therefore, in order to solve the modal matching problem, the present invention proposes a new cross-modal fusion method, which adopts a multi-granularity visual-text collaborative mechanism to capture both fine-grained and coarse-grained cross-modal associations to increase the complementarity between features. Specifically, the present invention designs text-guided dynamic gating, dynamically adjusts the weight of visual information according to the text content, obtains complementary image features of text at a specific level, and uses a visual prefix-based attention method for the fusion of visual and text features. Through a layer-by-layer iterative update method, the deep fusion of multimodal features is promoted. Finally, a cross-modal image-text matching loss is proposed to control the degree of integration of image information and reduce the influence of irrelevant visual features.

[0083] At the same time, the present invention takes noun phrases in the text as candidate entities and models their relevance to the target object in the image during the interaction process. Finally, the candidate entity is used as a bridge for interaction between text and image. Through entity-guided attention gating, all objects in the image are screened, and the key visual content is focused on information related to the text entity. By modeling the relevance between text candidate entities and target objects, cross-modal interaction is improved and modal noise is alleviated. The specific implementation process is:

[0084] like Figure 1 As shown, the embodiment of the present invention provides a multimodal entity and relationship extraction method based on cross-modal alignment and fusion, which includes the following operations:

[0085] Step 1: Use a diversified multi-modal feature acquisition module to process and encode the input text and paired images to obtain various image and text features, including fine-grained text features, coarse-grained text features, fine-grained pixel-level image representations, coarse-grained pixel-level image representations, fine-grained semantic-level image representations, and coarse-grained semantic-level image representations. Due to different forms, existing methods can only apply one of the pixel-level representation or semantic representation of images. However, different types of image features provide different perspectives and information for the same activity or phenomenon. The present invention comprehensively utilizes all different image representations to provide rich context information to facilitate multi-modal named entity recognition and multi-modal relationship extraction.

[0086] Step 2: The semantic representation and pixel-level representation of images, as well as the original images and texts, come from two different semantic spaces of language and vision respectively, and there is a significant modal gap in their feature distributions and semantic representations. This inconsistency significantly increases the difficulty for the model to capture cross-modal semantic correlations. However, most existing methods ignore the modal gap. The present invention proposes a semantic anchor contrast learning module for cross-modal alignment, using the semantic representation of images as the anchor point for aligning image and text features, and respectively performing feature alignment between fine-grained text features and fine-grained pixel-level image representations, as well as between coarse-grained text features and coarse-grained pixel-level image representations, to map image and text features to the same semantic space. Compared with directly interacting between images and texts, introducing these anchor models makes it easier to capture the representations of modal-invariant factors.

[0087] Step 3: Existing methods directly introduce all visual information, ignoring the correlation between text entities and objects in images, although they have the potential to improve cross-modal interaction. Therefore, the present invention proposes a new cross-modal fusion method, using a multi-granularity visual-text collaborative fusion module to perform multi-granularity feature fusion through text-guided dynamic gating aggregation, visual prefix cross-modal fusion, and cross-modal image-text matching, to obtain fine-grained and coarse-grained multi-modal feature representations. The present invention effectively improves cross-modal interaction and alleviates modal noise by modeling the correlation between text candidate entities and target objects.

[0088] Step 4: Input the fine-grained and coarse-grained multi-modal feature representations obtained in Step 3 into an entity-guided attention gating module to aggregate visual information related to text entities and obtain the final multi-modal fusion representation.

[0089] Step 5: Input the multi-modal fusion representation obtained in Step 4 into a task-specific decoder to perform task predictions for multi-modal named entity recognition and multi-modal relationship extraction respectively.

[0090] In this embodiment, the specific operations of the diversified multi-modal feature acquisition in Step 1 include the following:

[0091] Obtain sample data, where the sample data contains the original text and the original image data paired with the original text;

[0092] Use five different encoders to obtain diverse image representations and corresponding multi-granularity text features, including a complete text encoder, a noun phrase encoder, a complete image encoder, a local object encoder, and an image description and object label encoder;

[0093] Initialize the complete text encoder with the pre-trained language model Bert, encode the original text, and obtain fine-grained text features where d T is the dimension of each token; Use the Stanford CoreNLP tool to extract noun phrases from the original sentence text, and then input them into the noun phrase encoder initialized by the pre-trained language model Bert to obtain coarse-grained text features where f and c are used to represent fine-grained and coarse-grained respectively;

[0094] Initialize the complete image encoder with the pre-trained vision model Vision Transformer (Vit), and encode the original image to obtain fine-grained pixel-level visual features containing 12 layers of visual features: where represents the output embedding representation of the k-th layer of Vit, d v is the image feature dimension;

[0095] Use the noun phrases extracted from the original text by the Stanford CoreNLP tool, apply the visualgrounding toolkit to detect objects, extract the top N most relevant local objects from each original image, and input them into the local object encoder initialized by Vit to obtain coarse-grained pixel-level visual features containing 12 layers of visual features

[0096] Input the original image into the ViT-GPT2 model to obtain an image description representing the global information of the image, then regard the image description as pure text, and use the image description and object label encoder initialized by the pre-trained language model Bert to encode it to obtain a fine-grained semantic-level image representation of the image M c represents the text length of the obtained image description; Use the object detector Faster-RCNN to detect objects in the image, and then use the image description and object label encoder to encode the object labels to obtain a coarse-grained semantic-level image representation of the image M t represents the text length of the obtained concatenated object label text.

[0097] In this embodiment, the feature contrast learning between the image and the text in step two includes the following specific operations:

[0098] As Figure 2 shown, the obtained coarse-grained and fine-grained text representations, image pixel representations, and image semantic representations are processed respectively, including:

[0099] Align the features of the text and image semantic representations. For a batch of input text representations and image semantic representation pairs where B represents the batch size, and p, q represent the sample index labels. Assume that the positive examples are B pairs of representations {T p , S q} p＝q from the same input pair, and the other representations from different input pairs are regarded as negative examples;

[0100] Use two multi-layer perceptron layers to perform non-linear mapping on the global representation embeddings [CLS] of T p and S q to obtain the mapped low-dimensional embedding representations where G represents the global features after non-linear mapping, d represents the dimension of the embedding space, and the global representation embedding [CLS] comes from the output of the last layer of the image encoder and the text encoder;

[0101] Adopt cosine similarity to calculate the similarity between the text and image semantic representations. The similarity p of the text representation to the image semantic representation within a batch t2s is defined as

[0102] where τ is a learnable temperature parameter. Similarly, the similarity p of the image semantic representation to the text representation within a batch s2t is defined as

[0103] Let y t2s (I) and y s2t (I) represent the true one-hot similarity vectors. Maximize the similarity of positive examples and minimize the similarity of negative examples. The feature alignment contrast loss between the text and image semantic representations is defined as the binary cross-entropy H between p and y: L ts = H(p t2s , y t2s (I)) + H(p s2t , y s2t (I));

[0104] Use the same method to align the features of the image pixel representation and the image semantic representation. For a batch of input image pixel representation and image semantic representation pairs where B represents the batch size, and p, q represent the sample index labels. Assume that the positive examples are B pairs of representations from the same input pair {I p , S q}, p＝q and the other representations from different input pairs are regarded as negative examples;

[0105] Use two multi-layer perceptron layers to perform non-linear mapping on the global representation embedding [CLS] of I p and S q to obtain the mapped low-dimensional embedding representation where G represents the global features after non-linear mapping, d represents the dimension of the embedding space, and the global representation embedding [CLS] comes from the output of the last layer of the image encoder and the text encoder;

[0106] Adopt cosine similarity to calculate the similarity between the image pixel representation and the image semantic representation. The similarity p of the image pixel representation to the image semantic representation within a batch i2s is defined as

[0107] where τ is a learnable temperature parameter. Similarly, the similarity p of the image semantic representation to the image pixel representation within a batch s2i is defined as

[0108] Let y i2s (I) and y s2i (I) represent the true one-hot similarity vectors. Maximize the similarity of positive examples and minimize the similarity of negative examples. The feature alignment contrast loss between the image pixel representation and the image semantic representation is defined as the binary cross-entropy H between p and y: L is = H(p i2s , y i2s (I)) + H(p s2i , y s2i (I));

[0109] The final semantic anchor contrast loss is defined as the average of these two parts of losses:

[0110]

[0111] As Figure 3 shown, in this embodiment, the multi-granularity visual-text collaborative fusion in step three includes the following specific operations:

[0112] Process the coarsely-grained and finely-grained text representations, the image pixel representation, and the image semantic representation after feature alignment respectively; including:

[0113] Process the finely-grained text representation Tf and the image representation I f and S f perform fine-grained feature fusion based on a visual prefix Transformer block (consisting of several blocks);

[0114] perform text-guided dynamic gating aggregation to obtain complementary image features of the text at a specific level; the dynamic gating signal represents the probability from the k-th layer visual feature obtained from the image encoder to the l-th fine-grained visual prefix Transformer block:

[0115]

[0116] where k and l respectively represent the layer indices of the image encoder and the visual prefix Transformer block, and K represents the maximum number of layers of the image encoder, represents the fusion representation of the k-th layer visual feature and the fine-grained text feature, W I is a learnable weight parameter, (α f ) l is the dynamic gating signal of the l-th layer, represents the representation of the global text feature output by the previous layer of the visual prefix Transformer, and [,] represents the concatenation operation, represents the k-th layer fine-grained pixel-level visual feature, and [,] represents the concatenation operation.

[0117] Utilize the gate signal (α f ) l at each layer to obtain the finally aggregated fine-grained hierarchical visual features where is the gating weight corresponding to the k-th layer visual feature.

[0118] As Figure 4 shown, project the aggregated fine-grained hierarchical visual features into the visual key vector and the visual value vector in where is the visual attention mapping parameter. Encode and represent the fine-grained text feature of the (l - 1)-th layer project it into the cross-attention text query vector key vector and the value vector in, perform visual prefix cross-modal fusion, where are the attention mapping parameters respectively; calculate the implicit feature of the l-th layer through visual prefix attention Iteratively update layer by layer, and the hidden features of the last layer are the cross-modal fine-grained text feature encoding representations;

[0119] Similarity p obtained based on semantic anchor contrast learning t2s and p s2t , construct a similarity matrix for cross-modal image-text matching. Among them, the diagonal elements represent the matching text and semantic representation pairs, while other elements represent the non-matching text and semantic representation pairs;

[0120] Set the diagonal elements of the matrix to 0 to avoid comparison with positive samples; then, sample negative texts and negative images from the same batch according to the similarity distribution, where samples with higher similarity to the target image or text are given higher sampling probabilities;

[0121] Use the output embedding [CLS] token of the last layer of the visual prefix transformer block as the fused representation of the image-text pair to predict the probability p itm of the image and text matching, and obtain the image-text matching loss L itm =H(y itm ,p itm (T, I)), where y itm is a two-dimensional one-hot vector representing the ground truth label;

[0122] Perform probability predictions for the text representation T f and its paired image representation I f and S f respectively. Using the matching probabilities output by this module, obtain the fine-grained multi-modal feature M f ∈R n×d : where L represents the maximum number of layers of the fine-grained visual prefix transformer block, and represent the probabilities of image and text matching, respectively used to determine the proportions of the fine-grained multi-modal features after fusing the fine-grained text features and the fine-grained pixel-level visual features and the fine-grained multi-modal features after fusing the fine-grained text features and the fine-grained semantic representations ;

[0123] Perform coarse-grained feature fusion on the coarse-grained text representation T c as well as the image representation I c and S c in the same way, and obtain the coarse-grained multi-modal feature M C .

[0124] Perform text-guided dynamic gated aggregation to obtain complementary image features of the text at a specific level; the dynamic gating signal represents the probability from the k-th layer visual feature obtained from the image encoder to the l-th coarse-grained visual prefix transformer block:

[0125]

[0126] where k and l represent the layer indices of the image encoder and the visual prefix transformer block respectively, and K represents the maximum number of layers of the image encoder. represents the fused representation of the k-th layer visual feature and the coarse-grained text feature, and W I is a learnable weight parameter, and (α c ) l is the dynamic gating signal of the l-th layer. represents the representation of the global text feature output by the previous layer visual prefix transformer, and [,] represents the concatenation operation. represents the k-th layer coarse-grained pixel-level visual feature, and [,] represents the concatenation operation.

[0127] Utilize the gating signal (α c ) l of each layer to obtain the finally aggregated coarse-grained hierarchical visual feature where is the gating weight corresponding to the k-th layer visual feature.

[0128] As Figure 4 shown, project the aggregated fine-grained hierarchical visual feature into the visual key vector and the visual value vector in. where is the visual attention mapping parameter. Encode and represent the coarse-grained text feature of the (l - 1)-th layer and project it into the cross-attention text query vector key vector and value vector in. Perform visual prefix cross-modal fusion, where are the attention mapping parameters respectively; calculate the hidden feature of the l-th layer layer by layer through visual prefix attention and update iteratively. The hidden feature of the last layer is the cross-modal coarse-grained text feature encoding representation.

[0129] The similarities p t2s and p s2t, construct a similarity matrix for cross-modal image-text matching, where the diagonal elements represent the matching text and semantic representation pairs, while the other elements represent the non-matching text and semantic representation pairs;

[0130] Set the diagonal elements of the matrix to 0 to avoid comparison with positive samples; then, sample negative texts and negative images from the same batch according to the similarity distribution, where samples with higher similarity to the target image or text are given higher sampling probabilities;

[0131] Use the output embedding [CLS] token of the last layer of the visual prefix transformer block as the fused representation of the image-text pair to predict the probability p of image and text matching itm , obtain the image-text matching loss L itm = H(y itm , p itm (T, I)), where y itm is a two-dimensional one-hot vector representing the ground truth label;

[0132] Predict the matching probabilities of the text representation T c and its paired image representation I c and S c respectively. Using the matching probabilities output by this module, obtain the fine-grained multi-modal feature M c ∈ r n×d : where L represents the maximum number of layers of the fine-grained visual prefix transformer block, and represent the probabilities of image and text matching, which are used to determine the proportions of the coarse-grained multi-modal features after fusing the coarse-grained text features and the coarse-grained pixel-level visual features and the coarse-grained multi-modal features after fusing the coarse-grained text features and the coarse-grained semantic representations respectively;

[0133] In this embodiment, obtaining the final multi-modal fused representation in step four includes the following specific operations: Using the multi-modal features M C and M f as the input of the attention gate, obtain the fine-grained entity representation E f as the query Q, the coarse-grained noun phrase-object feature as the key K and the value V. E f is used to attend to the relevant image object features to generate the entity-aware object representation: E c = Cross-att(Q = E f , K = M c , V = M c )

[0134] Use the gate signal g e = sigmoid(W e [M c , E f ) to control the contribution of the object representation E perceived by the entity, where W C represents the trainable parameter, and obtain the final multi-modal fusion representation E: e E = g

[0135] * E e + (1 - g c ) * E e f

[0136] In this embodiment, the task-specific decoding in step five includes the following specific operations:

[0137] Based on the obtained multi-modal fusion representation E, perform named entity recognition and relation classification respectively; including:

[0138] Send the multi-modal fusion representation into the CRF layer. For the obtained label sequence y, use the cross-entropy loss function to calculate the multi-modal named entity recognition task loss: where n is the sequence length and i represents the index label of the sequence;

[0139] Input the multi-modal representation into a multi-layer perceptron, and perform relation classification through a softmax classifier; use the cross-entropy loss function to calculate the multi-modal relation extraction task loss: where r represents the true relation category;

[0140] Based on the above loss functions, the final loss function L = L ner / re + L SACL + L itm where L ner / re represents the loss of the NER or RE task (selected according to the specific task), and use the final loss function to iteratively optimize the model.

[0141] Figure 5 Finally, the present invention proves the effectiveness and practical application value of the proposed method SAGE through Figure 5 the two instances for multi-modal relation extraction in; where the node represents the text candidate entity and the target object in the image, and the subject and object are underlined. Define the relevance according to the attention score of the last layer of the coarse-grained feature fusion Transformer block in the multi-granularity visual-text collaborative fusion module.

[0142] ​In the example on the left, AMNet was misled by the information of the three people in the image and predicted a relationship between people. The method proposed by the present invention captured that although the object was related to people, the subject was weakly related to people, thus successfully predicting the "none" type. In the example on the right, the text-based BERT model could not capture the clues contained in the image, and the method proposed by the present invention obtained the correct prediction by establishing an effective link between the candidate entity and the image target object.

[0143] To prove the effectiveness of the multi-modal entity and relationship extraction method SAGE proposed by the present invention, the following public datasets were used to verify the method: Twitter2015 and Twitter2017 were used for the multi-modal named entity recognition (MNER) task. The MNRE dataset was used for the multi-modal relationship extraction (MRE) task. Precision, Recall, and F1-score were used as the main evaluation metrics.

[0144] A comprehensive comparison was made with currently advanced and representative models on the standard dataset. The method SAGE of the present invention was evaluated with three groups of models. Text-based models: Consider the following representative models, including Bert-CRF and MTB. LLM-based models: Two representative LLM baselines were selected, including ChatGPT and GPT4. Multi-modal models: Considered the multi-modal models previously used for the MNER and MRE tasks, including EEGA, IFAformer, HVPNet, MMIB, HVFormer, VisualPT-MoE, AMNet, DebiasCL, MGCMT, GBIT.

[0145] Table 1 MRE experimental results

[0146]

[0147] Table 2 MNER experimental results

[0148]

[0149] It can be observed from Table 1 and Table 2 that:

[0150] First of all, all multi-modal models perform better than text-based models. This shows that enhancing text semantics using visual features can indeed improve the performance of the model in NER and MRE tasks, which is particularly evident in the MRE task. However, for the MNER task, the introduction of visual information still has limited improvement on the model performance, which requires further exploration of multi-modal methods.

[0151] Secondly, compared with other multimodal methods, methods that can bridge the modality gap and alleviate modality noise, such as AMNet and MMIB, bring more performance gains to multimodal methods. This illustrates the importance of consistent image-text representations and modeling cross-modal associations. Nevertheless, the final performance of the two tasks still fails to meet expectations, indicating the need for more effective strategies to address these two issues.

[0152] Finally, the method proposed in the present invention has significantly improved performance in various performance metrics compared with previous models. Compared with the previous best-performing model, the method proposed in the present invention achieved F1-score performance gains of 3.02%, 0.30%, and 0.49% on three datasets respectively, demonstrating the effectiveness of the model in multimodal entity and relation extraction tasks.

[0153] To verify the effectiveness of each component in the present invention, experiments were conducted on variants of the model using the same experimental parameters. The experimental results are shown in Tables 3 and 4.

[0154] Table 3 Ablation experiment results for the MRE task

[0155]

[0156] Table 4 Ablation experiment results for the MNER task

[0157]

[0158]

[0159] w / o HVF: In this ablation experiment, the present invention only utilizes the highest-level features of the pixel-level representation of the image. As the present invention can see from the results, this leads to a performance decline, indicating that directly using the high-level image embeddings of the visual encoder as a prior method is suboptimal for multimodal entity and relation extraction. The multi-level pixel-level image representation provides rich context information for the task.

[0160] w / o SACL: In this ablation experiment, the present invention replaces semantic anchor contrast learning with the image-text contrastive loss (ITC) commonly used in large pre-trained models. Since this contrastive learning method for direct interaction between image and text features lacks explicit alignment information and no explicitly marked alignment, it leads to a decline in the model's performance.

[0161] The present invention further explores the roles of image captions and object tags in an image in the alignment of image and text features. "w / o SACL caption" and "w / o SACL tag" respectively indicate that only coarse-grained semantic-level image representations and fine-grained semantic-level image representations are used as semantic anchors for the alignment of image and text features. It can be seen from the results that both of these features are indispensable. They describe the image from different granularities and play important roles in the alignment of text and image features.

[0162] w / o EG: The present invention removes the entity-guided attention gating and directly splices the features after multi-granularity text-image fusion for model prediction. The present invention observes that the F1 values of the model performance decrease by 1.25%, 0.19%, and 0.30 respectively on three datasets, indicating that establishing an effective link between entities in the text and target objects in the image has the potential to improve cross-modal interaction.

[0163] To further verify the effectiveness and plug-and-play nature of the text-guided dynamic gating module in the method proposed by the present invention, this module is integrated into an existing model. The RSRNet model uses a ResNeSt network to obtain pyramid visual features. Similar to the HVPnet, it also uses a learnable gating controlled by a single image modality to determine the effectiveness of the image. Specifically:

[0164] AMNet believes that the gate signals obtained by this learnable neural network introduce noise and conducts experiments by using the first 6 layers and the last 6 layers of hierarchical visual features of the global image features and object image features combined in sequence as visual prefixes respectively. However, this approach abandons the comprehensive utilization of multi-level visual features.

[0165] The text-guided dynamic gating aggregation module proposed by the present invention is combined with the above models respectively. Taking the multi-modal relation extraction (MRE) task as an example, the experimental results are shown in Table 5. Compared with the original models, the F1 values of the models integrated with the text-guided dynamic gating module proposed by the present invention are increased by 1.33%, 0.63%, and 0.51% respectively.

[0166] Table 5 Experimental results of module verification

[0167]

[0168] To explore the effectiveness of image representations in different tasks, the present invention controls the image representations utilized in multi-granularity text-image feature fusion. Figure 6 It shows the importance of different image features for the multi-modal named entity recognition (MNER) and multi-modal relation extraction (MRE) tasks, taking the Twitter2017 and MNRE datasets as examples respectively.

[0169] First, it can be observed from the results that deleting any kind of image feature will lead to a performance decline. Diverse image features are complementary in multimodal tasks, providing the model with diverse information sources and observation perspectives. Second, for multimodal named entity recognition (MNER) and multimodal relation extraction (MRE) tasks, pixel-level representations play a more significant role than semantic representations. Multilevel pixel-level representations can capture richer visual information, thus providing more comprehensive support for the tasks. In addition, directly translating images into text may result in the loss of some information, thereby limiting the model's full understanding of the image content. Finally, the requirements for image information in different tasks are task-specific. Specifically, the MNER task pays more attention to the global image representation. In contrast, the MRE task relies more on local object representations.

[0170] To verify the effectiveness of the semantic anchor contrast learning proposed in the present invention in alleviating the modality gap, the present invention conducts a visual analysis of text and image features and compares them with other models. First, the present invention randomly selects 50 test samples from the MNRE dataset, obtains the text-image pair features output by the trained SAGE method, and maps them to a two-dimensional space using the t-SNE tool.

[0171] Figure 7 Shows the visual comparison results of the SAGE method proposed in the present invention with the HVPNet and AMNet models. It can be seen from the figure that: HVPNet regards the visual representation as an insertable visual prefix and inserts it into the text encoder, reducing the interference to the architecture representing the text. However, the visual results show that the modality gap still exists. AMNet introduces the image-text contrast loss commonly used in large visual language pre-training models. However, this contrast learning method of direct interaction between image and text features lacks explicit marker alignment and cannot obtain sufficient training in downstream tasks. Therefore, although it alleviates the modality gap to a certain extent, the results are not ideal, and there are still certain differences in the feature distributions. In contrast, SAGE performs well in aligning the features of text and images, and the feature distributions are more concentrated. The modality gap is effectively bridged by introducing semantic anchor contrast learning.

[0172] The present invention conducts experiments in resource-constrained scenarios, randomly extracting 5% to 50% from the original training set. The experimental results are as Figure 8As shown. From the figure, the present invention can observe that: 1) In low-resource scenarios, the performance of SAGE and HVPNet is always better than the compared baselines; this result indicates that using multi-level image pixel-level representations (including the complete image and the target object) has a significant promoting effect on NER and RE tasks. 2) In addition, the SAGE method proposed by the present invention shows significantly better performance than HVPNet. This performance improvement may be attributed to the fact that the model of the present invention uses more diverse image representations and corresponding multi-granularity text features, providing additional semantic support for cross-modal tasks and effectively alleviating the problem of information loss in resource-scarce scenarios. The experimental results further verify the effectiveness of the method proposed by the present invention and its advantages in data efficiency.

[0173] Embodiment 2

[0174] The present invention also provides a multi-modal entity and relationship extraction system based on cross-modal alignment and fusion, which is used to implement the method described in any one of the above, and the system includes: a diverse multi-modal feature acquisition module, a semantic anchor contrast learning module, a multi-granularity visual-text collaborative fusion module, an entity-guided attention gating module, and a task-specific decoder module;

[0175] The diverse multi-modal feature acquisition module is used to process and encode the input text and paired images to obtain various image and text features; among them, the various image and text features include: fine-grained text features, coarse-grained text features, fine-grained pixel-level image representations, coarse-grained pixel-level image representations, fine-grained semantic-level image representations, and coarse-grained semantic-level image representations;

[0176] The semantic anchor contrast learning module is used to use the semantic representation of the image as the anchor for aligning the image and text features, respectively perform feature alignment of the fine-grained text features and the fine-grained pixel-level image representations, and the coarse-grained text features and the coarse-grained pixel-level image representations, and map the image and text features to the same semantic space;

[0177] The multi-granularity visual-text collaborative fusion module is used to perform multi-granularity feature fusion through text-guided dynamic gating aggregation, visual prefix cross-modal fusion, and cross-modal image-text matching, model the association between noun phrases in the text and image objects while increasing feature complementarity, and obtain fine-grained and coarse-grained multi-modal feature representations;

[0178] The entity-guided attention gating module is used to input the obtained fine-grained and coarse-grained multi-modal feature representations into the entity-guided attention gating module, aggregate the visual information related to the text entity, and obtain the final multi-modal fusion representation;

[0179] The task-specific decoder module is used to input the obtained multi-modal fusion representation into the task-specific decoder to perform task predictions for multi-modal named entity recognition and multi-modal relation extraction respectively.

[0180] The embodiments described above are only descriptions of the preferred embodiments of the present invention, and do not limit the scope of the present invention. Without departing from the design spirit of the present invention, various deformations and improvements made by those of ordinary skill in the art to the technical solutions of the present invention shall fall within the protection scope determined by the claims of the present invention.

Claims

1. A multimodal entity and relationship extraction method based on cross-modal alignment and fusion, characterized in that: The method comprises: Step 1: Process and encode the input text and image to obtain a variety of image and text features; wherein the multiple image and text features include: fine-grained text features, coarse-grained text features, fine-grained pixel-level image representation, coarse-grained pixel-level image representation, fine-grained semantic-level image representation, and coarse-grained semantic-level image representation; Step 2: Use the semantic representation of the image as the anchor point for aligning image and text features, align fine-grained text features with fine-grained pixel-level image representations, and align coarse-grained text features with coarse-grained pixel-level image representations, and map image and text features to the same semantic space. Step 3: Through text-guided dynamic gating aggregation, visual prefix cross-modal fusion, and cross-modal image-text matching, multi-granularity feature fusion is performed to increase feature complementarity while modeling the association between noun phrases in the text and image objects, thus obtaining fine-grained and coarse-grained multi-modal feature representations; Step 4: Input the obtained multi-granular multimodal feature representation into the entity-guided attention gating module to aggregate the visual information related to the text entity and obtain the final multimodal fusion representation; Step 5: Input the obtained multimodal fusion representation into the task-specific decoder to perform task predictions for multimodal named entity recognition and multimodal relationship extraction respectively.

2. The method according to claim 1, characterized in that In step 1, the input text and paired image are processed and encoded to obtain a variety of image and text features including: Acquire sample data, wherein the sample data includes original text and original image data paired with the original text; According to the sample data, five different encoders are used to obtain diversified image representations and corresponding multi-granularity text features. The five different encoders include a complete text encoder, a noun phrase encoder, a complete image encoder, a local object encoder, and an image description and object label encoder.

3. The method according to claim 2, characterized in that According to the sample data, five different encoders are used to obtain diverse image representations and corresponding multi-granularity text features including: Use the pre-trained language model Bert to initialize the full text encoder, encode the original text, and obtain fine-grained text features; use the Stanford CoreNLP tool to extract noun phrases from the original sentence text and input them into the noun phrase encoder initialized by the pre-trained language model Bert to obtain coarse-grained text features; The full image encoder is initialized using the pre-trained vision model Vision Transformer, and the original image is encoded to obtain fine-grained pixel-level visual features containing 12 layers of visual features. The noun phrases extracted from the original text using the Stanford CoreNLP tool are used to detect objects using the visual grounding toolkit, and the top N most relevant local objects are extracted from each original image and input into the local object encoder initialized by the pre-trained vision model Vision Transformer to obtain coarse-grained pixel-level visual features containing 12 layers of visual features. The original image is input into the ViT-GPT2 model to obtain an image description representing the global information of the image. The image description is then regarded as plain text and encoded using the image description and object label encoder initialized by the pre-trained language model Bert to obtain a fine-grained semantic-level image representation of the image. The object detector Faster-RCNN is used to detect objects in the image, and then the object labels are encoded using the image description and object label encoder to obtain a coarse-grained semantic-level image representation of the image.

4. The method according to claim 3, characterized in that In the step 2, the semantic representation of the image is used as an anchor point for aligning the image and text features, and feature alignment of fine-grained text features and fine-grained pixel-level image representation, as well as coarse-grained text features and coarse-grained pixel-level image representation are performed respectively, and mapping the image and text features to the same semantic space includes: The obtained coarse-grained and fine-grained text representations, image pixel representations, and image semantic representations are processed separately, including: Perform feature alignment of text and image semantic representations. Where B represents the batch size, p, q represent the sample index labels, and assume that the positive example is a representation of B pairs from the same input pair {T p ,S q } p＝q , other representations from different input pairs are considered negative examples; Use two multi-layer perceptron layers to p and S q The global representation embedding [CLS] is nonlinearly mapped to obtain the low-dimensional embedding representation after mapping Where G represents the global feature after nonlinear mapping, d represents the dimension of the embedding space, and the global representation embedding [CLS] comes from the last layer output of the image encoder and the text encoder; Using cosine similarity To calculate the similarity between text and image semantic representation, the similarity p between text representation and image semantic representation in a batch t2s Defined as where τ is a learnable temperature parameter; similarly, the similarity p between the image semantic representation and the text representation within a batch s2t Defined as Let y t2s (I) and y s2t (I) represents the true one-hot similarity vector, maximizing the similarity of positive examples and minimizing the similarity of negative examples. The feature alignment contrast loss of text and image semantic representation is defined as the binary cross entropy H:L between p and y. ts =H(p t2s ,y t2s (I))+H(p s2t ,y s2t (I)); Perform feature alignment between image pixel representation and image semantic representation. Where B represents the batch size, p, q represent the sample index labels, and assume that the positive example is a pair of B pairs of representations from the same input pair {I p ,S q } p＝q , other representations from different input pairs are considered negative examples; Using two multi-layer perceptron layers, I p and S q The global representation embedding [CLS] is nonlinearly mapped to obtain the low-dimensional embedding representation after mapping Where G represents the global feature after nonlinear mapping, d represents the dimension of the embedding space, and the global representation embedding [CLS] comes from the last layer output of the image encoder and the text encoder; Using cosine similarity To calculate the similarity between the image pixel representation and the image semantic representation, the similarity p between the image pixel representation and the image semantic representation in a batch i2s Defined as Where τ is a learnable temperature parameter. Similarly, the similarity p between the semantic representation of an image and the pixel representation of an image within a batch is s2i Defined as Let y i2s (I) and y s2i (I) represents the true one-hot similarity vector, maximizing the similarity of positive examples and minimizing the similarity of negative examples. The feature alignment contrast loss between image pixel representation and image semantic representation is defined as the binary cross entropy H:L between p and y. is =H(p i2s ,y i2s (I))+H(p s2i ,y s2i (I)); The final semantic anchor contrast loss is defined as:

5. The method according to claim 4, characterized in that In the step 3, multi-granularity feature fusion is performed through text-guided dynamic gating aggregation, visual prefix cross-modal fusion, and cross-modal image-text matching to obtain fine-grained and coarse-grained multi-modal feature representations, including: The coarse-grained and fine-grained text representations, image pixel representations and image semantic representations after feature alignment are processed separately; including: The fine-grained text representation T f And the image representation I f and S f Perform fine-grained feature fusion based on visual prefix transformer blocks; Perform text-guided dynamic gating aggregation to obtain complementary image features of text at a specific level; the dynamic gating signal represents the probability of the k-th layer visual features obtained from the image encoder to the l-th fine-grained visual prefix transformer block: Where k and l represent the hierarchical indices of the image encoder and visual prefix transformer blocks, respectively, and K represents the maximum number of layers of the image encoder. represents the fusion representation of the k-th layer visual features and fine-grained text features, W I is a learnable weight parameter, (α f ) l is the dynamic gating signal of the lth layer, Represents the representation of the global text features output by the previous layer of visual prefix transformer, represents the k-th layer of fine-grained pixel-level visual features, [,] represents the cascade operation; Using the gate signal (α f ) l Get the final aggregated fine-grained hierarchical visual features Project the aggregated fine-grained hierarchical visual features to the visual key vector and visual value vector middle in is the visual attention mapping parameter, encoding the fine-grained text features of the l-1th layer Projection to the cross-attention text query vector key vector and value vector middle, Perform visual prefix cross-modal fusion, where They are the attention mapping parameters respectively; the latent features of the lth layer are calculated by visual prefix attention Iteratively update layer by layer, and the hidden features of the last layer are the cross-modal fine-grained text feature encoding representation; Based on the similarity matrix obtained by semantic anchor point comparison learning, cross-modal image and text matching is performed; Set the diagonal elements of the matrix to 0 to avoid comparison with positive samples; then, sample negative text and negative images from the same batch according to the similarity distribution, and assign sampling probabilities based on their similarity with the target image or text; Use the last layer output of the visual prefix transformer block to embed the [CLS] token as the fusion representation of the image-text pair and predict the probability p of the image and text matching. itm , and get the image-text matching loss L itm =H(y itm ,p itm (T,I)), where y itm is a two-dimensional one-hot vector representing the true value label; Respectively represent the text T f The paired image represents I f and S f The matching probability prediction of this module is used to obtain fine-grained multimodal features. Where L represents the maximum number of layers of the fine-grained visual prefix transformer block, and Represents the probability of image and text matching, which is used to determine the fine-grained multimodal features after the fusion of the retained fine-grained text features and the fine-grained pixel-level visual features. Fine-grained multimodal features after fusion of fine-grained text features and fine-grained semantic representations proportion; Perform fine-grained text representation T c And the image representation I c and S c The coarse-grained features of M are fused to obtain the coarse-grained multimodal features M C .

6. The method according to claim 5, characterized in that In the step 4, the obtained fine-grained and coarse-grained multimodal feature representations are input into the entity-guided attention gating module to obtain the final multimodal fusion representation, including: The multimodal feature M C and M f As the input of attention gating, we obtain the fine-grained entity representation E f As query Q, coarse-grained noun phrase-object features as keys K and values ​​V, E f is used to focus on relevant image object features and generate entity-aware object representations: E c =Cross-att(Q=E f ,K=M c ,V=M c ); Use the gate signal g e =sigmoid(W e [W c ,E f ]) Control entity-aware object representation E C The contribution of , to obtain the final multimodal fusion representation: E = g e *E c +(1-g e )*E f .

7. The method according to claim 6, characterized in that In the step 5, the obtained multimodal fusion representation is input into a task-specific decoder, and task predictions of multimodal named entity recognition and multimodal relationship extraction are performed respectively, including: Based on the obtained multimodal fusion representation E, named entity recognition and relationship classification are performed respectively; including: sending the multimodal fusion representation to the CRF layer, and for the obtained label sequence y, using the cross entropy loss function to calculate the multimodal named entity recognition task loss: Where n is the length of the sequence, and i represents the index number of the sequence; The multimodal representation is input into the multilayer perceptron, and the relationship is classified through the softmax classifier; the cross entropy loss function is used to calculate the multimodal relationship extraction task loss: Among them, r represents the real relationship category; Based on the multimodal named entity recognition task loss and the multimodal relationship extraction task loss, the final loss function L = L ner / re +L SACL +L itm , where L ner / re Represents the loss of the NER or RE task, and the final loss function is used to iteratively optimize the model.

8. A multimodal entity and relationship extraction system based on cross-modal alignment and fusion, the system being used to implement the method of any one of claims 1 to 7, characterized in that: The system includes: a diversified multimodal feature acquisition module, a semantic anchor contrast learning module, a multi-granularity visual text collaborative fusion module, an entity-guided attention gating module, and a task-specific decoder module; The diversified multimodal feature acquisition module is used to process and encode the input text and image to obtain a variety of image and text features; wherein the various image and text features include: fine-grained text features, coarse-grained text features, fine-grained pixel-level image representation, coarse-grained pixel-level image representation, fine-grained semantic-level image representation, and coarse-grained semantic-level image representation; The semantic anchor contrast learning module is used to use the semantic representation of the image as an anchor for aligning image and text features, and to align fine-grained text features with fine-grained pixel-level image representations, and coarse-grained text features with coarse-grained pixel-level image representations, respectively, to map image and text features to the same semantic space; The multi-granularity visual text collaborative fusion module is used to perform multi-granularity feature fusion through text-guided dynamic gating aggregation, visual prefix cross-modal fusion, and cross-modal image-text matching, thereby increasing feature complementarity and modeling the association between noun phrases in the text and image objects, thereby obtaining fine-grained and coarse-grained multi-modal feature representations; The entity-guided attention gating module is used to input the obtained multi-granularity multimodal feature representation into the entity-guided attention gating module, gather visual information related to the text entity, and obtain the final multimodal fusion representation; The task-specific decoder module is used to input the obtained multimodal fusion representation into the task-specific decoder to perform task predictions of multimodal named entity recognition and multimodal relationship extraction respectively.

Images